2024-03-18T01:10:40.349 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T01:10:40.349 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564 --name pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} -- /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/orig.config.yaml 2024-03-18T01:10:40.401 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T01:10:40.488 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7608564' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFGVfZM9nUxiR2h/U9LYTl5msB9B7oCYm/aF/4Wtym0Uh1rFBzt66mbiDiDwdoIJ/SypsKQviP9PmDjSV3ETBdQ= smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF4mBO/8A11c0wg5qNETN3FIlA//9VpkOejz2ACa8BDhRWPN6/24DTAEK3zqvKdPi1v1wr0axrF0qTQjc/OXvGI= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEkLfbf54T28KbGzzmjAKyRgVbMPkJs1/WkxT10hhTRpIIpvXnTVv0iyouhKjXpYOVnG01NaU9XgGSz9cr0SHJw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-17_23:17:27 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T01:10:40.488 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa; will attempt to use it 2024-03-18T01:10:40.489 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks 2024-03-18T01:10:40.489 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T01:10:40.490 INFO:teuthology.task.internal:Checking packages... 2024-03-18T01:10:40.551 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T01:10:40.551 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T01:10:40.551 INFO:teuthology.packaging:ref: None 2024-03-18T01:10:40.551 INFO:teuthology.packaging:tag: None 2024-03-18T01:10:40.551 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T01:10:40.551 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:10:40.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T01:10:40.689 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T01:10:40.690 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T01:10:40.696 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T01:10:40.696 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T01:10:40.703 INFO:teuthology.task.internal:Saving configuration 2024-03-18T01:10:40.720 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T01:10:40.726 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T01:10:40.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 01:05:31.806140', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFGVfZM9nUxiR2h/U9LYTl5msB9B7oCYm/aF/4Wtym0Uh1rFBzt66mbiDiDwdoIJ/SypsKQviP9PmDjSV3ETBdQ='} 2024-03-18T01:10:40.776 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 01:05:31.807473', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF4mBO/8A11c0wg5qNETN3FIlA//9VpkOejz2ACa8BDhRWPN6/24DTAEK3zqvKdPi1v1wr0axrF0qTQjc/OXvGI='} 2024-03-18T01:10:40.798 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 01:05:31.804310', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEkLfbf54T28KbGzzmjAKyRgVbMPkJs1/WkxT10hhTRpIIpvXnTVv0iyouhKjXpYOVnG01NaU9XgGSz9cr0SHJw='} 2024-03-18T01:10:40.798 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T01:10:40.803 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-18T01:10:40.803 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-18T01:10:40.803 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-18T01:10:40.803 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T01:10:40.899 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6c58dfb160>, signals=[15]) 2024-03-18T01:10:40.900 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T01:10:40.906 INFO:teuthology.task.internal:Opening connections... 2024-03-18T01:10:40.906 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-03-18T01:10:40.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:10:41.013 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-03-18T01:10:41.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:10:41.114 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-03-18T01:10:41.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:10:41.203 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T01:10:41.210 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-03-18T01:10:41.228 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-03-18T01:10:41.228 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-03-18T01:10:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T01:10:41.286 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T01:10:41.287 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-03-18T01:10:41.300 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-03-18T01:10:41.318 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-03-18T01:10:41.318 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-03-18T01:10:41.376 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-03-18T01:10:41.376 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-03-18T01:10:41.376 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T01:10:41.377 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T01:10:41.377 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-03-18T01:10:41.392 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-03-18T01:10:41.408 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-03-18T01:10:41.409 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T01:10:41.463 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-03-18T01:10:41.464 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T01:10:41.464 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T01:10:41.464 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T01:10:41.464 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-03-18T01:10:41.492 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T01:10:41.502 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T01:10:41.507 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T01:10:41.507 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-03-18T01:10:41.510 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-03-18T01:10:41.512 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-03-18T01:10:41.528 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T01:10:41.535 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T01:10:41.535 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-03-18T01:10:41.566 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-03-18T01:10:41.570 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-03-18T01:10:41.691 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T01:10:41.767 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T01:10:41.780 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-18T01:10:41.780 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-18T01:10:41.780 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-18T01:10:41.780 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T01:10:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:10:41.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-18T01:10:42.098 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T01:10:42.099 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3845b7ac715a3ff198582638d199a90e30e254da"... 2024-03-18T01:10:42.099 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-18T01:10:42.116 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2024-03-18T01:10:42.116 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T01:10:42.117 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-18T01:10:42.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum upgrade -y linux-firmware 2024-03-18T01:10:44.374 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-18T01:10:45.899 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-18T01:10:50.964 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-03-18T01:10:55.365 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-03-18T01:11:01.882 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-18T01:11:03.266 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-18T01:11:04.986 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-18T01:11:10.293 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-18T01:11:12.695 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 1 Package 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:11:12.712 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 375 M 2024-03-18T01:11:12.713 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-18T01:11:24.793 INFO:teuthology.orchestra.run.smithi007.stdout:linux-firmware-20240219-143.el9.noarch.rpm 31 MB/s | 375 MB 00:12 2024-03-18T01:11:24.794 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:11:24.794 INFO:teuthology.orchestra.run.smithi007.stdout:Total 31 MB/s | 375 MB 00:12 2024-03-18T01:11:24.794 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-18T01:11:24.880 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-18T01:11:24.880 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-18T01:11:27.366 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-18T01:11:27.366 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-18T01:11:33.960 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-18T01:11:34.176 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-18T01:11:34.280 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T01:11:35.463 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T01:11:35.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-18T01:11:35.878 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T01:11:35.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:11:35.878 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-03-18T01:11:35.878 INFO:teuthology.orchestra.run.smithi007.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-18T01:11:35.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:11:35.878 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-18T01:11:36.153 INFO:teuthology.task.kernel:Downloading kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-18T01:11:36.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T01:11:36.494 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ 2024-03-18T01:11:36.494 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-18T01:11:36.496 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi007.front.sepia.ceph.com to complete... 2024-03-18T01:12:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18 01:12:03 URL:https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm [336075563/336075563] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-18T01:12:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:FINISHED --2024-03-18 01:12:03-- 2024-03-18T01:12:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:Total wall clock time: 27s 2024-03-18T01:12:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:Downloaded: 1 files, 321M in 27s (11.9 MB/s) 2024-03-18T01:12:03.905 INFO:teuthology.task.kernel:Installing kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-18T01:12:03.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-18T01:12:05.934 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying... ######################################## 2024-03-18T01:12:05.988 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing... ######################################## 2024-03-18T01:12:06.073 INFO:teuthology.orchestra.run.smithi007.stdout:Updating / installing... 2024-03-18T01:12:14.005 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-6.8.0_rc7_g3845b7ac715a-1 ######################################## 2024-03-18T01:12:28.541 INFO:teuthology.orchestra.run.smithi007.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T01:12:51.796 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-18T01:12:51.797 DEBUG:teuthology.orchestra.run.smithi007:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/System.map 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/config 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-18T01:12:51.905 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-18T01:12:51.906 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-18T01:12:51.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-18T01:12:51.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-18T01:12:51.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel 2024-03-18T01:12:51.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu 2024-03-18T01:12:51.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce 2024-03-18T01:12:51.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-18T01:12:51.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm 2024-03-18T01:12:51.907 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ansi_cprng.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/anubis.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/arc4.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_memcpy.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_pq.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_tx.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_xor.ko 2024-03-18T01:12:51.908 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/raid6test.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authenc.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authencesn.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blake2b_generic.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_common.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_generic.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/camellia_generic.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast5_generic.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast6_generic.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast_common.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ccm.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cmac.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crc32_generic.ko 2024-03-18T01:12:51.909 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_engine.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_user.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cts.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/des_generic.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/dh_generic.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/echainiv.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/essiv.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/fcrypt.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/khazad.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4hc.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/md4.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/michael_mic.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcbc.ko 2024-03-18T01:12:51.910 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcrypt.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/rmd160.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/seed.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/serpent_generic.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tcrypt.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tea.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_common.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_generic.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/vmac.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/wp512.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xcbc.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xor.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xxhash_generic.ko 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi 2024-03-18T01:12:51.911 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_pad.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/custom_method.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/ec_sys.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/platform_profile.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbs.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbshc.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/video.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/acard-ahci.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ahci_platform.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ata_generic.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/libahci_platform.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_acpi.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ali.ko 2024-03-18T01:12:51.912 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_amd.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_artop.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atiixp.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atp867x.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd640.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd64x.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cypress.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_efar.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt366.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt37x.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it8213.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it821x.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_jmicron.ko 2024-03-18T01:12:51.913 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_marvell.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_mpiix.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_netcell.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ninja32.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87410.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87415.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_oldpiix.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_opti.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_optidma.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pcmcia.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_piccolo.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_rdc.ko 2024-03-18T01:12:51.914 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sch.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_serverworks.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sil680.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sis.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sl82c105.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_triflex.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_via.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pdc_adma.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_inic162x.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_mv.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_nv.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_promise.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_qstor.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil24.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sis.ko 2024-03-18T01:12:51.915 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_svw.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sx4.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_uli.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_via.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_vsc.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay/ks0108.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma/bcma.ko 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe 2024-03-18T01:12:51.916 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe/aoe.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/brd.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd/drbd.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/floppy.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/loop.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/nbd.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk/null_blk.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/pktcdvd.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/rbd.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/virtio_blk.ko 2024-03-18T01:12:51.917 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkfront.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram/zram.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hangcheck-timer.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/via-rng.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-18T01:12:51.918 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/lp.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave/mwave.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ppdev.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tlclk.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-18T01:12:51.919 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/uv_mmtimer.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/virtio_console.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp.ko 2024-03-18T01:12:51.920 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-aes.ko 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-sha.ko 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca/dca.ko 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-18T01:12:51.921 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/amd64_edac.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/e752x_edac.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/edac_mce_amd.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3000_edac.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3200_edac.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5100_edac.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5400_edac.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7300_edac.ko 2024-03-18T01:12:51.922 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7core_edac.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i82975x_edac.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/ie31200_edac.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/sb_edac.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/x38_edac.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/edd.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-ich.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-appleir.ko 2024-03-18T01:12:51.923 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-aureal.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-axff.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-dr.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elecom.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elo.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-emsff.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gaff.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gt683r.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gyration.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtekff.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-hyperv.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-icade.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-keytouch.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-kye.ko 2024-03-18T01:12:51.924 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lcpower.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-led.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lenovo.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-multitouch.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-ortek.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-penmount.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-petalynx.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-picolcd.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-pl.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-plantronics.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-primax.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-rmi.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-common.ko 2024-03-18T01:12:51.925 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-saitek.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-samsung.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sjoy.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sony.ko 2024-03-18T01:12:51.926 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-speedlink.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-steelseries.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sunplus.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tivo.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tmff.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-topseed.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-twinhan.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-uclogic.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-waltop.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-wiimote.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-xinmo.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zpff.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zydacron.ko 2024-03-18T01:12:51.927 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/uhid.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/wacom.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_balloon.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_utils.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_vmbus.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru3.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7414.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7418.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adc128d818.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1025.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1026.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1029.ko 2024-03-18T01:12:51.928 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1031.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm9240.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ads7828.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7410.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7411.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7462.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7470.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7475.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7x10.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/amc6821.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/applesmc.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asb100.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asc7621.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/atxp1.ko 2024-03-18T01:12:51.929 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/coretemp.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dme1737.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds1621.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds620.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc1403.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc6w201.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71805f.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71882fg.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f75375s.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fam15h_power.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fschmd.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g760a.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g762.ko 2024-03-18T01:12:51.930 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl518sm.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl520sm.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5500_temp.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5k_amb.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmaem.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmpex.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina209.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina2xx.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/it87.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k10temp.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k8temp.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lineage-pem.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm63.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm73.ko 2024-03-18T01:12:51.931 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm75.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm77.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm78.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm80.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm83.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm85.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm87.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm90.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm92.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm93.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95234.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95241.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95245.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc2945.ko 2024-03-18T01:12:51.932 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4151.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4215.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4222.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4245.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4260.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4261.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max16065.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1619.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1668.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max197.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6639.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6650.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6697.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/mcp3021.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6683.ko 2024-03-18T01:12:51.933 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775-core.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct7802.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87360.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87427.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pcf8591.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-18T01:12:51.934 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/powr1220.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5627.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5636.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht15.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht21.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/shtc1.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sis5595.ko 2024-03-18T01:12:51.935 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47b397.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m1.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m192.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/thmc50.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp102.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp103.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp401.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp421.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via-cputemp.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via686a.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt1211.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt8231.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627ehf.ko 2024-03-18T01:12:51.936 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627hf.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83781d.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83791d.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83792d.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83793.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83795.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l785ts.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l786ng.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-18T01:12:51.937 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-18T01:12:51.938 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-dev.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-mux.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-smbus.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-stub.ko 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband 2024-03-18T01:12:51.939 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_core.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-18T01:12:51.940 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-18T01:12:51.941 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/gameport.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/joydev.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/appletouch.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/bcm5974.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/cyapatp.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/sermouse.ko 2024-03-18T01:12:51.942 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/altera_ps2.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/arc_ps2.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/serio_raw.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/sparse-keymap.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-blinkm.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lm3530.ko 2024-03-18T01:12:51.943 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lp3944.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-ss4200.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache/bcache.ko 2024-03-18T01:12:51.944 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-bio-prison.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache-smq.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-crypt.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-delay.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-flakey.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-log-userspace.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-multipath.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-queue-length.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-raid.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-round-robin.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-service-time.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-switch.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-thin-pool.ko 2024-03-18T01:12:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-verity.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid0.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid1.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid10.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid456.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptbase.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptctl.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptfc.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptlan.ko 2024-03-18T01:12:51.946 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptsas.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptscsih.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptspi.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_ich.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_sch.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/mfd-core.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/sm501.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/viperboard.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/vx855.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/wl1273-core.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds9802als.ko 2024-03-18T01:12:51.947 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds990x.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/bh1770glc.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710/cb710.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/at24.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/max6875.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/enclosure.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/hpilo.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29003.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29020.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-18T01:12:51.948 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-me.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-txe.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_7xx1.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_core.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tsl2550.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_balloon.ko 2024-03-18T01:12:51.949 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding/bonding.ko 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-18T01:12:51.950 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dummy.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/eql.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-18T01:12:51.951 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-18T01:12:51.952 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-18T01:12:51.953 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-18T01:12:51.954 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-18T01:12:51.955 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dnet.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ethoc.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/fealnx.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-18T01:12:51.956 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/jme.ko 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-18T01:12:51.957 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-18T01:12:51.958 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-18T01:12:51.959 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-18T01:12:51.960 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-18T01:12:51.961 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/geneve.ko 2024-03-18T01:12:51.962 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ifb.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvlan.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvtap.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-18T01:12:51.963 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mii.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/net_failover.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/netconsole.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/nlmon.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/amd.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm87xx.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/broadcom.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/cicada.ko 2024-03-18T01:12:51.964 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/davicom.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/dp83640.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/et1011c.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/icplus.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/lxt.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/marvell.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/micrel.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/national.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/phylink.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/qsemi.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/realtek.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/smsc.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/ste10Xp.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/vitesse.ko 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp 2024-03-18T01:12:51.965 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_async.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppoe.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppox.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pptp.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slhc.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slip.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/sungem_phy.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tap.ko 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team 2024-03-18T01:12:51.966 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_random.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tun.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/veth.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/virtio_net.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan/vxlan.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-18T01:12:51.967 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netfront.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb/ntb.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme-core.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_cs.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_pc.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_serial.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-18T01:12:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/xen-pcifront.ko 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/i82092.ko 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/pd6729.ko 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acerhdf.ko 2024-03-18T01:12:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/hdaps.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel_ips.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-18T01:12:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-18T01:12:51.971 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps_parport.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/pps_core.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp_kvm.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-18T01:12:51.972 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-em3027.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-max6900.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-18T01:12:51.973 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-x1205.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-9xxx.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-sas.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-xxxx.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/BusLogic.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/a100u2w.ko 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid 2024-03-18T01:12:51.974 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/advansys.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/am53c974.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/atp870u.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa/bfa.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-18T01:12:51.975 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ch.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dc395x.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-18T01:12:51.976 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dmx3191d.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esp_scsi.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic/fnic.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hpsa.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hptiop.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hv_storvsc.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/initio.ko 2024-03-18T01:12:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ipr.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ips.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci/isci.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc/libfc.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas/libsas.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-18T01:12:51.978 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvumi.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pmcraid.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla1280.ko 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx 2024-03-18T01:12:51.979 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/raid_class.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_debug.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ses.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/st.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/stex.ko 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2 2024-03-18T01:12:51.980 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/virtio_scsi.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/wd719x.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb/ssb.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_file.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_iblock.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_mod.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_pscsi.ko 2024-03-18T01:12:51.981 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_user.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-18T01:12:51.982 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_gsm.ko 2024-03-18T01:12:51.983 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_hdlc.ko 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/nozomi.ko 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/arc_uart.ko 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio.ko 2024-03-18T01:12:51.984 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_aec.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_cif.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_pci_generic.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_sercos3.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-acm.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usblp.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usbtmc.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/mdc800.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/microtek.ko 2024-03-18T01:12:51.985 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/adutux.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/appledisplay.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi26.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi62.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ezusb.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/idmouse.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/iowarrior.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ldusb.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/legousbtower.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-18T01:12:51.986 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usb3503.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usblcd.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/uss720.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/yurex.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy/phy-generic.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/aircable.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ark3116.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ch341.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cp210x.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cyberjack.ko 2024-03-18T01:12:51.987 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-18T01:12:51.988 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-18T01:12:51.988 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/empeg.ko 2024-03-18T01:12:51.988 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-18T01:12:51.988 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-18T01:12:51.988 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-18T01:12:51.988 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_ti.ko 2024-03-18T01:12:51.989 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipaq.ko 2024-03-18T01:12:51.989 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipw.ko 2024-03-18T01:12:51.989 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ir-usb.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mct_u232.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7720.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7840.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/navman.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/omninet.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/opticon.ko 2024-03-18T01:12:51.990 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/option.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/oti6858.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/pl2303.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcaux.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcserial.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/quatech2.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/safe_serial.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/sierra.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ssu100.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/symbolserial.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_debug.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-18T01:12:51.991 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/visor.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/whiteheat.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/uas.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-karma.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-18T01:12:51.992 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/usb-storage.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost.ko 2024-03-18T01:12:51.993 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_net.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_scsi.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/apple_bl.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/lcd.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_balloon.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_mmio.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci.ko 2024-03-18T01:12:51.994 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_ring.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2482.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2490.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-18T01:12:51.995 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/wire.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/hpwdt.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/i6300esb.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ib700wdt.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ibmasr.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-18T01:12:51.996 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it87_wdt.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/machzwd.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/nv_tco.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/softdog.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/via_wdt.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/wdt_pci.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/xen_wdt.ko 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen 2024-03-18T01:12:51.997 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-evtchn.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntalloc.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntdev.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-privcmd.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p/9p.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs/affs.ko 2024-03-18T01:12:51.998 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs/befs.ko 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/binfmt_misc.ko 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs/btrfs.ko 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles/cachefiles.ko 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph/ceph.ko 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda/coda.ko 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs/cramfs.ko 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm/dlm.ko 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs 2024-03-18T01:12:51.999 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs/f2fs.ko 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/fat.ko 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/msdos.ko 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/vfat.ko 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/cuse.ko 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/fuse.ko 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2/gfs2.ko 2024-03-18T01:12:52.000 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs/hfs.ko 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus/hfsplus.ko 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs/isofs.ko 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs/jfs.ko 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd/lockd.ko 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix/minix.ko 2024-03-18T01:12:52.001 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs/netfs.ko 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfs.ko 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv3.ko 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv4.ko 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common 2024-03-18T01:12:52.002 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/grace.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/nfs_acl.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd/nfsd.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2/nilfs2.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-celtic.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-centeuro.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-croatian.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-cyrillic.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-gaelic.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-greek.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-iceland.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-inuit.ko 2024-03-18T01:12:52.003 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-roman.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-romanian.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-turkish.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1250.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1251.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1255.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp737.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp775.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp850.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp852.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp855.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp857.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp860.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp861.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp862.ko 2024-03-18T01:12:52.004 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp863.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp864.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp865.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp866.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp869.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp874.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp932.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp936.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp949.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp950.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_euc-jp.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-1.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-13.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-14.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-15.ko 2024-03-18T01:12:52.005 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-2.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-3.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-4.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-5.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-6.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-7.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-9.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-r.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-ru.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-u.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_utf8.ko 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster 2024-03-18T01:12:52.006 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs/overlay.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore/ramoops.ko 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs 2024-03-18T01:12:52.007 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs/reiserfs.ko 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs/romfs.ko 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client/cifs.ko 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_arc4.ko 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_md4.ko 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs/squashfs.ko 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv/sysv.ko 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf/udf.ko 2024-03-18T01:12:52.008 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs/ufs.ko 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs/xfs.ko 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu/rcutorture.ko 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/torture.ko 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/asn1_encoder.ko 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc-itu-t.ko 2024-03-18T01:12:52.009 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc8.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libarc4.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libdes.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/libcrc32c.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lru_cache.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4_compress.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4hc_compress.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math/cordic.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6/raid6_pq.ko 2024-03-18T01:12:52.010 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_bm.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_fsm.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_kmp.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd/zstd_compress.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm/hwpoison-inject.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/6lowpan.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_dest.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_fragment.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_hop.ko 2024-03-18T01:12:52.011 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_mobility.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_routing.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_udp.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/garp.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/mrp.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/p8022.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/psnap.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/stp.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q/8021q.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet.ko 2024-03-18T01:12:52.012 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_fd.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_rdma.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_virtio.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv/batman-adv.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/br_netfilter.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/bridge.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-18T01:12:52.013 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-18T01:12:52.014 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtables.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph/libceph.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/failover.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/pktgen.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_diag.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv4.ko 2024-03-18T01:12:52.015 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv6.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver/dns_resolver.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/dsa_core.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_brcm.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_dsa.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_none.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_trailer.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-18T01:12:52.016 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ah4.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/esp4.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/fou.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/gre.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/inet_diag.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_gre.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_tunnel.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_vti.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipcomp.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipip.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-18T01:12:52.017 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-18T01:12:52.018 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_bic.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_dctcp.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_diag.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_highspeed.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_htcp.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_hybla.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_illinois.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_lp.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_scalable.ko 2024-03-18T01:12:52.019 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_vegas.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_veno.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_westwood.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_yeah.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tunnel4.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_diag.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_tunnel.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ah6.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/esp6.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/fou6.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_tunnel.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_vti.ko 2024-03-18T01:12:52.020 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ipcomp6.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-18T01:12:52.021 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/sit.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/tunnel6.ko 2024-03-18T01:12:52.022 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key/af_key.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_core.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_eth.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip6.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_netlink.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ppp.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc/llc.ko 2024-03-18T01:12:52.023 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154/mac802154.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls/mpls_gso.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-18T01:12:52.024 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-18T01:12:52.025 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conncount.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-18T01:12:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_log_syslog.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-18T01:12:52.027 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_irc.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_sip.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_tables.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_log.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_chain_nat.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_compat.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_ct.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fib.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-18T01:12:52.028 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_hash.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_limit.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_log.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_masq.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_nat.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_numgen.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_queue.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_quota.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_redir.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject_inet.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_socket.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_tunnel.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_AUDIT.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-18T01:12:52.029 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CT.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_DSCP.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HL.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HMARK.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LED.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LOG.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NETMAP.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFLOG.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_RATEEST.ko 2024-03-18T01:12:52.030 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_SECMARK.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TEE.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TPROXY.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TRACE.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_addrtype.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_bpf.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cgroup.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cluster.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_comment.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connbytes.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlabel.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlimit.ko 2024-03-18T01:12:52.031 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connmark.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_conntrack.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cpu.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dccp.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_devgroup.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dscp.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ecn.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_esp.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hashlimit.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_helper.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hl.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipcomp.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_iprange.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipvs.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_l2tp.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_length.ko 2024-03-18T01:12:52.032 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_limit.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mac.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mark.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_multiport.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nat.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nfacct.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_osf.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_owner.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_physdev.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_pkttype.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_policy.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_quota.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_rateest.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_realm.ko 2024-03-18T01:12:52.033 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_recent.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_sctp.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_set.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_socket.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_state.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_statistic.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_string.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_tcpmss.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_time.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_u32.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink/netlink_diag.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh/nsh.ko 2024-03-18T01:12:52.034 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/openvswitch.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-geneve.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-gre.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-vxlan.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet/af_packet_diag.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_rdma.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_tcp.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_csum.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_gact.ko 2024-03-18T01:12:52.035 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_mirred.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_nat.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_pedit.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_police.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_simple.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_skbedit.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_vlan.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_basic.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_bpf.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_flow.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_fw.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_route.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_u32.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_cmp.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_ipset.ko 2024-03-18T01:12:52.036 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_meta.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_nbyte.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_text.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_u32.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_choke.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_codel.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_drr.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_fq.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_gred.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hfsc.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hhf.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_htb.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_ingress.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio.ko 2024-03-18T01:12:52.037 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio_lib.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_multiq.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_netem.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_pie.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_plug.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_prio.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_qfq.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_red.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfb.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfq.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_tbf.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_teql.ko 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp 2024-03-18T01:12:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp_diag.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/sunrpc.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/diag.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/tipc.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix/unix_diag.ko 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock 2024-03-18T01:12:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock.ko 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys/trusted.ko 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt 2024-03-18T01:12:52.040 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib/irqbypass.ko 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias.bin 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.alias.bin 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.bin 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.modinfo 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep.bin 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.devname 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.order 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.softdep 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols.bin 2024-03-18T01:12:52.041 INFO:teuthology.orchestra.run.smithi007.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-18T01:12:52.045 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-g3845b7ac715a 2024-03-18T01:12:52.046 DEBUG:teuthology.orchestra.run.smithi007:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-18T01:12:52.070 INFO:teuthology.orchestra.run.smithi007.stdout:postinstall scriptlet (using /bin/sh): 2024-03-18T01:12:52.070 INFO:teuthology.orchestra.run.smithi007.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-18T01:12:52.070 INFO:teuthology.orchestra.run.smithi007.stdout: /usr/bin/kernel-install add 6.8.0-rc7-g3845b7ac715a /lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-18T01:12:52.070 INFO:teuthology.orchestra.run.smithi007.stdout:fi 2024-03-18T01:12:52.070 INFO:teuthology.orchestra.run.smithi007.stdout:for file in vmlinuz System.map config; do 2024-03-18T01:12:52.070 INFO:teuthology.orchestra.run.smithi007.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a"; then 2024-03-18T01:12:52.070 INFO:teuthology.orchestra.run.smithi007.stdout: cp "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a" 2024-03-18T01:12:52.070 INFO:teuthology.orchestra.run.smithi007.stdout: fi 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:done 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:new-kernel-pkg --remove 6.8.0-rc7-g3845b7ac715a --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-g3845b7ac715a.img 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-install remove 6.8.0-rc7-g3845b7ac715a 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:fi 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-g3845b7ac715a 2024-03-18T01:12:52.071 INFO:teuthology.orchestra.run.smithi007.stdout:fi 2024-03-18T01:12:52.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-03-18T01:12:52.153 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-03-18T01:12:52.153 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-03-18T01:12:52.153 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T01:12:52.154 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-03-18T01:12:52.155 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-03-18T01:12:52.155 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:12:52.155 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T01:12:52.155 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T01:12:52.155 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T01:12:52.155 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-03-18T01:12:52.155 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:12:52.155 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T01:12:52.156 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T01:12:52.156 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 6.8.0-rc7-g3845b7ac715a 2024-03-18T01:12:52.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T01:12:52.795 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-03-18T01:12:54.176 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T01:12:54.196 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-03-18T01:12:54.198 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-18T01:12:54.214 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.xrfTBaXysD 2024-03-18T01:12:54.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xrfTBaXysD 2024-03-18T01:12:54.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.xrfTBaXysD 2024-03-18T01:12:54.428 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.xrfTBaXysD is 6KB 2024-03-18T01:12:54.477 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.xrfTBaXysD 2024-03-18T01:12:54.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T01:12:54.568 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T01:12:54.568 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T01:12:54.568 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T01:12:54.568 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a.conf 2024-03-18T01:12:54.569 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a 2024-03-18T01:12:54.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-03-18T01:13:24.722 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-18T01:13:24.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-18T01:13:24.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:13:43.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2024-03-18T01:13:52.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-18T01:13:52.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:14:52.203 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T01:15:04.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-18T01:15:04.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:15:04.577 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-18T01:15:05.284 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-18T01:15:05.284 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-g3845b7ac715a"... 2024-03-18T01:15:05.285 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-18T01:15:05.338 INFO:teuthology.orchestra.run.smithi007.stdout:6.8.0-rc7-g3845b7ac715a 2024-03-18T01:15:05.339 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-g3845b7ac715a vs 6.8.0-rc7-g3845b7ac715a 2024-03-18T01:15:05.339 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T01:15:05.339 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T01:15:06.339 DEBUG:teuthology.parallel:result is None 2024-03-18T01:15:06.340 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T01:15:06.348 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T01:15:06.348 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T01:15:06.356 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T01:15:06.359 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T01:15:06.407 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T01:15:06.413 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T01:15:06.420 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T01:15:06.420 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T01:15:06.426 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T01:15:06.428 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T01:15:06.497 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T01:15:06.504 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T01:15:06.504 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T01:15:06.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:15:06.521 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T01:15:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:15:06.538 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T01:15:06.548 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T01:15:06.563 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T01:15:06.604 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:15:06.629 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:15:06.631 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:15:06.651 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:15:06.675 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:15:06.743 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T01:15:06.748 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T01:15:06.755 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T01:15:06.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T01:15:06.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T01:15:06.763 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T01:15:06.850 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T01:15:06.859 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T01:15:06.859 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T01:15:06.864 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T01:15:06.866 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T01:15:06.910 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:15:06.997 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:15:07.108 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:15:07.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:15:07.296 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:15:07.296 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T01:15:07.394 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:15:07.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:15:07.499 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:15:07.543 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:15:07.592 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:15:07.592 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T01:15:07.665 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:15:07.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T01:15:07.770 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:15:07.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T01:15:07.858 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:15:07.858 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T01:15:07.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-03-18T01:15:07.934 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-03-18T01:15:07.936 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-03-18T01:15:07.978 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:15:08.000 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:15:08.054 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:15:08.382 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T01:15:08.390 INFO:teuthology.task.internal:Starting timer... 2024-03-18T01:15:08.390 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T01:15:08.401 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T01:15:08.410 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-18T01:15:08.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-03-18T01:15:08.544 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-03-18T01:15:08.549 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-03-18T01:15:08.606 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-03-18T01:15:08.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-03-18T01:15:08.652 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-03-18T01:15:08.653 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T01:15:08.654 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-03-18T01:15:08.711 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-03-18T01:15:08.712 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-03-18T01:15:08.735 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-03-18T01:15:08.736 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-03-18T01:15:08.767 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-03-18T01:15:08.767 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-03-18T01:15:08.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T01:15:08.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:15:08.864 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T01:15:08.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:15:08.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T01:15:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:15:08.924 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T01:15:08.924 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-03-18T01:15:09.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-03-18T01:15:09.062 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-03-18T01:15:09.093 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T01:15:09.111 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-18T01:15:09.117 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-18T01:15:09.278 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T01:15:09.391 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T01:15:09.411 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi046.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T01:21:14.136 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-03-18T01:21:14.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-18T01:21:14.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:21:14.285 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-18T01:21:14.444 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-18T01:21:14.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-18T01:21:14.445 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:21:14.551 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-03-18T01:21:14.592 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-18T01:21:14.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-18T01:21:14.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T01:21:14.684 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-18T01:21:14.758 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-18T01:21:14.759 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T01:21:14.779 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T01:21:14.779 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T01:21:14.780 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:21:14.787 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T01:21:14.787 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:21:14.790 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T01:21:14.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:21:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T01:21:14.830 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T01:21:14.847 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T01:21:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T01:21:14.878 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-03-18T01:21:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-03-18T01:21:14.890 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T01:21:14.891 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-03-18T01:21:14.904 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-03-18T01:21:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T01:21:14.924 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T01:21:14.925 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T01:21:14.943 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T01:21:14.945 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T01:21:14.968 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:21:14.974 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:21:14.974 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-18T01:21:14.974 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:14.975 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:14.975 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:14.975 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:14.993 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:21:14.997 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:21:14.997 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-03-18T01:21:14.997 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:14.997 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:14.997 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:14.997 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:15.043 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-03-18T01:21:15.088 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-03-18T01:21:15.145 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T01:21:15.197 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T01:21:15.331 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:21:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:21:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-18T01:21:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:15.339 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:21:15.340 INFO:teuthology.run_tasks:Running task install... 2024-03-18T01:21:15.349 DEBUG:teuthology.task.install:project ceph 2024-03-18T01:21:15.349 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T01:21:15.349 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T01:21:15.350 INFO:teuthology.task.install:Using flavor: default 2024-03-18T01:21:15.356 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T01:21:15.356 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T01:21:15.356 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T01:21:15.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:21:15.359 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T01:21:15.359 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:21:15.360 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T01:21:15.360 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:21:15.514 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T01:21:15.515 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T01:21:15.518 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T01:21:15.518 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T01:21:15.532 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T01:21:15.533 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T01:21:15.630 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T01:21:15.630 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:21:15.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T01:21:15.648 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T01:21:15.648 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:21:15.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T01:21:15.682 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T01:21:15.683 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T01:21:15.719 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T01:21:15.720 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:21:15.720 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T01:21:15.734 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T01:21:15.735 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T01:21:15.751 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T01:21:15.751 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T01:21:15.756 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T01:21:15.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T01:21:15.845 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T01:21:15.875 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-03-18T01:21:15.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-03-18T01:21:15.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T01:21:15.904 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T01:21:15.972 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-03-18T01:21:15.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-03-18T01:21:16.023 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T01:21:16.119 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-03-18T01:21:16.123 INFO:teuthology.orchestra.run.smithi046.stdout:74 files removed 2024-03-18T01:21:16.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-03-18T01:21:16.151 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T01:21:16.210 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-03-18T01:21:16.237 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T01:21:16.508 INFO:teuthology.orchestra.run.smithi007.stdout:74 files removed 2024-03-18T01:21:16.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T01:21:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 235 kB/s | 83 kB 00:00 2024-03-18T01:21:16.910 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 217 kB/s | 83 kB 00:00 2024-03-18T01:21:17.162 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-18T01:21:17.343 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-18T01:21:17.587 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-18T01:21:17.597 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 156 kB/s | 83 kB 00:00 2024-03-18T01:21:17.727 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-18T01:21:18.048 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-18T01:21:18.213 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-18T01:21:18.214 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-18T01:21:18.798 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-18T01:21:19.526 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-18T01:21:19.590 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-18T01:21:19.713 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-18T01:21:20.597 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-18T01:21:21.477 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-18T01:21:22.406 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-18T01:21:22.948 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-18T01:21:23.654 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-18T01:21:24.674 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-18T01:21:27.299 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.1 MB 00:00 2024-03-18T01:21:28.700 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-18T01:21:30.066 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-18T01:21:30.651 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-18T01:21:31.850 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-18T01:21:34.935 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-03-18T01:21:35.860 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-18T01:21:38.406 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T01:21:38.407 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T01:21:38.420 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T01:21:38.421 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T01:21:38.422 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T01:21:38.423 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T01:21:38.425 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T01:21:38.427 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T01:21:38.428 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T01:21:38.515 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-18T01:21:38.537 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T01:21:38.538 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T01:21:38.539 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T01:21:38.540 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T01:21:38.541 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T01:21:38.542 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T01:21:38.543 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T01:21:38.544 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T01:21:38.545 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T01:21:38.546 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T01:21:38.547 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T01:21:38.548 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T01:21:38.549 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T01:21:38.550 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T01:21:38.551 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout:Install 150 Packages 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 24 Packages 2024-03-18T01:21:38.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:21:38.553 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 224 M 2024-03-18T01:21:38.553 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-18T01:21:39.320 INFO:teuthology.orchestra.run.smithi046.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-03-18T01:21:39.570 INFO:teuthology.orchestra.run.smithi046.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 3.5 MB/s | 890 kB 00:00 2024-03-18T01:21:39.637 INFO:teuthology.orchestra.run.smithi046.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 2.1 MB/s | 145 kB 00:00 2024-03-18T01:21:39.995 INFO:teuthology.orchestra.run.smithi046.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 6.2 MB/s | 5.4 MB 00:00 2024-03-18T01:21:40.037 INFO:teuthology.orchestra.run.smithi046.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 5.9 MB/s | 2.4 MB 00:00 2024-03-18T01:21:40.179 INFO:teuthology.orchestra.run.smithi046.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 8.1 MB/s | 1.5 MB 00:00 2024-03-18T01:21:40.813 INFO:teuthology.orchestra.run.smithi046.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 6.1 MB/s | 4.7 MB 00:00 2024-03-18T01:21:40.826 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-18T01:21:41.472 INFO:teuthology.orchestra.run.smithi046.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 8.8 MB/s | 21 MB 00:02 2024-03-18T01:21:41.530 INFO:teuthology.orchestra.run.smithi046.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 432 kB/s | 25 kB 00:00 2024-03-18T01:21:41.839 INFO:teuthology.orchestra.run.smithi046.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 10 MB/s | 17 MB 00:01 2024-03-18T01:21:41.922 INFO:teuthology.orchestra.run.smithi046.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.6 MB/s | 221 kB 00:00 2024-03-18T01:21:41.989 INFO:teuthology.orchestra.run.smithi046.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 408 kB/s | 27 kB 00:00 2024-03-18T01:21:42.048 INFO:teuthology.orchestra.run.smithi046.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 543 kB/s | 32 kB 00:00 2024-03-18T01:21:42.239 INFO:teuthology.orchestra.run.smithi046.stdout:(14/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 6.9 MB/s | 9.8 MB 00:01 2024-03-18T01:21:42.265 INFO:teuthology.orchestra.run.smithi046.stdout:(15/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 3.4 MB/s | 747 kB 00:00 2024-03-18T01:21:42.298 INFO:teuthology.orchestra.run.smithi046.stdout:(16/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.7 MB/s | 163 kB 00:00 2024-03-18T01:21:42.332 INFO:teuthology.orchestra.run.smithi046.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 127 kB 00:00 2024-03-18T01:21:42.399 INFO:teuthology.orchestra.run.smithi046.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 4.9 MB/s | 501 kB 00:00 2024-03-18T01:21:42.474 INFO:teuthology.orchestra.run.smithi046.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 602 kB/s | 45 kB 00:00 2024-03-18T01:21:42.558 INFO:teuthology.orchestra.run.smithi046.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-18T01:21:42.641 INFO:teuthology.orchestra.run.smithi046.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 162 kB 00:00 2024-03-18T01:21:42.762 INFO:teuthology.orchestra.run.smithi046.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.6 MB/s | 320 kB 00:00 2024-03-18T01:21:42.842 INFO:teuthology.orchestra.run.smithi046.stdout:(23/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-18T01:21:42.867 INFO:teuthology.orchestra.run.smithi046.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.8 MB/s | 298 kB 00:00 2024-03-18T01:21:42.901 INFO:teuthology.orchestra.run.smithi046.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.7 MB/s | 99 kB 00:00 2024-03-18T01:21:42.926 INFO:teuthology.orchestra.run.smithi046.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-18T01:21:43.010 INFO:teuthology.orchestra.run.smithi046.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-18T01:21:43.093 INFO:teuthology.orchestra.run.smithi046.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 408 kB/s | 34 kB 00:00 2024-03-18T01:21:43.177 INFO:teuthology.orchestra.run.smithi046.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.6 MB/s | 139 kB 00:00 2024-03-18T01:21:43.227 INFO:teuthology.orchestra.run.smithi046.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 9.5 MB/s | 3.1 MB 00:00 2024-03-18T01:21:43.511 INFO:teuthology.orchestra.run.smithi046.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 5.4 MB/s | 1.8 MB 00:00 2024-03-18T01:21:43.603 INFO:teuthology.orchestra.run.smithi046.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.7 MB/s | 248 kB 00:00 2024-03-18T01:21:43.675 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T01:21:43.677 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T01:21:43.695 INFO:teuthology.orchestra.run.smithi046.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 539 kB/s | 49 kB 00:00 2024-03-18T01:21:43.700 INFO:teuthology.orchestra.run.smithi007.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T01:21:43.701 INFO:teuthology.orchestra.run.smithi007.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T01:21:43.703 INFO:teuthology.orchestra.run.smithi007.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T01:21:43.704 INFO:teuthology.orchestra.run.smithi007.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T01:21:43.707 INFO:teuthology.orchestra.run.smithi007.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T01:21:43.711 INFO:teuthology.orchestra.run.smithi007.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T01:21:43.712 INFO:teuthology.orchestra.run.smithi007.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T01:21:43.778 INFO:teuthology.orchestra.run.smithi046.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 197 kB/s | 16 kB 00:00 2024-03-18T01:21:43.810 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-18T01:21:43.827 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-18T01:21:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-18T01:21:43.828 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-18T01:21:43.828 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-18T01:21:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T01:21:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T01:21:43.828 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T01:21:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T01:21:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T01:21:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T01:21:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T01:21:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T01:21:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T01:21:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T01:21:43.829 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T01:21:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T01:21:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T01:21:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T01:21:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T01:21:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T01:21:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T01:21:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T01:21:43.830 INFO:teuthology.orchestra.run.smithi007.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T01:21:43.831 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T01:21:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T01:21:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T01:21:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T01:21:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T01:21:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T01:21:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T01:21:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T01:21:43.832 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T01:21:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T01:21:43.833 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-03-18T01:21:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T01:21:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T01:21:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T01:21:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T01:21:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T01:21:43.833 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T01:21:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T01:21:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T01:21:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T01:21:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T01:21:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T01:21:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T01:21:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T01:21:43.834 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T01:21:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T01:21:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T01:21:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T01:21:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T01:21:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T01:21:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T01:21:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T01:21:43.835 INFO:teuthology.orchestra.run.smithi007.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T01:21:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T01:21:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T01:21:43.836 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-18T01:21:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T01:21:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T01:21:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T01:21:43.836 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T01:21:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T01:21:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T01:21:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T01:21:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T01:21:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T01:21:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T01:21:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T01:21:43.837 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T01:21:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T01:21:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T01:21:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T01:21:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T01:21:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T01:21:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T01:21:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T01:21:43.838 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T01:21:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T01:21:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T01:21:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T01:21:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T01:21:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T01:21:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T01:21:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T01:21:43.839 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T01:21:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T01:21:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T01:21:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T01:21:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T01:21:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T01:21:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T01:21:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T01:21:43.840 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T01:21:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T01:21:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T01:21:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T01:21:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T01:21:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T01:21:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T01:21:43.841 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T01:21:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T01:21:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T01:21:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T01:21:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T01:21:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T01:21:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T01:21:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T01:21:43.842 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T01:21:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T01:21:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T01:21:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T01:21:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T01:21:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T01:21:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T01:21:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T01:21:43.843 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T01:21:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T01:21:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T01:21:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T01:21:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T01:21:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T01:21:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T01:21:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T01:21:43.844 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T01:21:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T01:21:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T01:21:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T01:21:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T01:21:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T01:21:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T01:21:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T01:21:43.845 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T01:21:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T01:21:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T01:21:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T01:21:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T01:21:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T01:21:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T01:21:43.846 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T01:21:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T01:21:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T01:21:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T01:21:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T01:21:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T01:21:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T01:21:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T01:21:43.847 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T01:21:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T01:21:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T01:21:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T01:21:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T01:21:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T01:21:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T01:21:43.848 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T01:21:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T01:21:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T01:21:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T01:21:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T01:21:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T01:21:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T01:21:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T01:21:43.849 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T01:21:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T01:21:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T01:21:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T01:21:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T01:21:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T01:21:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T01:21:43.850 INFO:teuthology.orchestra.run.smithi007.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T01:21:43.850 INFO:teuthology.orchestra.run.smithi007.stdout:Installing weak dependencies: 2024-03-18T01:21:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T01:21:43.851 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:21:43.851 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-18T01:21:43.851 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-18T01:21:43.852 INFO:teuthology.orchestra.run.smithi007.stdout:Install 150 Packages 2024-03-18T01:21:43.852 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 24 Packages 2024-03-18T01:21:43.852 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:21:43.853 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 224 M 2024-03-18T01:21:43.853 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-18T01:21:43.870 INFO:teuthology.orchestra.run.smithi046.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.8 MB/s | 264 kB 00:00 2024-03-18T01:21:43.962 INFO:teuthology.orchestra.run.smithi046.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 10 MB/s | 7.4 MB 00:00 2024-03-18T01:21:44.021 INFO:teuthology.orchestra.run.smithi046.stdout:(37/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.9 MB/s | 748 kB 00:00 2024-03-18T01:21:44.046 INFO:teuthology.orchestra.run.smithi046.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 219 kB/s | 18 kB 00:00 2024-03-18T01:21:44.230 INFO:teuthology.orchestra.run.smithi046.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 662 kB 00:00 2024-03-18T01:21:44.263 INFO:teuthology.orchestra.run.smithi046.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.2 MB/s | 106 kB 00:00 2024-03-18T01:21:44.289 INFO:teuthology.orchestra.run.smithi046.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 3.7 MB/s | 920 kB 00:00 2024-03-18T01:21:44.315 INFO:teuthology.orchestra.run.smithi046.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.9 MB/s | 307 kB 00:00 2024-03-18T01:21:44.340 INFO:teuthology.orchestra.run.smithi046.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 625 kB/s | 32 kB 00:00 2024-03-18T01:21:44.636 INFO:teuthology.orchestra.run.smithi007.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-03-18T01:21:44.791 INFO:teuthology.orchestra.run.smithi046.stdout:(44/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 15 MB/s | 48 MB 00:03 2024-03-18T01:21:44.841 INFO:teuthology.orchestra.run.smithi046.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.7 MB/s | 3.0 MB 00:00 2024-03-18T01:21:44.866 INFO:teuthology.orchestra.run.smithi046.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 33 kB/s | 17 kB 00:00 2024-03-18T01:21:44.886 INFO:teuthology.orchestra.run.smithi007.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 3.5 MB/s | 890 kB 00:00 2024-03-18T01:21:44.908 INFO:teuthology.orchestra.run.smithi046.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 88 kB/s | 10 kB 00:00 2024-03-18T01:21:44.933 INFO:teuthology.orchestra.run.smithi046.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 664 kB/s | 44 kB 00:00 2024-03-18T01:21:44.936 INFO:teuthology.orchestra.run.smithi007.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 2.9 MB/s | 145 kB 00:00 2024-03-18T01:21:44.959 INFO:teuthology.orchestra.run.smithi046.stdout:(49/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 159 kB 00:00 2024-03-18T01:21:44.984 INFO:teuthology.orchestra.run.smithi046.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-18T01:21:45.009 INFO:teuthology.orchestra.run.smithi046.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 7.7 MB/s | 578 kB 00:00 2024-03-18T01:21:45.034 INFO:teuthology.orchestra.run.smithi046.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 46 kB 00:00 2024-03-18T01:21:45.060 INFO:teuthology.orchestra.run.smithi046.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-18T01:21:45.085 INFO:teuthology.orchestra.run.smithi046.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-18T01:21:45.102 INFO:teuthology.orchestra.run.smithi046.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 833 kB/s | 34 kB 00:00 2024-03-18T01:21:45.127 INFO:teuthology.orchestra.run.smithi046.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-18T01:21:45.272 INFO:teuthology.orchestra.run.smithi007.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 7.1 MB/s | 2.4 MB 00:00 2024-03-18T01:21:45.327 INFO:teuthology.orchestra.run.smithi046.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-03-18T01:21:45.369 INFO:teuthology.orchestra.run.smithi046.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 5.7 MB/s | 1.5 MB 00:00 2024-03-18T01:21:45.395 INFO:teuthology.orchestra.run.smithi007.stdout:(5/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 5.3 MB/s | 5.4 MB 00:01 2024-03-18T01:21:45.403 INFO:teuthology.orchestra.run.smithi046.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 2.9 MB/s | 811 kB 00:00 2024-03-18T01:21:45.428 INFO:teuthology.orchestra.run.smithi046.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-18T01:21:45.529 INFO:teuthology.orchestra.run.smithi007.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-03-18T01:21:45.578 INFO:teuthology.orchestra.run.smithi046.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-18T01:21:45.654 INFO:teuthology.orchestra.run.smithi046.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-18T01:21:45.670 INFO:teuthology.orchestra.run.smithi046.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 323 kB/s | 78 kB 00:00 2024-03-18T01:21:45.804 INFO:teuthology.orchestra.run.smithi046.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-18T01:21:45.829 INFO:teuthology.orchestra.run.smithi046.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 157 kB/s | 24 kB 00:00 2024-03-18T01:21:45.871 INFO:teuthology.orchestra.run.smithi007.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 9.9 MB/s | 4.7 MB 00:00 2024-03-18T01:21:45.971 INFO:teuthology.orchestra.run.smithi046.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 61 kB/s | 18 kB 00:00 2024-03-18T01:21:45.996 INFO:teuthology.orchestra.run.smithi046.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 752 kB/s | 144 kB 00:00 2024-03-18T01:21:46.021 INFO:teuthology.orchestra.run.smithi046.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 636 kB/s | 32 kB 00:00 2024-03-18T01:21:46.147 INFO:teuthology.orchestra.run.smithi046.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-18T01:21:46.172 INFO:teuthology.orchestra.run.smithi046.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-18T01:21:46.197 INFO:teuthology.orchestra.run.smithi046.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-18T01:21:46.222 INFO:teuthology.orchestra.run.smithi046.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-03-18T01:21:46.247 INFO:teuthology.orchestra.run.smithi046.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-18T01:21:46.264 INFO:teuthology.orchestra.run.smithi046.stdout:(74/174): python3-prettytable-0.7.2-27.el9.noar 616 kB/s | 41 kB 00:00 2024-03-18T01:21:46.290 INFO:teuthology.orchestra.run.smithi046.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-18T01:21:46.315 INFO:teuthology.orchestra.run.smithi046.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-18T01:21:46.332 INFO:teuthology.orchestra.run.smithi046.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-18T01:21:46.357 INFO:teuthology.orchestra.run.smithi046.stdout:(78/174): python3-pytz-2021.1-5.el9.noarch.rpm 713 kB/s | 47 kB 00:00 2024-03-18T01:21:46.491 INFO:teuthology.orchestra.run.smithi046.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 245 kB/s | 39 kB 00:00 2024-03-18T01:21:46.525 INFO:teuthology.orchestra.run.smithi046.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 251 kB/s | 41 kB 00:00 2024-03-18T01:21:46.691 INFO:teuthology.orchestra.run.smithi046.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-18T01:21:46.716 INFO:teuthology.orchestra.run.smithi046.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-18T01:21:46.750 INFO:teuthology.orchestra.run.smithi046.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 253 kB/s | 56 kB 00:00 2024-03-18T01:21:46.775 INFO:teuthology.orchestra.run.smithi046.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 754 kB/s | 63 kB 00:00 2024-03-18T01:21:46.801 INFO:teuthology.orchestra.run.smithi046.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-18T01:21:46.821 INFO:teuthology.orchestra.run.smithi007.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 10 MB/s | 9.8 MB 00:00 2024-03-18T01:21:46.825 INFO:teuthology.orchestra.run.smithi046.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-18T01:21:46.851 INFO:teuthology.orchestra.run.smithi046.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-18T01:21:46.876 INFO:teuthology.orchestra.run.smithi046.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-18T01:21:46.888 INFO:teuthology.orchestra.run.smithi007.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 383 kB/s | 25 kB 00:00 2024-03-18T01:21:46.910 INFO:teuthology.orchestra.run.smithi046.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-18T01:21:46.935 INFO:teuthology.orchestra.run.smithi046.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-18T01:21:46.969 INFO:teuthology.orchestra.run.smithi046.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-18T01:21:47.036 INFO:teuthology.orchestra.run.smithi046.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-03-18T01:21:47.061 INFO:teuthology.orchestra.run.smithi046.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-03-18T01:21:47.094 INFO:teuthology.orchestra.run.smithi046.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2024-03-18T01:21:47.128 INFO:teuthology.orchestra.run.smithi046.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-18T01:21:47.153 INFO:teuthology.orchestra.run.smithi046.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 233 kB/s | 21 kB 00:00 2024-03-18T01:21:47.179 INFO:teuthology.orchestra.run.smithi046.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-03-18T01:21:47.205 INFO:teuthology.orchestra.run.smithi007.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 10 MB/s | 17 MB 00:01 2024-03-18T01:21:47.270 INFO:teuthology.orchestra.run.smithi046.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-18T01:21:47.280 INFO:teuthology.orchestra.run.smithi007.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.9 MB/s | 221 kB 00:00 2024-03-18T01:21:47.295 INFO:teuthology.orchestra.run.smithi046.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-18T01:21:47.347 INFO:teuthology.orchestra.run.smithi007.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 412 kB/s | 27 kB 00:00 2024-03-18T01:21:47.354 INFO:teuthology.orchestra.run.smithi046.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 600 kB/s | 35 kB 00:00 2024-03-18T01:21:47.379 INFO:teuthology.orchestra.run.smithi046.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.9 MB/s | 216 kB 00:00 2024-03-18T01:21:47.405 INFO:teuthology.orchestra.run.smithi046.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-18T01:21:47.431 INFO:teuthology.orchestra.run.smithi007.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 384 kB/s | 32 kB 00:00 2024-03-18T01:21:47.496 INFO:teuthology.orchestra.run.smithi046.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-03-18T01:21:47.688 INFO:teuthology.orchestra.run.smithi046.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 304 kB/s | 94 kB 00:00 2024-03-18T01:21:47.764 INFO:teuthology.orchestra.run.smithi046.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.2 MB/s | 863 kB 00:00 2024-03-18T01:21:47.820 INFO:teuthology.orchestra.run.smithi007.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 1.9 MB/s | 747 kB 00:00 2024-03-18T01:21:47.907 INFO:teuthology.orchestra.run.smithi007.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.9 MB/s | 163 kB 00:00 2024-03-18T01:21:47.973 INFO:teuthology.orchestra.run.smithi046.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-03-18T01:21:47.997 INFO:teuthology.orchestra.run.smithi046.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-18T01:21:48.157 INFO:teuthology.orchestra.run.smithi007.stdout:(16/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 5.5 MB/s | 21 MB 00:03 2024-03-18T01:21:48.182 INFO:teuthology.orchestra.run.smithi007.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 462 kB/s | 127 kB 00:00 2024-03-18T01:21:48.265 INFO:teuthology.orchestra.run.smithi007.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 4.6 MB/s | 501 kB 00:00 2024-03-18T01:21:48.341 INFO:teuthology.orchestra.run.smithi007.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 607 kB/s | 45 kB 00:00 2024-03-18T01:21:48.389 INFO:teuthology.orchestra.run.smithi046.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 47 kB/s | 19 kB 00:00 2024-03-18T01:21:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.4 MB/s | 131 kB 00:00 2024-03-18T01:21:48.465 INFO:teuthology.orchestra.run.smithi046.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-03-18T01:21:48.523 INFO:teuthology.orchestra.run.smithi046.stdout:(110/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.0 MB/s | 4.1 MB 00:01 2024-03-18T01:21:48.541 INFO:teuthology.orchestra.run.smithi007.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 162 kB 00:00 2024-03-18T01:21:48.548 INFO:teuthology.orchestra.run.smithi046.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 583 kB/s | 49 kB 00:00 2024-03-18T01:21:48.599 INFO:teuthology.orchestra.run.smithi046.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-18T01:21:48.641 INFO:teuthology.orchestra.run.smithi046.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 575 kB/s | 67 kB 00:00 2024-03-18T01:21:48.658 INFO:teuthology.orchestra.run.smithi007.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.7 MB/s | 320 kB 00:00 2024-03-18T01:21:48.775 INFO:teuthology.orchestra.run.smithi007.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.5 MB/s | 298 kB 00:00 2024-03-18T01:21:48.808 INFO:teuthology.orchestra.run.smithi046.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-18T01:21:48.866 INFO:teuthology.orchestra.run.smithi046.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 747 kB/s | 43 kB 00:00 2024-03-18T01:21:48.909 INFO:teuthology.orchestra.run.smithi007.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 747 kB/s | 99 kB 00:00 2024-03-18T01:21:48.917 INFO:teuthology.orchestra.run.smithi046.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-03-18T01:21:48.942 INFO:teuthology.orchestra.run.smithi046.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 429 kB/s | 32 kB 00:00 2024-03-18T01:21:48.967 INFO:teuthology.orchestra.run.smithi046.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-03-18T01:21:49.017 INFO:teuthology.orchestra.run.smithi007.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 800 kB/s | 85 kB 00:00 2024-03-18T01:21:49.051 INFO:teuthology.orchestra.run.smithi046.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-18T01:21:49.176 INFO:teuthology.orchestra.run.smithi046.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-03-18T01:21:49.201 INFO:teuthology.orchestra.run.smithi046.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.5 MB/s | 358 kB 00:00 2024-03-18T01:21:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-03-18T01:21:49.310 INFO:teuthology.orchestra.run.smithi046.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-18T01:21:49.394 INFO:teuthology.orchestra.run.smithi046.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-18T01:21:49.436 INFO:teuthology.orchestra.run.smithi046.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-03-18T01:21:49.486 INFO:teuthology.orchestra.run.smithi046.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 354 kB/s | 18 kB 00:00 2024-03-18T01:21:49.526 INFO:teuthology.orchestra.run.smithi007.stdout:(26/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 3.8 MB/s | 5.1 MB 00:01 2024-03-18T01:21:49.570 INFO:teuthology.orchestra.run.smithi046.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-03-18T01:21:49.609 INFO:teuthology.orchestra.run.smithi007.stdout:(27/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 5.2 MB/s | 3.1 MB 00:00 2024-03-18T01:21:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 390 kB/s | 19 kB 00:00 2024-03-18T01:21:49.643 INFO:teuthology.orchestra.run.smithi007.stdout:(28/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.4 MB/s | 170 kB 00:00 2024-03-18T01:21:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.4 MB/s | 2.0 MB 00:00 2024-03-18T01:21:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-03-18T01:21:49.829 INFO:teuthology.orchestra.run.smithi046.stdout:(131/174): libarrow-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-03-18T01:21:49.854 INFO:teuthology.orchestra.run.smithi046.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-03-18T01:21:49.896 INFO:teuthology.orchestra.run.smithi046.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-03-18T01:21:49.922 INFO:teuthology.orchestra.run.smithi046.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 863 kB/s | 58 kB 00:00 2024-03-18T01:21:49.947 INFO:teuthology.orchestra.run.smithi046.stdout:(135/174): python3-more-itertools-8.12.0-2.el9. 672 kB/s | 79 kB 00:00 2024-03-18T01:21:50.022 INFO:teuthology.orchestra.run.smithi046.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-18T01:21:50.048 INFO:teuthology.orchestra.run.smithi046.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 131 kB/s | 16 kB 00:00 2024-03-18T01:21:50.089 INFO:teuthology.orchestra.run.smithi046.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 633 kB/s | 90 kB 00:00 2024-03-18T01:21:50.115 INFO:teuthology.orchestra.run.smithi046.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-03-18T01:21:50.140 INFO:teuthology.orchestra.run.smithi046.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-18T01:21:50.169 INFO:teuthology.orchestra.run.smithi007.stdout:(29/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 15 MB/s | 48 MB 00:03 2024-03-18T01:21:50.173 INFO:teuthology.orchestra.run.smithi046.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-18T01:21:50.185 INFO:teuthology.orchestra.run.smithi007.stdout:(30/174): ceph-grafana-dashboards-19.0.0-2162.g 59 kB/s | 34 kB 00:00 2024-03-18T01:21:50.210 INFO:teuthology.orchestra.run.smithi007.stdout:(31/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 245 kB/s | 139 kB 00:00 2024-03-18T01:21:50.215 INFO:teuthology.orchestra.run.smithi046.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-18T01:21:50.234 INFO:teuthology.orchestra.run.smithi046.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 389 kB/s | 36 kB 00:00 2024-03-18T01:21:50.249 INFO:teuthology.orchestra.run.smithi046.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-03-18T01:21:50.308 INFO:teuthology.orchestra.run.smithi046.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-18T01:21:50.311 INFO:teuthology.orchestra.run.smithi007.stdout:(32/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 13 MB/s | 1.8 MB 00:00 2024-03-18T01:21:50.344 INFO:teuthology.orchestra.run.smithi007.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2162.g3d 1.8 MB/s | 248 kB 00:00 2024-03-18T01:21:50.370 INFO:teuthology.orchestra.run.smithi007.stdout:(34/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 857 kB/s | 49 kB 00:00 2024-03-18T01:21:50.375 INFO:teuthology.orchestra.run.smithi046.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 301 kB/s | 20 kB 00:00 2024-03-18T01:21:50.408 INFO:teuthology.orchestra.run.smithi007.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2162.g3 332 kB/s | 16 kB 00:00 2024-03-18T01:21:50.409 INFO:teuthology.orchestra.run.smithi046.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-03-18T01:21:50.428 INFO:teuthology.orchestra.run.smithi007.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 4.5 MB/s | 264 kB 00:00 2024-03-18T01:21:50.433 INFO:teuthology.orchestra.run.smithi046.stdout:(148/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-03-18T01:21:50.459 INFO:teuthology.orchestra.run.smithi046.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-18T01:21:50.487 INFO:teuthology.orchestra.run.smithi007.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 318 kB/s | 18 kB 00:00 2024-03-18T01:21:50.529 INFO:teuthology.orchestra.run.smithi007.stdout:(38/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 5.5 MB/s | 748 kB 00:00 2024-03-18T01:21:50.651 INFO:teuthology.orchestra.run.smithi046.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-18T01:21:50.718 INFO:teuthology.orchestra.run.smithi046.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-03-18T01:21:50.843 INFO:teuthology.orchestra.run.smithi046.stdout:(152/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 8.2 MB/s | 3.1 MB 00:00 2024-03-18T01:21:50.902 INFO:teuthology.orchestra.run.smithi046.stdout:(153/174): librados2-19.0.0-2162.g3d78f5ba.el9. 7.3 MB/s | 3.4 MB 00:00 2024-03-18T01:21:50.913 INFO:teuthology.orchestra.run.smithi007.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-03-18T01:21:50.943 INFO:teuthology.orchestra.run.smithi046.stdout:(154/174): NetworkManager-1.46.0-1.el9.x86_64.r 10 MB/s | 2.3 MB 00:00 2024-03-18T01:21:50.988 INFO:teuthology.orchestra.run.smithi007.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 920 kB 00:00 2024-03-18T01:21:50.993 INFO:teuthology.orchestra.run.smithi046.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 463 kB/s | 42 kB 00:00 2024-03-18T01:21:51.013 INFO:teuthology.orchestra.run.smithi007.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-03-18T01:21:51.036 INFO:teuthology.orchestra.run.smithi046.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.6 MB/s | 248 kB 00:00 2024-03-18T01:21:51.063 INFO:teuthology.orchestra.run.smithi007.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 307 kB 00:00 2024-03-18T01:21:51.069 INFO:teuthology.orchestra.run.smithi046.stdout:(157/174): NetworkManager-libnm-1.46.0-1.el9.x8 8.1 MB/s | 1.8 MB 00:00 2024-03-18T01:21:51.089 INFO:teuthology.orchestra.run.smithi007.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 424 kB/s | 32 kB 00:00 2024-03-18T01:21:51.094 INFO:teuthology.orchestra.run.smithi046.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 710 kB/s | 71 kB 00:00 2024-03-18T01:21:51.120 INFO:teuthology.orchestra.run.smithi046.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 436 kB/s | 36 kB 00:00 2024-03-18T01:21:51.162 INFO:teuthology.orchestra.run.smithi046.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-18T01:21:51.204 INFO:teuthology.orchestra.run.smithi046.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-18T01:21:51.237 INFO:teuthology.orchestra.run.smithi046.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-18T01:21:51.239 INFO:teuthology.orchestra.run.smithi007.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 7.0 MB/s | 7.4 MB 00:01 2024-03-18T01:21:51.262 INFO:teuthology.orchestra.run.smithi046.stdout:(163/174): libacl-2.3.1-4.el9.x86_64.rpm 162 kB/s | 23 kB 00:00 2024-03-18T01:21:51.273 INFO:teuthology.orchestra.run.smithi007.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 94 kB/s | 17 kB 00:00 2024-03-18T01:21:51.304 INFO:teuthology.orchestra.run.smithi046.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-18T01:21:51.330 INFO:teuthology.orchestra.run.smithi046.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 955 kB/s | 64 kB 00:00 2024-03-18T01:21:51.355 INFO:teuthology.orchestra.run.smithi046.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-18T01:21:51.380 INFO:teuthology.orchestra.run.smithi046.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 381 kB/s | 29 kB 00:00 2024-03-18T01:21:51.381 INFO:teuthology.orchestra.run.smithi007.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.4 MB/s | 3.0 MB 00:00 2024-03-18T01:21:51.407 INFO:teuthology.orchestra.run.smithi007.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 64 kB/s | 10 kB 00:00 2024-03-18T01:21:51.440 INFO:teuthology.orchestra.run.smithi007.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-18T01:21:51.456 INFO:teuthology.orchestra.run.smithi046.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-18T01:21:51.490 INFO:teuthology.orchestra.run.smithi007.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.0 MB/s | 661 kB 00:00 2024-03-18T01:21:51.524 INFO:teuthology.orchestra.run.smithi007.stdout:(50/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 385 kB/s | 44 kB 00:00 2024-03-18T01:21:51.531 INFO:teuthology.orchestra.run.smithi046.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-03-18T01:21:51.558 INFO:teuthology.orchestra.run.smithi007.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 578 kB 00:00 2024-03-18T01:21:51.583 INFO:teuthology.orchestra.run.smithi007.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 539 kB/s | 46 kB 00:00 2024-03-18T01:21:51.616 INFO:teuthology.orchestra.run.smithi007.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-18T01:21:51.669 INFO:teuthology.orchestra.run.smithi007.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-18T01:21:51.698 INFO:teuthology.orchestra.run.smithi046.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.2 MB/s | 2.3 MB 00:00 2024-03-18T01:21:51.700 INFO:teuthology.orchestra.run.smithi007.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-03-18T01:21:51.732 INFO:teuthology.orchestra.run.smithi046.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 1.6 MB/s | 465 kB 00:00 2024-03-18T01:21:51.742 INFO:teuthology.orchestra.run.smithi007.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 472 kB/s | 34 kB 00:00 2024-03-18T01:21:51.756 INFO:teuthology.orchestra.run.smithi046.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 138 kB/s | 31 kB 00:00 2024-03-18T01:21:51.832 INFO:teuthology.orchestra.run.smithi046.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-03-18T01:21:51.865 INFO:teuthology.orchestra.run.smithi046.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 557 kB/s | 74 kB 00:00 2024-03-18T01:21:51.869 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:21:51.869 INFO:teuthology.orchestra.run.smithi046.stdout:Total 17 MB/s | 224 MB 00:13 2024-03-18T01:21:51.945 INFO:teuthology.orchestra.run.smithi007.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-03-18T01:21:52.026 INFO:teuthology.orchestra.run.smithi007.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 4.9 MB/s | 1.5 MB 00:00 2024-03-18T01:21:52.059 INFO:teuthology.orchestra.run.smithi007.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 2.7 MB/s | 811 kB 00:00 2024-03-18T01:21:52.077 INFO:teuthology.orchestra.run.smithi007.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-03-18T01:21:52.177 INFO:teuthology.orchestra.run.smithi007.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.8 MB/s | 1.0 MB 00:00 2024-03-18T01:21:52.319 INFO:teuthology.orchestra.run.smithi007.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 22 MB/s | 5.7 MB 00:00 2024-03-18T01:21:52.344 INFO:teuthology.orchestra.run.smithi007.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 292 kB/s | 78 kB 00:00 2024-03-18T01:21:52.370 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-18T01:21:52.494 INFO:teuthology.orchestra.run.smithi007.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-18T01:21:52.511 INFO:teuthology.orchestra.run.smithi007.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 162 kB/s | 24 kB 00:00 2024-03-18T01:21:52.528 INFO:teuthology.orchestra.run.smithi007.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 108 kB/s | 18 kB 00:00 2024-03-18T01:21:52.561 INFO:teuthology.orchestra.run.smithi007.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 999 kB/s | 32 kB 00:00 2024-03-18T01:21:52.612 INFO:teuthology.orchestra.run.smithi007.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-18T01:21:52.673 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-18T01:21:52.673 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-18T01:21:52.762 INFO:teuthology.orchestra.run.smithi007.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-18T01:21:52.804 INFO:teuthology.orchestra.run.smithi007.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 998 kB/s | 217 kB 00:00 2024-03-18T01:21:52.829 INFO:teuthology.orchestra.run.smithi007.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 870 kB/s | 174 kB 00:00 2024-03-18T01:21:52.862 INFO:teuthology.orchestra.run.smithi007.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 2.1 MB/s | 69 kB 00:00 2024-03-18T01:21:52.879 INFO:teuthology.orchestra.run.smithi007.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 850 kB/s | 41 kB 00:00 2024-03-18T01:21:52.913 INFO:teuthology.orchestra.run.smithi007.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-18T01:21:52.946 INFO:teuthology.orchestra.run.smithi007.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-18T01:21:52.971 INFO:teuthology.orchestra.run.smithi007.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 211 kB 00:00 2024-03-18T01:21:52.988 INFO:teuthology.orchestra.run.smithi007.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 635 kB/s | 47 kB 00:00 2024-03-18T01:21:53.022 INFO:teuthology.orchestra.run.smithi007.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 876 kB/s | 43 kB 00:00 2024-03-18T01:21:53.072 INFO:teuthology.orchestra.run.smithi007.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 467 kB/s | 39 kB 00:00 2024-03-18T01:21:53.448 INFO:teuthology.orchestra.run.smithi007.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-18T01:21:53.481 INFO:teuthology.orchestra.run.smithi007.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 93 kB/s | 41 kB 00:00 2024-03-18T01:21:53.506 INFO:teuthology.orchestra.run.smithi007.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 729 kB/s | 300 kB 00:00 2024-03-18T01:21:53.557 INFO:teuthology.orchestra.run.smithi007.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-18T01:21:53.582 INFO:teuthology.orchestra.run.smithi007.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 851 kB/s | 63 kB 00:00 2024-03-18T01:21:53.607 INFO:teuthology.orchestra.run.smithi007.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 447 kB/s | 44 kB 00:00 2024-03-18T01:21:53.649 INFO:teuthology.orchestra.run.smithi007.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-18T01:21:53.691 INFO:teuthology.orchestra.run.smithi007.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.9 MB/s | 241 kB 00:00 2024-03-18T01:21:53.733 INFO:teuthology.orchestra.run.smithi007.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 9.3 MB/s | 1.1 MB 00:00 2024-03-18T01:21:53.783 INFO:teuthology.orchestra.run.smithi007.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 858 kB/s | 103 kB 00:00 2024-03-18T01:21:53.817 INFO:teuthology.orchestra.run.smithi007.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-18T01:21:53.842 INFO:teuthology.orchestra.run.smithi007.stdout:(91/174): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-03-18T01:21:53.959 INFO:teuthology.orchestra.run.smithi007.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-18T01:21:53.992 INFO:teuthology.orchestra.run.smithi007.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-18T01:21:54.026 INFO:teuthology.orchestra.run.smithi007.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.9 MB/s | 125 kB 00:00 2024-03-18T01:21:54.051 INFO:teuthology.orchestra.run.smithi007.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 373 kB/s | 21 kB 00:00 2024-03-18T01:21:54.076 INFO:teuthology.orchestra.run.smithi007.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 512 kB/s | 25 kB 00:00 2024-03-18T01:21:54.118 INFO:teuthology.orchestra.run.smithi007.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-18T01:21:54.152 INFO:teuthology.orchestra.run.smithi007.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-18T01:21:54.186 INFO:teuthology.orchestra.run.smithi007.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.7 MB/s | 248 kB 00:00 2024-03-18T01:21:54.219 INFO:teuthology.orchestra.run.smithi007.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-03-18T01:21:54.244 INFO:teuthology.orchestra.run.smithi007.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 605 kB/s | 35 kB 00:00 2024-03-18T01:21:54.270 INFO:teuthology.orchestra.run.smithi007.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 906 kB/s | 44 kB 00:00 2024-03-18T01:21:54.295 INFO:teuthology.orchestra.run.smithi007.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.9 MB/s | 94 kB 00:00 2024-03-18T01:21:54.345 INFO:teuthology.orchestra.run.smithi007.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 300 kB/s | 22 kB 00:00 2024-03-18T01:21:54.404 INFO:teuthology.orchestra.run.smithi007.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.8 MB/s | 863 kB 00:00 2024-03-18T01:21:54.514 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-18T01:21:54.515 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-18T01:21:54.520 INFO:teuthology.orchestra.run.smithi007.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.8 MB/s | 4.1 MB 00:00 2024-03-18T01:21:54.638 INFO:teuthology.orchestra.run.smithi007.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 168 kB/s | 19 kB 00:00 2024-03-18T01:21:54.729 INFO:teuthology.orchestra.run.smithi007.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 951 kB/s | 309 kB 00:00 2024-03-18T01:21:54.788 INFO:teuthology.orchestra.run.smithi007.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 452 kB/s | 26 kB 00:00 2024-03-18T01:21:54.822 INFO:teuthology.orchestra.run.smithi007.stdout:(110/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-18T01:21:54.864 INFO:teuthology.orchestra.run.smithi007.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 653 kB/s | 49 kB 00:00 2024-03-18T01:21:54.889 INFO:teuthology.orchestra.run.smithi007.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-18T01:21:54.981 INFO:teuthology.orchestra.run.smithi007.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-18T01:21:55.090 INFO:teuthology.orchestra.run.smithi007.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-03-18T01:21:55.131 INFO:teuthology.orchestra.run.smithi007.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-18T01:21:55.157 INFO:teuthology.orchestra.run.smithi007.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 658 kB/s | 43 kB 00:00 2024-03-18T01:21:55.191 INFO:teuthology.orchestra.run.smithi007.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 558 kB/s | 32 kB 00:00 2024-03-18T01:21:55.216 INFO:teuthology.orchestra.run.smithi007.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 259 kB/s | 15 kB 00:00 2024-03-18T01:21:55.250 INFO:teuthology.orchestra.run.smithi007.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-18T01:21:55.283 INFO:teuthology.orchestra.run.smithi007.stdout:(120/174): python3-asyncssh-2.13.2-4.el9.noarch 1.8 MB/s | 548 kB 00:00 2024-03-18T01:21:55.316 INFO:teuthology.orchestra.run.smithi007.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.5 MB/s | 303 kB 00:00 2024-03-18T01:21:55.350 INFO:teuthology.orchestra.run.smithi007.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-18T01:21:55.367 INFO:teuthology.orchestra.run.smithi007.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 425 kB/s | 35 kB 00:00 2024-03-18T01:21:55.401 INFO:teuthology.orchestra.run.smithi007.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-18T01:21:55.451 INFO:teuthology.orchestra.run.smithi007.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 218 kB/s | 11 kB 00:00 2024-03-18T01:21:55.494 INFO:teuthology.orchestra.run.smithi007.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-03-18T01:21:55.518 INFO:teuthology.orchestra.run.smithi007.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 267 kB/s | 18 kB 00:00 2024-03-18T01:21:55.577 INFO:teuthology.orchestra.run.smithi007.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-03-18T01:21:55.593 INFO:teuthology.orchestra.run.smithi007.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 235 kB/s | 23 kB 00:00 2024-03-18T01:21:55.610 INFO:teuthology.orchestra.run.smithi007.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 213 kB/s | 19 kB 00:00 2024-03-18T01:21:55.627 INFO:teuthology.orchestra.run.smithi007.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 398 kB/s | 20 kB 00:00 2024-03-18T01:21:55.661 INFO:teuthology.orchestra.run.smithi007.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 947 kB/s | 46 kB 00:00 2024-03-18T01:21:55.686 INFO:teuthology.orchestra.run.smithi007.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-18T01:21:55.720 INFO:teuthology.orchestra.run.smithi007.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.0 MB/s | 58 kB 00:00 2024-03-18T01:21:55.753 INFO:teuthology.orchestra.run.smithi007.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-18T01:21:55.778 INFO:teuthology.orchestra.run.smithi007.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 285 kB/s | 16 kB 00:00 2024-03-18T01:21:55.812 INFO:teuthology.orchestra.run.smithi007.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-18T01:21:55.837 INFO:teuthology.orchestra.run.smithi007.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 487 kB/s | 28 kB 00:00 2024-03-18T01:21:55.871 INFO:teuthology.orchestra.run.smithi007.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 531 kB/s | 31 kB 00:00 2024-03-18T01:21:55.905 INFO:teuthology.orchestra.run.smithi007.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-18T01:21:55.930 INFO:teuthology.orchestra.run.smithi007.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-18T01:21:55.963 INFO:teuthology.orchestra.run.smithi007.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 620 kB/s | 36 kB 00:00 2024-03-18T01:21:55.989 INFO:teuthology.orchestra.run.smithi007.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 881 kB/s | 51 kB 00:00 2024-03-18T01:21:56.039 INFO:teuthology.orchestra.run.smithi007.stdout:(144/174): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-03-18T01:21:56.089 INFO:teuthology.orchestra.run.smithi007.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 899 kB/s | 90 kB 00:00 2024-03-18T01:21:56.123 INFO:teuthology.orchestra.run.smithi007.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-03-18T01:21:56.157 INFO:teuthology.orchestra.run.smithi007.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 301 kB/s | 20 kB 00:00 2024-03-18T01:21:56.190 INFO:teuthology.orchestra.run.smithi007.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-18T01:21:56.249 INFO:teuthology.orchestra.run.smithi007.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.0 MB/s | 427 kB 00:00 2024-03-18T01:21:56.299 INFO:teuthology.orchestra.run.smithi007.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-18T01:21:56.349 INFO:teuthology.orchestra.run.smithi007.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-18T01:21:56.558 INFO:teuthology.orchestra.run.smithi007.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-18T01:21:56.658 INFO:teuthology.orchestra.run.smithi007.stdout:(153/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-03-18T01:21:56.753 INFO:teuthology.orchestra.run.smithi007.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 9.4 MB/s | 1.8 MB 00:00 2024-03-18T01:21:56.828 INFO:teuthology.orchestra.run.smithi007.stdout:(155/174): librados2-19.0.0-2162.g3d78f5ba.el9. 5.3 MB/s | 3.4 MB 00:00 2024-03-18T01:21:56.853 INFO:teuthology.orchestra.run.smithi007.stdout:(156/174): NetworkManager-team-1.46.0-1.el9.x86 219 kB/s | 42 kB 00:00 2024-03-18T01:21:56.887 INFO:teuthology.orchestra.run.smithi007.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.8 MB/s | 248 kB 00:00 2024-03-18T01:21:56.912 INFO:teuthology.orchestra.run.smithi007.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-18T01:21:56.938 INFO:teuthology.orchestra.run.smithi007.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 439 kB/s | 36 kB 00:00 2024-03-18T01:21:56.971 INFO:teuthology.orchestra.run.smithi007.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-18T01:21:56.996 INFO:teuthology.orchestra.run.smithi007.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 400 kB/s | 23 kB 00:00 2024-03-18T01:21:57.106 INFO:teuthology.orchestra.run.smithi007.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.3 MB/s | 256 kB 00:00 2024-03-18T01:21:57.130 INFO:teuthology.orchestra.run.smithi007.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 686 kB/s | 108 kB 00:00 2024-03-18T01:21:57.164 INFO:teuthology.orchestra.run.smithi007.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 932 kB/s | 155 kB 00:00 2024-03-18T01:21:57.189 INFO:teuthology.orchestra.run.smithi007.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-18T01:21:57.239 INFO:teuthology.orchestra.run.smithi007.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 385 kB/s | 29 kB 00:00 2024-03-18T01:21:57.265 INFO:teuthology.orchestra.run.smithi007.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 136 kB 00:00 2024-03-18T01:21:57.340 INFO:teuthology.orchestra.run.smithi007.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 1.2 MB 00:00 2024-03-18T01:21:57.409 INFO:teuthology.orchestra.run.smithi007.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.9 MB/s | 465 kB 00:00 2024-03-18T01:21:57.482 INFO:teuthology.orchestra.run.smithi007.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.9 MB/s | 2.2 MB 00:00 2024-03-18T01:21:57.508 INFO:teuthology.orchestra.run.smithi007.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 313 kB/s | 31 kB 00:00 2024-03-18T01:21:57.574 INFO:teuthology.orchestra.run.smithi007.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.4 MB/s | 2.3 MB 00:00 2024-03-18T01:21:57.599 INFO:teuthology.orchestra.run.smithi007.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 822 kB/s | 74 kB 00:00 2024-03-18T01:21:57.650 INFO:teuthology.orchestra.run.smithi007.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.5 MB/s | 1.1 MB 00:00 2024-03-18T01:21:57.658 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:21:57.658 INFO:teuthology.orchestra.run.smithi007.stdout:Total 16 MB/s | 224 MB 00:13 2024-03-18T01:21:59.158 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-18T01:21:59.559 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-18T01:21:59.559 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-18T01:22:01.773 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-18T01:22:01.886 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T01:22:01.920 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T01:22:01.959 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T01:22:01.969 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T01:22:01.986 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-18T01:22:01.986 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-18T01:22:02.010 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T01:22:02.063 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T01:22:02.096 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T01:22:02.130 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T01:22:02.172 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T01:22:02.509 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T01:22:02.572 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:22:02.680 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:22:02.832 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:22:02.907 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:22:02.936 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T01:22:02.976 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T01:22:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T01:22:03.226 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T01:22:03.253 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:22:03.413 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:22:03.438 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T01:22:03.485 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T01:22:03.525 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T01:22:03.548 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T01:22:03.564 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T01:22:03.733 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T01:22:03.764 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T01:22:03.795 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T01:22:03.895 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T01:22:03.973 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T01:22:04.033 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T01:22:04.107 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:22:04.159 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:22:04.385 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:22:04.496 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T01:22:04.600 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T01:22:04.640 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T01:22:04.666 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T01:22:04.710 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T01:22:04.788 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T01:22:04.840 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T01:22:04.877 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T01:22:04.954 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T01:22:05.001 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T01:22:05.045 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T01:22:05.077 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T01:22:05.105 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T01:22:05.178 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T01:22:05.210 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T01:22:05.254 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T01:22:05.282 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T01:22:05.357 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T01:22:05.405 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T01:22:05.483 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T01:22:05.638 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T01:22:05.692 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T01:22:06.258 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T01:22:06.340 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T01:22:06.424 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T01:22:06.464 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T01:22:06.473 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T01:22:06.504 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T01:22:06.545 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T01:22:06.583 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T01:22:06.612 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T01:22:06.851 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T01:22:07.015 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T01:22:07.090 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T01:22:07.135 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T01:22:07.269 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T01:22:07.329 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T01:22:07.385 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T01:22:07.459 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T01:22:07.732 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T01:22:07.778 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T01:22:07.827 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T01:22:07.860 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T01:22:07.904 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T01:22:07.971 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T01:22:08.117 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T01:22:08.167 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T01:22:08.196 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T01:22:08.238 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T01:22:08.338 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T01:22:08.371 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T01:22:08.411 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T01:22:08.436 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T01:22:08.461 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T01:22:08.536 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T01:22:08.669 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T01:22:08.785 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T01:22:08.829 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T01:22:08.887 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T01:22:08.950 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T01:22:09.144 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T01:22:09.200 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T01:22:09.347 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T01:22:09.389 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T01:22:09.421 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T01:22:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T01:22:09.545 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T01:22:09.727 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T01:22:09.761 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-18T01:22:09.848 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T01:22:09.876 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T01:22:10.011 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T01:22:10.040 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T01:22:10.040 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T01:22:10.093 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T01:22:10.113 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T01:22:10.139 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T01:22:10.150 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T01:22:10.184 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T01:22:10.226 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T01:22:10.227 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T01:22:10.268 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T01:22:10.335 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T01:22:10.362 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T01:22:10.396 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T01:22:10.434 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T01:22:10.461 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T01:22:10.508 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T01:22:10.526 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T01:22:10.547 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T01:22:10.596 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T01:22:10.705 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T01:22:10.758 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T01:22:10.958 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T01:22:11.006 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:22:11.166 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:22:11.201 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:22:11.344 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:22:11.394 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T01:22:11.470 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T01:22:11.533 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T01:22:11.579 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T01:22:11.613 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T01:22:11.663 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T01:22:11.753 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T01:22:11.771 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T01:22:11.789 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T01:22:11.823 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T01:22:11.848 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:22:11.975 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T01:22:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T01:22:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T01:22:12.261 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:22:12.292 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T01:22:12.382 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T01:22:12.428 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T01:22:12.460 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T01:22:12.528 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T01:22:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T01:22:12.890 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T01:22:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T01:22:12.935 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T01:22:12.986 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T01:22:13.351 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T01:22:13.543 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T01:22:13.639 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T01:22:13.816 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:22:13.888 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:22:14.197 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:22:14.218 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T01:22:14.296 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T01:22:14.326 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T01:22:14.357 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T01:22:14.385 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T01:22:14.389 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T01:22:14.516 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T01:22:14.552 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T01:22:14.588 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T01:22:14.647 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T01:22:14.714 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T01:22:14.748 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T01:22:14.761 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T01:22:14.830 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T01:22:14.839 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T01:22:14.869 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T01:22:14.878 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T01:22:14.907 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T01:22:14.993 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T01:22:15.069 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T01:22:15.139 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T01:22:15.192 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T01:22:15.230 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T01:22:15.354 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T01:22:15.391 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T01:22:15.403 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T01:22:15.434 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T01:22:15.458 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T01:22:15.498 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T01:22:15.506 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T01:22:15.536 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T01:22:15.604 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T01:22:15.666 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T01:22:15.769 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T01:22:15.950 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T01:22:16.046 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T01:22:16.776 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T01:22:16.882 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T01:22:17.009 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T01:22:17.093 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T01:22:17.121 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T01:22:17.170 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T01:22:17.241 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T01:22:17.294 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T01:22:17.335 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T01:22:17.754 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T01:22:17.901 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:22:17.941 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:22:17.982 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T01:22:18.116 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T01:22:18.162 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T01:22:18.364 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T01:22:18.411 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T01:22:18.489 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T01:22:18.508 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:22:18.544 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T01:22:18.585 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T01:22:18.626 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T01:22:18.626 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T01:22:18.626 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T01:22:18.626 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:18.636 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T01:22:18.654 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T01:22:18.677 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T01:22:18.746 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T01:22:18.793 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T01:22:18.861 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T01:22:18.925 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T01:22:19.128 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T01:22:19.182 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T01:22:19.220 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T01:22:19.278 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T01:22:19.423 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T01:22:19.469 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T01:22:19.516 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T01:22:19.551 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T01:22:19.584 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T01:22:19.777 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T01:22:19.953 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T01:22:20.107 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T01:22:20.179 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T01:22:20.247 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T01:22:20.328 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T01:22:20.588 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T01:22:20.658 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T01:22:20.835 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T01:22:20.888 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T01:22:20.938 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T01:22:20.975 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T01:22:21.017 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T01:22:21.064 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T01:22:21.130 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T01:22:21.198 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T01:22:21.249 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T01:22:21.411 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T01:22:21.454 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T01:22:21.490 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T01:22:21.555 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T01:22:21.601 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T01:22:21.751 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T01:22:21.894 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T01:22:21.947 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T01:22:21.991 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T01:22:22.050 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T01:22:22.194 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T01:22:22.266 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T01:22:23.271 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T01:22:23.323 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T01:22:23.357 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T01:22:23.434 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T01:22:23.559 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T01:22:23.599 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T01:22:23.851 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T01:22:23.887 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T01:22:23.982 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T01:22:24.835 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T01:22:24.979 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T01:22:26.590 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T01:22:26.701 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T01:22:26.739 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T01:22:26.769 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T01:22:26.810 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T01:22:27.204 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T01:22:27.258 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T01:22:27.382 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T01:22:27.415 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T01:22:27.472 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T01:22:28.033 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T01:22:28.060 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T01:22:28.161 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T01:22:28.188 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T01:22:30.790 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T01:22:30.790 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-03-18T01:22:30.790 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-03-18T01:22:30.790 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-03-18T01:22:30.791 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-03-18T01:22:30.791 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-03-18T01:22:30.791 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-03-18T01:22:30.791 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-03-18T01:22:30.791 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:30.892 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:22:30.934 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:22:30.959 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T01:22:30.976 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T01:22:30.976 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:30.976 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T01:22:30.976 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:31.591 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:22:31.629 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T01:22:31.801 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T01:22:31.801 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T01:22:31.801 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T01:22:31.801 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:31.872 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T01:22:32.504 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T01:22:32.531 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T01:22:32.532 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:32.532 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T01:22:32.532 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:22:32.532 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:22:32.532 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:32.763 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T01:22:32.787 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T01:22:32.787 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:32.787 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T01:22:32.787 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:22:32.787 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:22:32.788 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:33.256 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T01:22:33.280 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T01:22:33.280 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:33.280 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T01:22:33.281 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:22:33.281 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:22:33.281 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:33.374 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T01:22:33.828 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T01:22:33.856 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T01:22:33.889 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T01:22:33.951 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T01:22:33.979 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T01:22:34.233 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T01:22:34.263 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T01:22:34.904 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 275 kB/s | 19 MB 01:10 2024-03-18T01:22:35.284 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T01:22:35.314 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T01:22:35.424 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T01:22:35.592 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T01:22:35.615 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T01:22:35.640 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T01:22:35.640 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:35.640 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T01:22:35.641 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:22:35.641 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:22:35.641 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:35.683 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T01:22:35.694 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T01:22:35.741 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T01:22:36.660 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T01:22:36.693 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T01:22:36.717 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T01:22:36.718 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:36.718 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T01:22:36.718 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:22:36.718 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:22:36.718 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:36.761 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T01:22:36.784 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T01:22:36.784 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:36.784 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T01:22:36.785 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:36.836 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:22:36.860 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:22:36.860 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:36.861 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T01:22:36.861 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T01:22:36.861 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T01:22:36.861 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:37.188 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T01:22:37.212 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T01:22:37.212 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:37.212 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T01:22:37.212 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:22:37.212 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:22:37.212 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:40.038 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-18T01:22:41.397 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 16 kB 00:00 2024-03-18T01:22:41.917 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T01:22:41.959 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T01:22:41.988 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T01:22:42.016 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T01:22:42.207 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T01:22:42.338 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T01:22:42.349 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-18T01:22:42.381 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T01:22:42.420 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T01:22:42.472 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T01:22:42.515 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T01:22:42.548 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T01:22:42.580 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T01:22:42.611 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T01:22:42.665 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T01:22:42.725 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T01:22:42.758 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T01:22:43.327 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:22:43.375 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:22:43.395 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:22:44.307 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T01:22:44.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T01:22:44.429 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T01:22:44.513 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T01:22:44.549 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T01:22:44.574 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T01:22:44.599 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T01:22:44.626 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T01:22:44.626 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T01:22:44.714 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T01:22:44.743 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T01:22:44.743 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T01:22:44.837 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T01:22:44.862 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T01:22:44.904 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T01:22:44.930 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T01:22:44.954 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T01:22:44.989 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T01:22:45.027 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T01:22:45.068 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T01:22:45.093 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T01:22:45.118 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T01:22:45.143 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T01:22:45.169 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T01:22:45.176 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:22:45.176 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:22:45.363 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:22:45.363 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T01:22:45.446 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T01:22:45.471 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T01:22:45.496 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T01:22:45.530 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T01:22:45.564 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T01:22:45.633 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T01:22:45.739 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T01:22:45.739 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-03-18T01:22:45.739 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-03-18T01:22:45.739 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-03-18T01:22:45.739 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-03-18T01:22:45.740 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-03-18T01:22:45.740 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2024-03-18T01:22:45.740 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-03-18T01:22:45.740 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:45.999 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T01:22:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T01:22:46.047 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:46.047 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T01:22:46.047 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:47.340 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-18T01:22:47.716 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T01:22:47.820 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T01:22:47.821 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:47.821 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T01:22:47.821 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:22:47.821 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:22:47.821 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:48.083 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T01:22:48.180 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T01:22:48.180 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:48.180 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T01:22:48.180 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:22:48.180 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:22:48.180 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:48.692 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T01:22:48.776 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T01:22:48.777 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:48.777 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T01:22:48.777 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:22:48.777 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:22:48.777 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:48.902 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T01:22:49.520 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T01:22:49.554 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T01:22:49.608 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T01:22:49.711 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T01:22:49.740 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T01:22:49.785 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T01:22:49.785 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T01:22:49.798 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T01:22:49.798 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T01:22:49.799 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T01:22:49.800 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T01:22:49.802 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T01:22:49.804 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T01:22:49.805 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T01:22:49.886 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-18T01:22:49.909 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-18T01:22:49.909 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-18T01:22:49.909 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T01:22:49.910 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T01:22:49.911 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T01:22:49.912 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T01:22:49.913 INFO:teuthology.orchestra.run.smithi203.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T01:22:49.914 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T01:22:49.915 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T01:22:49.916 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T01:22:49.917 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T01:22:49.918 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T01:22:49.919 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T01:22:49.920 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T01:22:49.921 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T01:22:49.922 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout:Install 150 Packages 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 24 Packages 2024-03-18T01:22:49.923 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:22:49.924 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 224 M 2024-03-18T01:22:49.924 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-18T01:22:50.080 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T01:22:50.116 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T01:22:50.122 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T01:22:50.122 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T01:22:50.123 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T01:22:50.124 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T01:22:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T01:22:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T01:22:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T01:22:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T01:22:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T01:22:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T01:22:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T01:22:50.140 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T01:22:50.141 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T01:22:50.142 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T01:22:50.143 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T01:22:50.144 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T01:22:50.145 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T01:22:50.146 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T01:22:50.147 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T01:22:50.148 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T01:22:50.148 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T01:22:50.148 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T01:22:50.148 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T01:22:50.149 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T01:22:50.150 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T01:22:50.151 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T01:22:50.152 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T01:22:50.153 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T01:22:50.916 INFO:teuthology.orchestra.run.smithi203.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-18T01:22:51.293 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T01:22:51.334 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T01:22:51.485 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T01:22:51.600 INFO:teuthology.orchestra.run.smithi203.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.3 MB/s | 890 kB 00:00 2024-03-18T01:22:51.664 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T01:22:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T01:22:51.773 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T01:22:51.774 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:51.774 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T01:22:51.774 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:22:51.774 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:22:51.774 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:51.836 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T01:22:51.877 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T01:22:51.900 INFO:teuthology.orchestra.run.smithi203.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 482 kB/s | 145 kB 00:00 2024-03-18T01:22:52.000 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T01:22:53.015 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T01:22:53.049 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T01:22:53.121 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T01:22:53.121 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:53.121 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T01:22:53.121 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:22:53.121 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:22:53.121 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:53.166 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T01:22:53.238 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T01:22:53.238 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:53.238 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T01:22:53.238 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T01:22:53.250 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.251 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.252 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.253 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T01:22:53.254 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.6-16.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T01:22:53.255 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T01:22:53.256 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T01:22:53.256 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T01:22:53.256 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.256 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.256 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T01:22:53.256 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.256 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T01:22:53.257 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T01:22:53.258 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T01:22:53.259 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T01:22:53.260 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T01:22:53.261 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T01:22:53.262 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T01:22:53.263 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T01:22:53.264 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T01:22:53.265 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T01:22:53.265 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-18T01:22:53.265 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-18T01:22:53.317 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:22:53.396 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:22:53.396 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:53.397 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T01:22:53.397 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T01:22:53.397 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T01:22:53.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:53.459 INFO:teuthology.orchestra.run.smithi203.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 1.5 MB/s | 2.4 MB 00:01 2024-03-18T01:22:53.698 DEBUG:teuthology.parallel:result is None 2024-03-18T01:22:53.768 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T01:22:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T01:22:53.844 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:22:53.844 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T01:22:53.844 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:22:53.844 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:22:53.844 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:22:54.417 INFO:teuthology.orchestra.run.smithi203.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 1.6 MB/s | 1.5 MB 00:00 2024-03-18T01:22:55.917 INFO:teuthology.orchestra.run.smithi203.stdout:(6/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.0 MB/s | 5.4 MB 00:05 2024-03-18T01:22:56.501 INFO:teuthology.orchestra.run.smithi203.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 2.3 MB/s | 4.7 MB 00:02 2024-03-18T01:22:56.801 INFO:teuthology.orchestra.run.smithi203.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 3.3 MB/s | 21 MB 00:06 2024-03-18T01:22:56.918 INFO:teuthology.orchestra.run.smithi203.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 216 kB/s | 25 kB 00:00 2024-03-18T01:22:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T01:22:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T01:22:58.980 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T01:22:59.010 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T01:22:59.285 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T01:22:59.418 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T01:22:59.477 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T01:22:59.532 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T01:22:59.757 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T01:22:59.804 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T01:22:59.849 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T01:22:59.891 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T01:22:59.944 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T01:23:00.030 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T01:23:00.169 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T01:23:00.199 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T01:23:01.145 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:23:01.188 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:23:01.249 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:23:02.026 INFO:teuthology.orchestra.run.smithi203.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 1.8 MB/s | 9.8 MB 00:05 2024-03-18T01:23:02.259 INFO:teuthology.orchestra.run.smithi203.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 950 kB/s | 221 kB 00:00 2024-03-18T01:23:02.343 INFO:teuthology.orchestra.run.smithi203.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 324 kB/s | 27 kB 00:00 2024-03-18T01:23:02.410 INFO:teuthology.orchestra.run.smithi203.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 476 kB/s | 32 kB 00:00 2024-03-18T01:23:02.785 INFO:teuthology.orchestra.run.smithi203.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 1.9 MB/s | 747 kB 00:00 2024-03-18T01:23:02.829 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T01:23:02.919 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T01:23:02.944 INFO:teuthology.orchestra.run.smithi203.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.0 MB/s | 163 kB 00:00 2024-03-18T01:23:02.979 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T01:23:03.044 INFO:teuthology.orchestra.run.smithi203.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.2 MB/s | 127 kB 00:00 2024-03-18T01:23:03.112 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T01:23:03.155 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T01:23:03.189 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T01:23:03.223 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T01:23:03.374 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T01:23:03.374 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T01:23:03.486 INFO:teuthology.orchestra.run.smithi203.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 1.1 MB/s | 501 kB 00:00 2024-03-18T01:23:03.527 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T01:23:03.594 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T01:23:03.595 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T01:23:03.732 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T01:23:03.776 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T01:23:03.821 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T01:23:03.855 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T01:23:03.887 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T01:23:03.961 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T01:23:04.036 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T01:23:04.084 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T01:23:04.117 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T01:23:04.180 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T01:23:04.218 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T01:23:04.251 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T01:23:04.290 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:23:04.291 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:23:04.752 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:23:04.752 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T01:23:04.883 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T01:23:04.913 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T01:23:04.946 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T01:23:05.004 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T01:23:05.056 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T01:23:05.179 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T01:23:05.447 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T01:23:05.448 INFO:teuthology.orchestra.run.smithi007.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-18T01:23:05.448 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:23:06.202 INFO:teuthology.orchestra.run.smithi203.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 1.9 MB/s | 5.1 MB 00:02 2024-03-18T01:23:06.303 INFO:teuthology.orchestra.run.smithi203.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 452 kB/s | 45 kB 00:00 2024-03-18T01:23:06.395 INFO:teuthology.orchestra.run.smithi203.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.4 MB/s | 131 kB 00:00 2024-03-18T01:23:06.536 INFO:teuthology.orchestra.run.smithi203.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.1 MB/s | 162 kB 00:00 2024-03-18T01:23:06.745 INFO:teuthology.orchestra.run.smithi203.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 1.5 MB/s | 320 kB 00:00 2024-03-18T01:23:06.879 INFO:teuthology.orchestra.run.smithi203.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.2 MB/s | 298 kB 00:00 2024-03-18T01:23:06.954 INFO:teuthology.orchestra.run.smithi203.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 99 kB 00:00 2024-03-18T01:23:07.029 INFO:teuthology.orchestra.run.smithi203.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-18T01:23:08.413 INFO:teuthology.orchestra.run.smithi203.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 2.2 MB/s | 3.1 MB 00:01 2024-03-18T01:23:08.588 INFO:teuthology.orchestra.run.smithi203.stdout:(27/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 1.3 MB/s | 17 MB 00:12 2024-03-18T01:23:08.613 INFO:teuthology.orchestra.run.smithi203.stdout:(28/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 852 kB/s | 170 kB 00:00 2024-03-18T01:23:08.706 INFO:teuthology.orchestra.run.smithi203.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2162.g 291 kB/s | 34 kB 00:00 2024-03-18T01:23:08.731 INFO:teuthology.orchestra.run.smithi203.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.2 MB/s | 139 kB 00:00 2024-03-18T01:23:10.172 INFO:teuthology.orchestra.run.smithi203.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 1.2 MB/s | 1.8 MB 00:01 2024-03-18T01:23:10.455 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:23:10.456 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T01:23:10.457 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T01:23:10.458 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T01:23:10.459 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T01:23:10.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T01:23:10.461 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T01:23:10.462 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T01:23:10.463 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T01:23:10.464 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T01:23:10.465 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T01:23:10.466 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T01:23:10.467 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T01:23:10.468 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T01:23:10.469 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T01:23:10.470 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T01:23:10.471 INFO:teuthology.orchestra.run.smithi203.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 877 kB/s | 248 kB 00:00 2024-03-18T01:23:10.573 INFO:teuthology.orchestra.run.smithi203.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 423 kB/s | 49 kB 00:00 2024-03-18T01:23:10.673 INFO:teuthology.orchestra.run.smithi203.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 165 kB/s | 16 kB 00:00 2024-03-18T01:23:11.390 INFO:teuthology.orchestra.run.smithi203.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 368 kB/s | 264 kB 00:00 2024-03-18T01:23:11.482 INFO:teuthology.orchestra.run.smithi203.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 2.7 MB/s | 7.4 MB 00:02 2024-03-18T01:23:11.599 INFO:teuthology.orchestra.run.smithi203.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 157 kB/s | 18 kB 00:00 2024-03-18T01:23:11.774 INFO:teuthology.orchestra.run.smithi203.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 662 kB 00:00 2024-03-18T01:23:11.841 INFO:teuthology.orchestra.run.smithi203.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 13 MB/s | 920 kB 00:00 2024-03-18T01:23:11.875 INFO:teuthology.orchestra.run.smithi203.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-18T01:23:11.917 INFO:teuthology.orchestra.run.smithi203.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-18T01:23:11.950 INFO:teuthology.orchestra.run.smithi203.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 952 kB/s | 32 kB 00:00 2024-03-18T01:23:12.051 INFO:teuthology.orchestra.run.smithi203.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-18T01:23:12.084 INFO:teuthology.orchestra.run.smithi203.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 513 kB/s | 17 kB 00:00 2024-03-18T01:23:12.118 INFO:teuthology.orchestra.run.smithi203.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-03-18T01:23:12.160 INFO:teuthology.orchestra.run.smithi203.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-18T01:23:12.198 INFO:teuthology.orchestra.run.smithi203.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-18T01:23:12.248 INFO:teuthology.orchestra.run.smithi203.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T01:23:12.282 INFO:teuthology.orchestra.run.smithi203.stdout:(49/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 840 kB/s | 748 kB 00:00 2024-03-18T01:23:12.307 INFO:teuthology.orchestra.run.smithi203.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.6 MB/s | 578 kB 00:00 2024-03-18T01:23:12.324 INFO:teuthology.orchestra.run.smithi203.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-18T01:23:12.350 INFO:teuthology.orchestra.run.smithi203.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-18T01:23:12.375 INFO:teuthology.orchestra.run.smithi203.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T01:23:12.400 INFO:teuthology.orchestra.run.smithi203.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-18T01:23:12.426 INFO:teuthology.orchestra.run.smithi203.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2024-03-18T01:23:12.526 INFO:teuthology.orchestra.run.smithi203.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-18T01:23:12.568 INFO:teuthology.orchestra.run.smithi203.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-18T01:23:12.602 INFO:teuthology.orchestra.run.smithi203.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-18T01:23:12.619 INFO:teuthology.orchestra.run.smithi203.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-18T01:23:12.761 INFO:teuthology.orchestra.run.smithi203.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-18T01:23:12.794 INFO:teuthology.orchestra.run.smithi203.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-18T01:23:12.820 INFO:teuthology.orchestra.run.smithi203.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-18T01:23:12.887 INFO:teuthology.orchestra.run.smithi203.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 374 kB/s | 24 kB 00:00 2024-03-18T01:23:12.956 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T01:23:12.956 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:23:12.956 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-03-18T01:23:12.956 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.957 INFO:teuthology.orchestra.run.smithi007.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T01:23:12.958 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.959 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T01:23:12.960 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-16.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T01:23:12.961 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T01:23:12.962 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T01:23:12.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T01:23:12.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T01:23:12.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.966 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T01:23:12.967 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T01:23:12.968 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T01:23:12.969 INFO:teuthology.orchestra.run.smithi203.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-18T01:23:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T01:23:12.969 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:23:12.969 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-18T01:23:12.987 INFO:teuthology.orchestra.run.smithi203.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 184 kB/s | 18 kB 00:00 2024-03-18T01:23:13.013 INFO:teuthology.orchestra.run.smithi203.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-18T01:23:13.029 INFO:teuthology.orchestra.run.smithi203.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 757 kB/s | 32 kB 00:00 2024-03-18T01:23:13.155 INFO:teuthology.orchestra.run.smithi203.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-18T01:23:13.180 INFO:teuthology.orchestra.run.smithi203.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-18T01:23:13.206 INFO:teuthology.orchestra.run.smithi203.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-18T01:23:13.231 INFO:teuthology.orchestra.run.smithi203.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-18T01:23:13.247 INFO:teuthology.orchestra.run.smithi203.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 987 kB/s | 41 kB 00:00 2024-03-18T01:23:13.273 INFO:teuthology.orchestra.run.smithi203.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-18T01:23:13.298 INFO:teuthology.orchestra.run.smithi203.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T01:23:13.323 INFO:teuthology.orchestra.run.smithi203.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-18T01:23:13.340 INFO:teuthology.orchestra.run.smithi203.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-18T01:23:13.357 INFO:teuthology.orchestra.run.smithi203.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-18T01:23:13.458 INFO:teuthology.orchestra.run.smithi203.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 387 kB/s | 39 kB 00:00 2024-03-18T01:23:13.665 DEBUG:teuthology.parallel:result is None 2024-03-18T01:23:13.733 INFO:teuthology.orchestra.run.smithi203.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-18T01:23:13.750 INFO:teuthology.orchestra.run.smithi203.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 141 kB/s | 41 kB 00:00 2024-03-18T01:23:13.783 INFO:teuthology.orchestra.run.smithi203.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-18T01:23:13.800 INFO:teuthology.orchestra.run.smithi203.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-18T01:23:13.825 INFO:teuthology.orchestra.run.smithi203.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-18T01:23:13.842 INFO:teuthology.orchestra.run.smithi203.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-18T01:23:13.867 INFO:teuthology.orchestra.run.smithi203.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-18T01:23:13.893 INFO:teuthology.orchestra.run.smithi203.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-18T01:23:13.927 INFO:teuthology.orchestra.run.smithi203.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-18T01:23:13.952 INFO:teuthology.orchestra.run.smithi203.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-18T01:23:13.977 INFO:teuthology.orchestra.run.smithi203.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-18T01:23:14.003 INFO:teuthology.orchestra.run.smithi203.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-18T01:23:14.128 INFO:teuthology.orchestra.run.smithi203.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-18T01:23:14.178 INFO:teuthology.orchestra.run.smithi203.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-18T01:23:14.212 INFO:teuthology.orchestra.run.smithi203.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-03-18T01:23:14.237 INFO:teuthology.orchestra.run.smithi203.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-18T01:23:14.304 INFO:teuthology.orchestra.run.smithi203.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-18T01:23:14.330 INFO:teuthology.orchestra.run.smithi203.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-03-18T01:23:14.513 INFO:teuthology.orchestra.run.smithi203.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-18T01:23:14.687 INFO:teuthology.orchestra.run.smithi203.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-03-18T01:23:14.787 INFO:teuthology.orchestra.run.smithi203.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.1 MB/s | 216 kB 00:00 2024-03-18T01:23:14.846 INFO:teuthology.orchestra.run.smithi203.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 600 kB/s | 35 kB 00:00 2024-03-18T01:23:14.896 INFO:teuthology.orchestra.run.smithi203.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 894 kB/s | 44 kB 00:00 2024-03-18T01:23:14.938 INFO:teuthology.orchestra.run.smithi203.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-18T01:23:14.980 INFO:teuthology.orchestra.run.smithi203.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 533 kB/s | 22 kB 00:00 2024-03-18T01:23:15.164 INFO:teuthology.orchestra.run.smithi203.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 4.8 MB/s | 4.1 MB 00:00 2024-03-18T01:23:15.223 INFO:teuthology.orchestra.run.smithi203.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.5 MB/s | 863 kB 00:00 2024-03-18T01:23:15.440 INFO:teuthology.orchestra.run.smithi203.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-03-18T01:23:15.465 INFO:teuthology.orchestra.run.smithi203.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-18T01:23:15.499 INFO:teuthology.orchestra.run.smithi203.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 331 kB/s | 19 kB 00:00 2024-03-18T01:23:15.549 INFO:teuthology.orchestra.run.smithi203.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 522 kB/s | 26 kB 00:00 2024-03-18T01:23:15.599 INFO:teuthology.orchestra.run.smithi203.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-03-18T01:23:15.658 INFO:teuthology.orchestra.run.smithi203.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-18T01:23:15.725 INFO:teuthology.orchestra.run.smithi203.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-18T01:23:15.750 INFO:teuthology.orchestra.run.smithi203.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-18T01:23:15.792 INFO:teuthology.orchestra.run.smithi203.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-18T01:23:15.834 INFO:teuthology.orchestra.run.smithi203.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-03-18T01:23:15.859 INFO:teuthology.orchestra.run.smithi203.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 650 kB/s | 43 kB 00:00 2024-03-18T01:23:16.110 INFO:teuthology.orchestra.run.smithi203.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 117 kB/s | 32 kB 00:00 2024-03-18T01:23:16.135 INFO:teuthology.orchestra.run.smithi203.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 54 kB/s | 15 kB 00:00 2024-03-18T01:23:16.185 INFO:teuthology.orchestra.run.smithi203.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-18T01:23:16.219 INFO:teuthology.orchestra.run.smithi203.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-18T01:23:16.261 INFO:teuthology.orchestra.run.smithi203.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-18T01:23:16.286 INFO:teuthology.orchestra.run.smithi203.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 527 kB/s | 35 kB 00:00 2024-03-18T01:23:16.328 INFO:teuthology.orchestra.run.smithi203.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-18T01:23:16.387 INFO:teuthology.orchestra.run.smithi203.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-03-18T01:23:16.412 INFO:teuthology.orchestra.run.smithi203.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-18T01:23:16.446 INFO:teuthology.orchestra.run.smithi203.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-03-18T01:23:16.471 INFO:teuthology.orchestra.run.smithi203.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-03-18T01:23:16.505 INFO:teuthology.orchestra.run.smithi203.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-03-18T01:23:16.547 INFO:teuthology.orchestra.run.smithi203.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-03-18T01:23:16.572 INFO:teuthology.orchestra.run.smithi203.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 292 kB/s | 20 kB 00:00 2024-03-18T01:23:16.631 INFO:teuthology.orchestra.run.smithi203.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 794 kB/s | 46 kB 00:00 2024-03-18T01:23:16.664 INFO:teuthology.orchestra.run.smithi203.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-03-18T01:23:16.690 INFO:teuthology.orchestra.run.smithi203.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-18T01:23:16.723 INFO:teuthology.orchestra.run.smithi203.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-03-18T01:23:16.765 INFO:teuthology.orchestra.run.smithi203.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-18T01:23:16.791 INFO:teuthology.orchestra.run.smithi203.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-03-18T01:23:16.824 INFO:teuthology.orchestra.run.smithi203.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-18T01:23:16.850 INFO:teuthology.orchestra.run.smithi203.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 481 kB/s | 28 kB 00:00 2024-03-18T01:23:16.884 INFO:teuthology.orchestra.run.smithi203.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 526 kB/s | 31 kB 00:00 2024-03-18T01:23:16.909 INFO:teuthology.orchestra.run.smithi203.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-18T01:23:16.934 INFO:teuthology.orchestra.run.smithi203.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-03-18T01:23:16.960 INFO:teuthology.orchestra.run.smithi203.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-03-18T01:23:16.985 INFO:teuthology.orchestra.run.smithi203.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-18T01:23:17.018 INFO:teuthology.orchestra.run.smithi203.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-18T01:23:17.044 INFO:teuthology.orchestra.run.smithi203.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-18T01:23:17.086 INFO:teuthology.orchestra.run.smithi203.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-03-18T01:23:17.103 INFO:teuthology.orchestra.run.smithi203.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 340 kB/s | 20 kB 00:00 2024-03-18T01:23:17.145 INFO:teuthology.orchestra.run.smithi203.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-18T01:23:17.187 INFO:teuthology.orchestra.run.smithi203.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-18T01:23:18.388 INFO:teuthology.orchestra.run.smithi203.stdout:(150/174): ceph-test-19.0.0-2162.g3d78f5ba.el9. 2.2 MB/s | 48 MB 00:21 2024-03-18T01:23:18.429 INFO:teuthology.orchestra.run.smithi203.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-18T01:23:18.554 INFO:teuthology.orchestra.run.smithi203.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 2.4 MB/s | 3.4 MB 00:01 2024-03-18T01:23:18.663 INFO:teuthology.orchestra.run.smithi203.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 9.8 MB/s | 2.3 MB 00:00 2024-03-18T01:23:18.705 INFO:teuthology.orchestra.run.smithi203.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-18T01:23:18.755 INFO:teuthology.orchestra.run.smithi203.stdout:(155/174): NetworkManager-libnm-1.46.0-1.el9.x8 9.1 MB/s | 1.8 MB 00:00 2024-03-18T01:23:18.781 INFO:teuthology.orchestra.run.smithi203.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 3.2 MB/s | 248 kB 00:00 2024-03-18T01:23:18.806 INFO:teuthology.orchestra.run.smithi203.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-18T01:23:18.831 INFO:teuthology.orchestra.run.smithi203.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 723 kB/s | 36 kB 00:00 2024-03-18T01:23:18.857 INFO:teuthology.orchestra.run.smithi203.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.8 MB/s | 192 kB 00:00 2024-03-18T01:23:18.882 INFO:teuthology.orchestra.run.smithi203.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-18T01:23:18.907 INFO:teuthology.orchestra.run.smithi203.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-18T01:23:18.933 INFO:teuthology.orchestra.run.smithi203.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-18T01:23:18.958 INFO:teuthology.orchestra.run.smithi203.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-18T01:23:18.983 INFO:teuthology.orchestra.run.smithi203.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-18T01:23:19.000 INFO:teuthology.orchestra.run.smithi203.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-03-18T01:23:19.017 INFO:teuthology.orchestra.run.smithi203.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 856 kB/s | 29 kB 00:00 2024-03-18T01:23:19.067 INFO:teuthology.orchestra.run.smithi203.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 18 MB/s | 1.2 MB 00:00 2024-03-18T01:23:19.184 INFO:teuthology.orchestra.run.smithi203.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 13 MB/s | 2.2 MB 00:00 2024-03-18T01:23:19.235 INFO:teuthology.orchestra.run.smithi203.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-03-18T01:23:19.260 INFO:teuthology.orchestra.run.smithi203.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.0 MB/s | 465 kB 00:00 2024-03-18T01:23:19.277 INFO:teuthology.orchestra.run.smithi203.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 744 kB/s | 31 kB 00:00 2024-03-18T01:23:19.327 INFO:teuthology.orchestra.run.smithi203.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-18T01:23:19.395 INFO:teuthology.orchestra.run.smithi203.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 636 kB/s | 74 kB 00:00 2024-03-18T01:23:20.353 INFO:teuthology.orchestra.run.smithi203.stdout:(174/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 1.0 MB/s | 3.1 MB 00:03 2024-03-18T01:23:20.357 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:23:20.357 INFO:teuthology.orchestra.run.smithi203.stdout:Total 7.4 MB/s | 224 MB 00:30 2024-03-18T01:23:20.814 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-18T01:23:21.083 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-18T01:23:21.083 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-18T01:23:22.794 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-18T01:23:22.795 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-18T01:23:29.647 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-18T01:23:29.748 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T01:23:29.782 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T01:23:29.805 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T01:23:29.814 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T01:23:29.863 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T01:23:29.917 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T01:23:29.959 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T01:23:29.993 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T01:23:30.035 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T01:23:30.355 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T01:23:30.419 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:23:30.524 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:23:30.562 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:23:30.652 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:23:30.683 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T01:23:30.722 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T01:23:30.901 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T01:23:30.948 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T01:23:30.976 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:23:31.063 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:23:31.085 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T01:23:31.133 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T01:23:31.181 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T01:23:31.205 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T01:23:31.219 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T01:23:31.403 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T01:23:31.435 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T01:23:31.475 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T01:23:31.572 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T01:23:31.695 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T01:23:31.739 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T01:23:31.811 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:23:31.873 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:23:32.085 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T01:23:32.197 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T01:23:32.291 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T01:23:32.337 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T01:23:32.363 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T01:23:32.399 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T01:23:32.473 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T01:23:32.527 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T01:23:32.566 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T01:23:32.630 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T01:23:32.665 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T01:23:32.709 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T01:23:32.741 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T01:23:32.770 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T01:23:32.838 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T01:23:32.899 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T01:23:32.943 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T01:23:32.972 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T01:23:33.041 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T01:23:33.094 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T01:23:33.168 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T01:23:33.302 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T01:23:33.364 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T01:23:33.886 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T01:23:33.968 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T01:23:34.046 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T01:23:34.081 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T01:23:34.089 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T01:23:34.120 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T01:23:34.161 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T01:23:34.199 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T01:23:34.229 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T01:23:34.463 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T01:23:34.653 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T01:23:34.732 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T01:23:34.776 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T01:23:34.904 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T01:23:34.963 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T01:23:35.027 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T01:23:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T01:23:35.139 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T01:23:35.176 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T01:23:35.223 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T01:23:35.259 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T01:23:35.302 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T01:23:35.344 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T01:23:35.480 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T01:23:35.525 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T01:23:35.555 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T01:23:35.597 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T01:23:35.692 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T01:23:35.730 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T01:23:35.771 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T01:23:35.796 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T01:23:35.821 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T01:23:35.893 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T01:23:36.032 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T01:23:36.139 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T01:23:36.180 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T01:23:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T01:23:36.285 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T01:23:36.470 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T01:23:36.518 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T01:23:36.648 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T01:23:36.683 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T01:23:36.715 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T01:23:36.746 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T01:23:36.774 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T01:23:36.809 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T01:23:36.848 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T01:23:36.880 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T01:23:36.935 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T01:23:37.012 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T01:23:37.045 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T01:23:37.078 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T01:23:37.120 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T01:23:37.155 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T01:23:37.245 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T01:23:37.336 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T01:23:37.386 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T01:23:37.418 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T01:23:37.465 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T01:23:37.564 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T01:23:37.611 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T01:23:38.360 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T01:23:38.409 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T01:23:38.451 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T01:23:38.499 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T01:23:38.587 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T01:23:38.618 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T01:23:38.788 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T01:23:38.816 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T01:23:38.894 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T01:23:39.538 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T01:23:39.648 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T01:23:40.885 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T01:23:40.986 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T01:23:41.017 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T01:23:41.048 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T01:23:41.076 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T01:23:41.386 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T01:23:41.414 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T01:23:41.495 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T01:23:41.528 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T01:23:41.564 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T01:23:42.022 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T01:23:42.214 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T01:23:42.572 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T01:23:42.603 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T01:23:44.861 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:23:44.893 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:23:45.428 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T01:23:45.464 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T01:23:45.536 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T01:23:45.536 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T01:23:45.536 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T01:23:45.536 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:23:45.557 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T01:23:57.378 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T01:23:57.378 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-03-18T01:23:57.378 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-03-18T01:23:57.378 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-03-18T01:23:57.378 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-03-18T01:23:57.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-03-18T01:23:57.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-03-18T01:23:57.379 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-03-18T01:23:57.379 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:23:57.542 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T01:23:57.556 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T01:23:57.557 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:23:57.557 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T01:23:57.557 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:23:59.020 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T01:23:59.045 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T01:23:59.045 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:23:59.045 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T01:23:59.046 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:23:59.046 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T01:23:59.046 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:23:59.262 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T01:23:59.284 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T01:23:59.285 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:23:59.285 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T01:23:59.285 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:23:59.285 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T01:23:59.285 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:23:59.722 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T01:23:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T01:23:59.744 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:23:59.744 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T01:23:59.744 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:23:59.744 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T01:23:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:23:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T01:24:00.276 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T01:24:00.306 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T01:24:00.338 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T01:24:00.398 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T01:24:00.429 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T01:24:00.661 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T01:24:00.697 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T01:24:01.656 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T01:24:01.680 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T01:24:01.781 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T01:24:01.942 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T01:24:01.974 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T01:24:01.997 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T01:24:01.997 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:24:01.997 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T01:24:01.997 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:24:01.997 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T01:24:01.998 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:24:02.042 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T01:24:02.052 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T01:24:02.098 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T01:24:02.966 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T01:24:02.994 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T01:24:03.017 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T01:24:03.017 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:24:03.017 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T01:24:03.017 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:24:03.017 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T01:24:03.017 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:24:03.062 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T01:24:03.083 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T01:24:03.083 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:24:03.084 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T01:24:03.084 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:24:03.130 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:24:03.151 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:24:03.151 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:24:03.151 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T01:24:03.152 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T01:24:03.152 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T01:24:03.152 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:24:03.463 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T01:24:03.486 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T01:24:03.486 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T01:24:03.486 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T01:24:03.486 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:24:03.486 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T01:24:03.487 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:24:07.902 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T01:24:07.951 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T01:24:07.981 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T01:24:08.009 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T01:24:08.192 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T01:24:08.317 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T01:24:08.357 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T01:24:08.397 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T01:24:08.457 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T01:24:08.501 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T01:24:08.525 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T01:24:08.574 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T01:24:08.604 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T01:24:08.650 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T01:24:08.721 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T01:24:08.753 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T01:24:09.362 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:24:09.402 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:24:09.422 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T01:24:10.258 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T01:24:10.342 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T01:24:10.380 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T01:24:10.461 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T01:24:10.494 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T01:24:10.519 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T01:24:10.545 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T01:24:10.571 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T01:24:10.571 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T01:24:10.659 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T01:24:10.697 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T01:24:10.697 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T01:24:10.782 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T01:24:10.808 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T01:24:10.841 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T01:24:10.867 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T01:24:10.892 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T01:24:10.925 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T01:24:10.965 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T01:24:10.997 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T01:24:11.023 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T01:24:11.048 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T01:24:11.074 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T01:24:11.099 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T01:24:11.105 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:24:11.105 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:24:11.293 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T01:24:11.293 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T01:24:11.368 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T01:24:11.393 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T01:24:11.418 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T01:24:11.451 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T01:24:11.478 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T01:24:11.542 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T01:24:16.427 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T01:24:16.428 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T01:24:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T01:24:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T01:24:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T01:24:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T01:24:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T01:24:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T01:24:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T01:24:16.429 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T01:24:16.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T01:24:16.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T01:24:16.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T01:24:16.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T01:24:16.447 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T01:24:16.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T01:24:16.449 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T01:24:16.450 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T01:24:16.451 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T01:24:16.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T01:24:16.453 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T01:24:16.454 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T01:24:16.455 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T01:24:16.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T01:24:16.457 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T01:24:16.458 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T01:24:19.376 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T01:24:19.377 INFO:teuthology.orchestra.run.smithi203.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T01:24:19.378 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.379 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.380 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T01:24:19.381 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.382 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T01:24:19.383 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.384 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T01:24:19.385 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T01:24:19.386 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.387 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T01:24:19.388 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T01:24:19.389 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T01:24:19.390 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T01:24:19.390 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:24:19.390 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-18T01:24:19.891 DEBUG:teuthology.parallel:result is None 2024-03-18T01:24:19.891 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:24:20.122 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T01:24:20.200 INFO:teuthology.orchestra.run.smithi007.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T01:24:20.200 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T01:24:20.201 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T01:24:20.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:24:20.341 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T01:24:20.368 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T01:24:20.368 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T01:24:20.368 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T01:24:20.370 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:24:20.510 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T01:24:20.535 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T01:24:20.535 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T01:24:20.535 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T01:24:20.537 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T01:24:20.538 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:24:20.538 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T01:24:20.649 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:24:20.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T01:24:20.688 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:24:20.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T01:24:20.723 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T01:24:20.723 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:24:20.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T01:24:20.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T01:24:20.973 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:24:20.974 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T01:24:21.010 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T01:24:21.084 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:24:21.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T01:24:21.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T01:24:21.186 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T01:24:21.186 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:24:21.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T01:24:21.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T01:24:21.439 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:24:21.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T01:24:21.470 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T01:24:21.540 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:24:21.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T01:24:21.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T01:24:21.642 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T01:24:21.643 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:24:21.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T01:24:21.742 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T01:24:21.838 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:24:21.838 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T01:24:21.868 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T01:24:21.938 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:24:21.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T01:24:21.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T01:24:22.040 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T01:24:22.135 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2}} 2024-03-18T01:24:22.135 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:24:22.135 INFO:tasks.cephadm:Cluster fsid is 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:24:22.136 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T01:24:22.136 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.b': '172.21.15.46', 'mon.c': '172.21.15.203'} 2024-03-18T01:24:22.136 INFO:tasks.cephadm:First mon is mon.a on smithi007 2024-03-18T01:24:22.136 INFO:tasks.cephadm:First mgr is x 2024-03-18T01:24:22.136 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T01:24:22.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2024-03-18T01:24:22.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo hostname $(hostname -s) 2024-03-18T01:24:22.275 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-03-18T01:24:22.308 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-18T01:24:22.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:24:22.521 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-18T01:24:22.661 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:24:22.663 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-18T01:24:22.663 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:24.002 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 01:24 /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:24.003 DEBUG:teuthology.orchestra.run.smithi046:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:24.524 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 01:24 /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:24.525 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:25.030 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 01:24 /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:25.030 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:25.093 DEBUG:teuthology.orchestra.run.smithi046:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:25.112 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T01:24:25.150 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-18T01:24:25.150 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T01:24:25.157 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T01:24:25.159 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T01:24:25.937 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T01:24:25.960 INFO:teuthology.orchestra.run.smithi046.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T01:24:26.262 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T01:25:21.740 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-18T01:25:21.741 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T01:25:21.741 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T01:25:21.741 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-03-18T01:25:21.741 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T01:25:21.741 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-03-18T01:25:21.741 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-18T01:25:25.823 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:25:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T01:25:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T01:25:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2024-03-18T01:25:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T01:25:25.824 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:25:25.824 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:25:27.839 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-18T01:25:27.839 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T01:25:27.839 INFO:teuthology.orchestra.run.smithi046.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T01:25:27.840 INFO:teuthology.orchestra.run.smithi046.stdout: "repo_digests": [ 2024-03-18T01:25:27.840 INFO:teuthology.orchestra.run.smithi046.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T01:25:27.840 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-18T01:25:27.840 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-18T01:25:27.865 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2024-03-18T01:25:27.975 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph 2024-03-18T01:25:28.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-03-18T01:25:28.044 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2024-03-18T01:25:28.149 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /etc/ceph 2024-03-18T01:25:28.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-03-18T01:25:28.209 INFO:tasks.cephadm:Writing seed config... 2024-03-18T01:25:28.210 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T01:25:28.210 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T01:25:28.210 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T01:25:28.210 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T01:25:28.210 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T01:25:28.210 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-18T01:25:28.211 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T01:25:28.212 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T01:25:28.213 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-03-18T01:25:28.213 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T01:25:28.213 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T01:25:28.213 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-18T01:25:28.213 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-18T01:25:28.213 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:25:28.214 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T01:25:28.274 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T01:25:28.275 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a.service 2024-03-18T01:25:28.322 DEBUG:teuthology.orchestra.run.smithi007:mgr.x> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.x.service 2024-03-18T01:25:28.368 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T01:25:28.368 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:25:28.647 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-18T01:25:28.647 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', '3fdfdc7c-e4c6-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2024-03-18T01:25:28.709 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T01:25:28.710 INFO:teuthology.orchestra.run.smithi007.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T01:25:28.710 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying podman|docker is present... 2024-03-18T01:25:28.768 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T01:25:28.768 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying lvm2 is present... 2024-03-18T01:25:28.768 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying time synchronization is in place... 2024-03-18T01:25:28.783 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T01:25:28.784 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T01:25:28.798 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T01:25:28.799 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-18T01:25:28.819 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-03-18T01:25:28.839 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-03-18T01:25:28.839 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-03-18T01:25:28.839 INFO:teuthology.orchestra.run.smithi007.stdout:Repeating the final host check... 2024-03-18T01:25:28.902 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T01:25:28.903 INFO:teuthology.orchestra.run.smithi007.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-18T01:25:28.903 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl is present 2024-03-18T01:25:28.903 INFO:teuthology.orchestra.run.smithi007.stdout:lvcreate is present 2024-03-18T01:25:28.920 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T01:25:28.920 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T01:25:28.938 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T01:25:28.939 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-18T01:25:28.961 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout enabled 2024-03-18T01:25:28.983 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout active 2024-03-18T01:25:28.984 INFO:teuthology.orchestra.run.smithi007.stdout:Unit chronyd.service is enabled and running 2024-03-18T01:25:28.984 INFO:teuthology.orchestra.run.smithi007.stdout:Host looks OK 2024-03-18T01:25:28.984 INFO:teuthology.orchestra.run.smithi007.stdout:Cluster fsid: 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:28.984 INFO:teuthology.orchestra.run.smithi007.stdout:Acquiring lock 140468437663168 on /run/cephadm/3fdfdc7c-e4c6-11ee-95c9-87774f69a715.lock 2024-03-18T01:25:28.985 INFO:teuthology.orchestra.run.smithi007.stdout:Lock 140468437663168 acquired on /run/cephadm/3fdfdc7c-e4c6-11ee-95c9-87774f69a715.lock 2024-03-18T01:25:28.986 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 3300 ... 2024-03-18T01:25:28.987 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying IP 172.21.15.7 port 6789 ... 2024-03-18T01:25:28.988 INFO:teuthology.orchestra.run.smithi007.stdout:Base mon IP(s) is [172.21.15.7:3300, 172.21.15.7:6789], mon addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-03-18T01:25:29.004 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.7 metric 100 2024-03-18T01:25:29.004 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 metric 100 2024-03-18T01:25:29.016 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T01:25:29.017 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-18T01:25:29.017 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-18T01:25:29.029 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T01:25:29.029 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T01:25:29.029 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T01:25:29.030 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-18T01:25:29.030 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-03-18T01:25:29.030 INFO:teuthology.orchestra.run.smithi007.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T01:25:29.033 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-03-18T01:25:29.033 INFO:teuthology.orchestra.run.smithi007.stdout:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-03-18T01:25:29.033 INFO:teuthology.orchestra.run.smithi007.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T01:25:29.036 INFO:teuthology.orchestra.run.smithi007.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T01:25:29.038 INFO:teuthology.orchestra.run.smithi007.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T01:25:29.732 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T01:25:29.732 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T01:25:29.732 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T01:25:29.732 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-18T01:25:29.733 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T01:25:29.733 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T01:25:29.733 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T01:25:30.908 INFO:teuthology.orchestra.run.smithi007.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T01:25:30.908 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T01:25:30.908 INFO:teuthology.orchestra.run.smithi007.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T01:25:32.182 INFO:teuthology.orchestra.run.smithi007.stdout:stat: stdout 167 167 2024-03-18T01:25:32.186 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial keys... 2024-03-18T01:25:33.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQAMmPdlZX2fJhAAzeol9P/w/WEUjouqdzOdwg== 2024-03-18T01:25:34.484 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQANmPdlj6HqMhAAXizrC6g74011jk3EvnE+MA== 2024-03-18T01:25:35.536 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph-authtool: stdout AQAOmPdlYSBRNxAAAf4pnfPAYyv2pkGqt77mAQ== 2024-03-18T01:25:35.539 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial monmap... 2024-03-18T01:25:36.671 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T01:25:36.671 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-18T01:25:36.671 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:36.671 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T01:25:36.671 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T01:25:36.671 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-03-18T01:25:36.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: set fsid to 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:36.672 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T01:25:36.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:25:36.672 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mon... 2024-03-18T01:25:46.152 INFO:teuthology.orchestra.run.smithi007.stdout:create mon.a on 2024-03-18T01:25:46.520 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-18T01:25:46.768 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T01:25:47.061 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715.target. 2024-03-18T01:25:47.062 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715.target. 2024-03-18T01:25:47.454 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a 2024-03-18T01:25:47.455 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-18T01:25:47.749 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715.target.wants/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@.service. 2024-03-18T01:25:48.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:48 smithi007 podman[31612]: 2024-03-18 01:25:48.395344482 +0000 UTC m=+0.374105194 container start 9fa81d1f10c659fb3b23dd705cd9bfb070bdb474612c8d4f0e4cff848d807253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-18T01:25:48.552 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T01:25:48.552 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-03-18T01:25:48.574 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T01:25:48.574 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-18T01:25:48.574 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-03-18T01:25:48.575 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T01:25:48.575 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon to start... 2024-03-18T01:25:48.575 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mon... 2024-03-18T01:25:48.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:48 smithi007 ceph-mon[31631]: mkfs 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:48.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:48 smithi007 ceph-mon[31631]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T01:25:48.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:48 smithi007 bash[31612]: 9fa81d1f10c659fb3b23dd705cd9bfb070bdb474612c8d4f0e4cff848d807253 2024-03-18T01:25:48.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:48 smithi007 systemd[1]: Started Ceph mon.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:25:49.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: monmap epoch 1 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: last_changed 2024-03-18T01:25:36.021301+0000 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: min_mon_release 19 (squid) 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: election_strategy: 1 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: fsmap 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: mgrmap e1: no daemons active 2024-03-18T01:25:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:49 smithi007 ceph-mon[31631]: from='client.? 172.21.15.7:0/1581575102' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T01:25:49.858 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T01:25:49.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout id: 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:49.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T01:25:49.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:25:49.859 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout services: 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.768087s) 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout data: 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T01:25:49.861 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T01:25:49.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:25:49.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.146+0000 7fc86d400700 1 Processor -- start 2024-03-18T01:25:49.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.146+0000 7fc86d400700 1 -- start start 2024-03-18T01:25:49.864 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.146+0000 7fc86d400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc868111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.146+0000 7fc86d400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc868082460 con 0x7fc86810ec60 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.147+0000 7fc866800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc868111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.147+0000 7fc866800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc868111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36306/0 (socket says 172.21.15.7:36306) 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.147+0000 7fc866800700 1 -- 172.21.15.7:0/332022576 learned_addr learned my addr 172.21.15.7:0/332022576 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.148+0000 7fc866800700 1 -- 172.21.15.7:0/332022576 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc868111590 con 0x7fc86810ec60 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.148+0000 7fc866800700 1 --2- 172.21.15.7:0/332022576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc868111050 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc8580098a0 tx=0x7fc858009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d07c42d3dca8f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.148+0000 7fc865e00700 1 -- 172.21.15.7:0/332022576 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc858004160 con 0x7fc86810ec60 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.148+0000 7fc865e00700 1 -- 172.21.15.7:0/332022576 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc858004300 con 0x7fc86810ec60 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.149+0000 7fc865e00700 1 -- 172.21.15.7:0/332022576 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8580036a0 con 0x7fc86810ec60 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.149+0000 7fc86d400700 1 -- 172.21.15.7:0/332022576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 msgr2=0x7fc868111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.149+0000 7fc86d400700 1 --2- 172.21.15.7:0/332022576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc868111050 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc8580098a0 tx=0x7fc858009bb0 comp rx=0 tx=0).stop 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.149+0000 7fc86d400700 1 -- 172.21.15.7:0/332022576 shutdown_connections 2024-03-18T01:25:49.865 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.149+0000 7fc86d400700 1 --2- 172.21.15.7:0/332022576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc868111050 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:49.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.149+0000 7fc86d400700 1 -- 172.21.15.7:0/332022576 >> 172.21.15.7:0/332022576 conn(0x7fc86807d220 msgr2=0x7fc86807f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:49.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.150+0000 7fc86d400700 1 -- 172.21.15.7:0/332022576 shutdown_connections 2024-03-18T01:25:49.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.150+0000 7fc86d400700 1 -- 172.21.15.7:0/332022576 wait complete. 2024-03-18T01:25:49.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.151+0000 7fc86d400700 1 Processor -- start 2024-03-18T01:25:49.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.151+0000 7fc86d400700 1 -- start start 2024-03-18T01:25:49.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.151+0000 7fc86d400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc8681a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:49.866 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.151+0000 7fc86d400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc868111a60 con 0x7fc86810ec60 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.152+0000 7fc866800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc8681a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.152+0000 7fc866800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc8681a18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36308/0 (socket says 172.21.15.7:36308) 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.152+0000 7fc866800700 1 -- 172.21.15.7:0/1581575102 learned_addr learned my addr 172.21.15.7:0/1581575102 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.152+0000 7fc866800700 1 -- 172.21.15.7:0/1581575102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8681a1e00 con 0x7fc86810ec60 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.152+0000 7fc866800700 1 --2- 172.21.15.7:0/1581575102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc8681a18c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc858000c00 tx=0x7fc858004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.153+0000 7fc85fe00700 1 -- 172.21.15.7:0/1581575102 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc858003db0 con 0x7fc86810ec60 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.153+0000 7fc85fe00700 1 -- 172.21.15.7:0/1581575102 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc858011630 con 0x7fc86810ec60 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.153+0000 7fc85fe00700 1 -- 172.21.15.7:0/1581575102 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc858011950 con 0x7fc86810ec60 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.153+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8681a2090 con 0x7fc86810ec60 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.153+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8681a4d30 con 0x7fc86810ec60 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.153+0000 7fc85fe00700 1 -- 172.21.15.7:0/1581575102 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc85801a460 con 0x7fc86810ec60 2024-03-18T01:25:49.869 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.153+0000 7fc85fe00700 1 -- 172.21.15.7:0/1581575102 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc85801b780 con 0x7fc86810ec60 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.154+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc868063900 con 0x7fc86810ec60 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.158+0000 7fc85fe00700 1 -- 172.21.15.7:0/1581575102 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc85801ba20 con 0x7fc86810ec60 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.238+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fc86802de70 con 0x7fc86810ec60 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.239+0000 7fc85fe00700 1 -- 172.21.15.7:0/1581575102 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fc85801bc00 con 0x7fc86810ec60 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.240+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 msgr2=0x7fc8681a18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.240+0000 7fc86d400700 1 --2- 172.21.15.7:0/1581575102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc8681a18c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc858000c00 tx=0x7fc858004750 comp rx=0 tx=0).stop 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.240+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 shutdown_connections 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.240+0000 7fc86d400700 1 --2- 172.21.15.7:0/1581575102 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc86810ec60 0x7fc8681a18c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.240+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 >> 172.21.15.7:0/1581575102 conn(0x7fc86807d220 msgr2=0x7fc86807f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.240+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 shutdown_connections 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:49.241+0000 7fc86d400700 1 -- 172.21.15.7:0/1581575102 wait complete. 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:mon is available 2024-03-18T01:25:49.870 INFO:teuthology.orchestra.run.smithi007.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T01:25:50.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:50 smithi007 ceph-mon[31631]: from='client.? 172.21.15.7:0/500050975' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T01:25:50.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:50 smithi007 ceph-mon[31631]: from='client.? 172.21.15.7:0/500050975' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T01:25:51.381 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:25:51.381 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T01:25:51.381 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:51.381 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-03-18T01:25:51.381 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T01:25:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T01:25:51.384 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T01:25:51.384 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T01:25:51.386 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.460+0000 7fe94ae00700 1 Processor -- start 2024-03-18T01:25:51.386 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.460+0000 7fe94ae00700 1 -- start start 2024-03-18T01:25:51.386 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.460+0000 7fe94ae00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe944111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.460+0000 7fe94ae00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe944082470 con 0x7fe94410ec60 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.461+0000 7fe943e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe944111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.461+0000 7fe943e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe944111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36318/0 (socket says 172.21.15.7:36318) 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.461+0000 7fe943e00700 1 -- 172.21.15.7:0/633538164 learned_addr learned my addr 172.21.15.7:0/633538164 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.461+0000 7fe943e00700 1 -- 172.21.15.7:0/633538164 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe944111590 con 0x7fe94410ec60 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.462+0000 7fe943e00700 1 --2- 172.21.15.7:0/633538164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe944111050 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe9340098a0 tx=0x7fe934009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb9581643aad273 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.462+0000 7fe942a00700 1 -- 172.21.15.7:0/633538164 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe934004160 con 0x7fe94410ec60 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.462+0000 7fe942a00700 1 -- 172.21.15.7:0/633538164 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe934004300 con 0x7fe94410ec60 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.462+0000 7fe942a00700 1 -- 172.21.15.7:0/633538164 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9340036a0 con 0x7fe94410ec60 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.463+0000 7fe94ae00700 1 -- 172.21.15.7:0/633538164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 msgr2=0x7fe944111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.463+0000 7fe94ae00700 1 --2- 172.21.15.7:0/633538164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe944111050 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fe9340098a0 tx=0x7fe934009bb0 comp rx=0 tx=0).stop 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.463+0000 7fe94ae00700 1 -- 172.21.15.7:0/633538164 shutdown_connections 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.463+0000 7fe94ae00700 1 --2- 172.21.15.7:0/633538164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe944111050 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:51.387 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.463+0000 7fe94ae00700 1 -- 172.21.15.7:0/633538164 >> 172.21.15.7:0/633538164 conn(0x7fe94407d210 msgr2=0x7fe94407f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:51.388 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.463+0000 7fe94ae00700 1 -- 172.21.15.7:0/633538164 shutdown_connections 2024-03-18T01:25:51.388 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.463+0000 7fe94ae00700 1 -- 172.21.15.7:0/633538164 wait complete. 2024-03-18T01:25:51.388 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.464+0000 7fe94ae00700 1 Processor -- start 2024-03-18T01:25:51.388 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.464+0000 7fe94ae00700 1 -- start start 2024-03-18T01:25:51.388 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.465+0000 7fe94ae00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe9441a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:51.388 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.465+0000 7fe94ae00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe944111a60 con 0x7fe94410ec60 2024-03-18T01:25:51.388 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.465+0000 7fe943e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe9441a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:51.390 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.465+0000 7fe943e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe9441a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36320/0 (socket says 172.21.15.7:36320) 2024-03-18T01:25:51.390 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.465+0000 7fe943e00700 1 -- 172.21.15.7:0/500050975 learned_addr learned my addr 172.21.15.7:0/500050975 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.465+0000 7fe943e00700 1 -- 172.21.15.7:0/500050975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9441a1e10 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.466+0000 7fe943e00700 1 --2- 172.21.15.7:0/500050975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe9441a18d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe934000c00 tx=0x7fe934004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.466+0000 7fe940c00700 1 -- 172.21.15.7:0/500050975 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe934003db0 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.466+0000 7fe940c00700 1 -- 172.21.15.7:0/500050975 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe934011630 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.466+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9441a20a0 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.466+0000 7fe940c00700 1 -- 172.21.15.7:0/500050975 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe934011950 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.466+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9441a4d90 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.467+0000 7fe940c00700 1 -- 172.21.15.7:0/500050975 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe93401a460 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.467+0000 7fe940c00700 1 -- 172.21.15.7:0/500050975 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe93401b780 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.468+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe944063900 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.474+0000 7fe940c00700 1 -- 172.21.15.7:0/500050975 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe93401b980 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.564+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fe94402ddc0 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.568+0000 7fe940c00700 1 -- 172.21.15.7:0/500050975 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+452 (secure 0 0 0) 0x7fe93401bb60 con 0x7fe94410ec60 2024-03-18T01:25:51.391 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.569+0000 7fe940c00700 1 -- 172.21.15.7:0/500050975 <== mon.0 v2:172.21.15.7:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe93401ada0 con 0x7fe94410ec60 2024-03-18T01:25:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.570+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 msgr2=0x7fe9441a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.570+0000 7fe94ae00700 1 --2- 172.21.15.7:0/500050975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe9441a18d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fe934000c00 tx=0x7fe934004750 comp rx=0 tx=0).stop 2024-03-18T01:25:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.570+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 shutdown_connections 2024-03-18T01:25:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.570+0000 7fe94ae00700 1 --2- 172.21.15.7:0/500050975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe94410ec60 0x7fe9441a18d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.570+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 >> 172.21.15.7:0/500050975 conn(0x7fe94407d210 msgr2=0x7fe94407f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.571+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 shutdown_connections 2024-03-18T01:25:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:50.571+0000 7fe94ae00700 1 -- 172.21.15.7:0/500050975 wait complete. 2024-03-18T01:25:51.392 INFO:teuthology.orchestra.run.smithi007.stdout:Generating new minimal ceph.conf... 2024-03-18T01:25:52.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:52 smithi007 ceph-mon[31631]: from='client.? 172.21.15.7:0/1485747477' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:25:52.708 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.945+0000 7fdb83a00700 1 Processor -- start 2024-03-18T01:25:52.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.945+0000 7fdb83a00700 1 -- start start 2024-03-18T01:25:52.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.946+0000 7fdb83a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c10e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:52.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.946+0000 7fdb83a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c10ec40 con 0x7fdb7c10e250 2024-03-18T01:25:52.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.947+0000 7fdb81000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c10e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:52.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.947+0000 7fdb81000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c10e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36334/0 (socket says 172.21.15.7:36334) 2024-03-18T01:25:52.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.947+0000 7fdb81000700 1 -- 172.21.15.7:0/3124192442 learned_addr learned my addr 172.21.15.7:0/3124192442 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:52.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.947+0000 7fdb81000700 1 -- 172.21.15.7:0/3124192442 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c10f400 con 0x7fdb7c10e250 2024-03-18T01:25:52.709 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.948+0000 7fdb81000700 1 --2- 172.21.15.7:0/3124192442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c10e670 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdb700098a0 tx=0x7fdb70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b9cd2c25b473ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:52.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.948+0000 7fdb7b400700 1 -- 172.21.15.7:0/3124192442 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb70004160 con 0x7fdb7c10e250 2024-03-18T01:25:52.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.948+0000 7fdb7b400700 1 -- 172.21.15.7:0/3124192442 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdb70004300 con 0x7fdb7c10e250 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.948+0000 7fdb7b400700 1 -- 172.21.15.7:0/3124192442 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb70011b50 con 0x7fdb7c10e250 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.949+0000 7fdb83a00700 1 -- 172.21.15.7:0/3124192442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 msgr2=0x7fdb7c10e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.949+0000 7fdb83a00700 1 --2- 172.21.15.7:0/3124192442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c10e670 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdb700098a0 tx=0x7fdb70009bb0 comp rx=0 tx=0).stop 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.949+0000 7fdb83a00700 1 -- 172.21.15.7:0/3124192442 shutdown_connections 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.949+0000 7fdb83a00700 1 --2- 172.21.15.7:0/3124192442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c10e670 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.949+0000 7fdb83a00700 1 -- 172.21.15.7:0/3124192442 >> 172.21.15.7:0/3124192442 conn(0x7fdb7c07d220 msgr2=0x7fdb7c07f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.949+0000 7fdb83a00700 1 -- 172.21.15.7:0/3124192442 shutdown_connections 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.949+0000 7fdb83a00700 1 -- 172.21.15.7:0/3124192442 wait complete. 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.950+0000 7fdb83a00700 1 Processor -- start 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.951+0000 7fdb83a00700 1 -- start start 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.951+0000 7fdb83a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c1a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:52.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.951+0000 7fdb83a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1127b0 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.951+0000 7fdb81000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c1a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.951+0000 7fdb81000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c1a21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36348/0 (socket says 172.21.15.7:36348) 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.951+0000 7fdb81000700 1 -- 172.21.15.7:0/1485747477 learned_addr learned my addr 172.21.15.7:0/1485747477 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.952+0000 7fdb81000700 1 -- 172.21.15.7:0/1485747477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c1a26f0 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.952+0000 7fdb81000700 1 --2- 172.21.15.7:0/1485747477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c1a21b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdb70011850 tx=0x7fdb70011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.952+0000 7fdb79600700 1 -- 172.21.15.7:0/1485747477 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb70003af0 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.952+0000 7fdb79600700 1 -- 172.21.15.7:0/1485747477 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdb70020070 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.952+0000 7fdb79600700 1 -- 172.21.15.7:0/1485747477 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb700195a0 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.952+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7c1a2980 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.953+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb7c1a5670 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.953+0000 7fdb79600700 1 -- 172.21.15.7:0/1485747477 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdb70019be0 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.953+0000 7fdb79600700 1 -- 172.21.15.7:0/1485747477 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdb70011e00 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.954+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb7c063900 con 0x7fdb7c10e250 2024-03-18T01:25:52.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:51.957+0000 7fdb79600700 1 -- 172.21.15.7:0/1485747477 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdb70003c90 con 0x7fdb7c10e250 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.034+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fdb7c1a5840 con 0x7fdb7c10e250 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.035+0000 7fdb79600700 1 -- 172.21.15.7:0/1485747477 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+173 (secure 0 0 0) 0x7fdb70025460 con 0x7fdb7c10e250 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.037+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 msgr2=0x7fdb7c1a21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.037+0000 7fdb83a00700 1 --2- 172.21.15.7:0/1485747477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c1a21b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdb70011850 tx=0x7fdb70011c10 comp rx=0 tx=0).stop 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.037+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 shutdown_connections 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.037+0000 7fdb83a00700 1 --2- 172.21.15.7:0/1485747477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdb7c10e250 0x7fdb7c1a21b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.037+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 >> 172.21.15.7:0/1485747477 conn(0x7fdb7c07d220 msgr2=0x7fdb7c07ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.037+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 shutdown_connections 2024-03-18T01:25:52.716 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:52.037+0000 7fdb83a00700 1 -- 172.21.15.7:0/1485747477 wait complete. 2024-03-18T01:25:52.717 INFO:teuthology.orchestra.run.smithi007.stdout:Restarting the monitor... 2024-03-18T01:25:53.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:52 smithi007 systemd[1]: Stopping Ceph mon.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:25:53.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:53 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a[31627]: 2024-03-18T01:25:53.184+0000 7f0305000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:25:53.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:53 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a[31627]: 2024-03-18T01:25:53.184+0000 7f0305000700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T01:25:53.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:53 smithi007 podman[31875]: 2024-03-18 01:25:53.21610323 +0000 UTC m=+0.149368421 container died 9fa81d1f10c659fb3b23dd705cd9bfb070bdb474612c8d4f0e4cff848d807253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-18T01:25:53.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:53 smithi007 podman[31875]: 2024-03-18 01:25:53.625546058 +0000 UTC m=+0.558811245 container cleanup 9fa81d1f10c659fb3b23dd705cd9bfb070bdb474612c8d4f0e4cff848d807253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T01:25:53.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:53 smithi007 bash[31875]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a 2024-03-18T01:25:54.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:53 smithi007 podman[31889]: 2024-03-18 01:25:53.801171048 +0000 UTC m=+0.569979027 container remove 9fa81d1f10c659fb3b23dd705cd9bfb070bdb474612c8d4f0e4cff848d807253 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T01:25:54.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 systemd[1]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T01:25:54.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 systemd[1]: Stopped Ceph mon.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:25:54.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 systemd[1]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a.service: Consumed 1.940s CPU time. 2024-03-18T01:25:54.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 systemd[1]: Starting Ceph mon.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:25:54.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 podman[31978]: 2024-03-18 01:25:54.606491689 +0000 UTC m=+0.034726287 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:25:54.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 podman[31978]: 2024-03-18 01:25:54.701929834 +0000 UTC m=+0.130164488 container create f5ed85e015f25a9911abb223ea01342bc4eac987d8cdb5e31bbedb7b4e2d6399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T01:25:54.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 podman[31978]: 2024-03-18 01:25:54.922157834 +0000 UTC m=+0.350392472 container init f5ed85e015f25a9911abb223ea01342bc4eac987d8cdb5e31bbedb7b4e2d6399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-18T01:25:54.982 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 podman[31978]: 2024-03-18 01:25:54.933380307 +0000 UTC m=+0.361614942 container start f5ed85e015f25a9911abb223ea01342bc4eac987d8cdb5e31bbedb7b4e2d6399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-18T01:25:55.064 INFO:teuthology.orchestra.run.smithi007.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T01:25:55.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T01:25:55.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T01:25:55.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: pidfile_write: ignore empty --pid-file 2024-03-18T01:25:55.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: load: jerasure load: lrc 2024-03-18T01:25:55.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: RocksDB version: 7.9.2 2024-03-18T01:25:55.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Git sha 0 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: DB SUMMARY 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: DB Session ID: 18SDCGFW23LJLH2IK85J 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: CURRENT file: CURRENT 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103620 ; 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.error_if_exists: 0 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.create_if_missing: 0 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.paranoid_checks: 1 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T01:25:55.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.env: 0x55759fdb9c60 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.info_log: 0x5575a0a7bb00 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.statistics: (nil) 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.use_fsync: 0 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_log_file_size: 0 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.allow_fallocate: 1 2024-03-18T01:25:55.236 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T01:25:55.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T01:25:55.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.use_direct_reads: 0 2024-03-18T01:25:55.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T01:25:55.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T01:25:55.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.db_log_dir: 2024-03-18T01:25:55.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.wal_dir: 2024-03-18T01:25:55.237 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.write_buffer_manager: 0x5575a0a4e1e0 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T01:25:55.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.unordered_write: 0 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.row_cache: None 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.wal_filter: None 2024-03-18T01:25:55.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.two_write_queues: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.wal_compression: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.atomic_flush: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.log_readahead_size: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T01:25:55.240 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_background_jobs: 2 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_background_compactions: -1 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_subcompactions: 1 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T01:25:55.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_open_files: -1 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_background_flushes: -1 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Compression algorithms supported: 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: kZSTD supported: 0 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: kXpressCompression supported: 0 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: kZlibCompression supported: 1 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: kSnappyCompression supported: 1 2024-03-18T01:25:55.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: kLZ4Compression supported: 1 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: kBZip2Compression supported: 0 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.merge_operator: 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_filter: None 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T01:25:55.243 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5575a0a7bc20) 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 4 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x5575a0a81090 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2024-03-18T01:25:55.244 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: bloomfilter 2024-03-18T01:25:55.245 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 5 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: max_auto_readahead_size: 262144 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: prepopulate_block_cache: 0 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: initial_auto_readahead_size: 8192 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression: NoCompression 2024-03-18T01:25:55.246 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.num_levels: 7 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T01:25:55.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T01:25:55.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T01:25:55.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T01:25:55.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T01:25:55.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T01:25:55.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T01:25:55.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T01:25:55.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.table_properties_collectors: 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.inplace_update_support: 0 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.bloom_locality: 0 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.max_successive_merges: 0 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T01:25:55.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.ttl: 2592000 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.enable_blob_files: false 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.min_blob_size: 0 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T01:25:55.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: caf6e308-df7e-4b28-98bc-81c450550285 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710725154992500, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710725154994798, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96825, "index_size": 313, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13820, "raw_average_key_size": 49, "raw_value_size": 89275, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710725154, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "caf6e308-df7e-4b28-98bc-81c450550285", "db_session_id": "18SDCGFW23LJLH2IK85J", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710725154995048, "job": 1, "event": "recovery_finished"} 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5575a0b74000 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: DB pointer 0x5575a0b5e000 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T01:25:55.254 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 98.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 56.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 98.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 56.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-18T01:25:55.255 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 56.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 56.7 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T01:25:55.256 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 12.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 12.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache BinnedLRUCache@0x5575a0a81090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 3.2e-05 secs_since: 0 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:54 smithi007 ceph-mon[31996]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a@-1(???) e1 preinit fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a@-1(???).mds e1 new map 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a@-1(???).mds e1 print_map 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T01:25:55.257 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 bash[31978]: f5ed85e015f25a9911abb223ea01342bc4eac987d8cdb5e31bbedb7b4e2d6399 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 systemd[1]: Started Ceph mon.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: monmap epoch 1 2024-03-18T01:25:55.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:25:55.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: last_changed 2024-03-18T01:25:36.021301+0000 2024-03-18T01:25:55.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:25:55.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: min_mon_release 19 (squid) 2024-03-18T01:25:55.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: election_strategy: 1 2024-03-18T01:25:55.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:25:55.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: fsmap 2024-03-18T01:25:55.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T01:25:55.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:55 smithi007 ceph-mon[31996]: mgrmap e1: no daemons active 2024-03-18T01:25:56.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.727+0000 7efd72200700 1 Processor -- start 2024-03-18T01:25:56.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.727+0000 7efd72200700 1 -- start start 2024-03-18T01:25:56.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.727+0000 7efd72200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c10e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:56.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.727+0000 7efd72200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c10ec40 con 0x7efd6c10e250 2024-03-18T01:25:56.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.728+0000 7efd6b400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c10e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:56.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.728+0000 7efd6b400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c10e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36356/0 (socket says 172.21.15.7:36356) 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.728+0000 7efd6b400700 1 -- 172.21.15.7:0/385583899 learned_addr learned my addr 172.21.15.7:0/385583899 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.729+0000 7efd6b400700 1 -- 172.21.15.7:0/385583899 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6c10f400 con 0x7efd6c10e250 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.729+0000 7efd6b400700 1 --2- 172.21.15.7:0/385583899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c10e670 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0098a0 tx=0x7efd5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8d0a41d70882e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.729+0000 7efd6a000700 1 -- 172.21.15.7:0/385583899 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd5c004160 con 0x7efd6c10e250 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.729+0000 7efd6a000700 1 -- 172.21.15.7:0/385583899 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efd5c004300 con 0x7efd6c10e250 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.729+0000 7efd6a000700 1 -- 172.21.15.7:0/385583899 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd5c011b50 con 0x7efd6c10e250 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.730+0000 7efd72200700 1 -- 172.21.15.7:0/385583899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 msgr2=0x7efd6c10e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.730+0000 7efd72200700 1 --2- 172.21.15.7:0/385583899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c10e670 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0098a0 tx=0x7efd5c009bb0 comp rx=0 tx=0).stop 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.730+0000 7efd72200700 1 -- 172.21.15.7:0/385583899 shutdown_connections 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.730+0000 7efd72200700 1 --2- 172.21.15.7:0/385583899 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c10e670 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.730+0000 7efd72200700 1 -- 172.21.15.7:0/385583899 >> 172.21.15.7:0/385583899 conn(0x7efd6c07d220 msgr2=0x7efd6c07f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.730+0000 7efd72200700 1 -- 172.21.15.7:0/385583899 shutdown_connections 2024-03-18T01:25:56.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.730+0000 7efd72200700 1 -- 172.21.15.7:0/385583899 wait complete. 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.731+0000 7efd72200700 1 Processor -- start 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.731+0000 7efd72200700 1 -- start start 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.732+0000 7efd72200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c1a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.732+0000 7efd72200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c1127b0 con 0x7efd6c10e250 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.732+0000 7efd6b400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c1a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.732+0000 7efd6b400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c1a21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36370/0 (socket says 172.21.15.7:36370) 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.732+0000 7efd6b400700 1 -- 172.21.15.7:0/3181511693 learned_addr learned my addr 172.21.15.7:0/3181511693 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.732+0000 7efd6b400700 1 -- 172.21.15.7:0/3181511693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6c1a26f0 con 0x7efd6c10e250 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.733+0000 7efd6b400700 1 --2- 172.21.15.7:0/3181511693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c1a21b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efd5c011850 tx=0x7efd5c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.733+0000 7efd63e00700 1 -- 172.21.15.7:0/3181511693 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd5c003af0 con 0x7efd6c10e250 2024-03-18T01:25:56.730 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.733+0000 7efd63e00700 1 -- 172.21.15.7:0/3181511693 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efd5c020070 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.733+0000 7efd63e00700 1 -- 172.21.15.7:0/3181511693 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd5c0195a0 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.733+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd6c1a2980 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.733+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6c1a5670 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.734+0000 7efd63e00700 1 -- 172.21.15.7:0/3181511693 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efd5c0198a0 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.734+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd6c063900 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.734+0000 7efd63e00700 1 -- 172.21.15.7:0/3181511693 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efd5c011e00 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.738+0000 7efd63e00700 1 -- 172.21.15.7:0/3181511693 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7efd5c003c90 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.820+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7efd6c1a5840 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.823+0000 7efd63e00700 1 -- 172.21.15.7:0/3181511693 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7efd5c019d20 con 0x7efd6c10e250 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.824+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 msgr2=0x7efd6c1a21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:56.731 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.824+0000 7efd72200700 1 --2- 172.21.15.7:0/3181511693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c1a21b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efd5c011850 tx=0x7efd5c011c10 comp rx=0 tx=0).stop 2024-03-18T01:25:56.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.824+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 shutdown_connections 2024-03-18T01:25:56.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.824+0000 7efd72200700 1 --2- 172.21.15.7:0/3181511693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd6c10e250 0x7efd6c1a21b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:56.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.824+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 >> 172.21.15.7:0/3181511693 conn(0x7efd6c07d220 msgr2=0x7efd6c07dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:56.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.825+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 shutdown_connections 2024-03-18T01:25:56.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:55.825+0000 7efd72200700 1 -- 172.21.15.7:0/3181511693 wait complete. 2024-03-18T01:25:56.732 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T01:25:56.811 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:25:56.811 INFO:teuthology.orchestra.run.smithi007.stdout:Creating mgr... 2024-03-18T01:25:56.812 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-18T01:25:56.813 INFO:teuthology.orchestra.run.smithi007.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-18T01:25:57.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:56 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3181511693' entity='client.admin' 2024-03-18T01:25:57.216 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.x 2024-03-18T01:25:57.216 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-18T01:25:57.515 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715.target.wants/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@.service. 2024-03-18T01:25:57.802 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:57 smithi007 systemd[1]: Starting Ceph mgr.x for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:25:58.232 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:57 smithi007 podman[32222]: 2024-03-18 01:25:57.83265653 +0000 UTC m=+0.037796741 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T01:25:58.232 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:57 smithi007 podman[32222]: 2024-03-18 01:25:57.929992881 +0000 UTC m=+0.135133129 container create 7d71c73ad82fbb3a7186dae6d09c5ba04e885c4554ef15603d73bbf7fd395afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:25:58.232 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:58 smithi007 podman[32222]: 2024-03-18 01:25:58.150566974 +0000 UTC m=+0.355707229 container init 7d71c73ad82fbb3a7186dae6d09c5ba04e885c4554ef15603d73bbf7fd395afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-18T01:25:58.232 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:58 smithi007 podman[32222]: 2024-03-18 01:25:58.160651659 +0000 UTC m=+0.365791914 container start 7d71c73ad82fbb3a7186dae6d09c5ba04e885c4554ef15603d73bbf7fd395afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T01:25:58.232 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:58 smithi007 bash[32222]: 7d71c73ad82fbb3a7186dae6d09c5ba04e885c4554ef15603d73bbf7fd395afe 2024-03-18T01:25:58.261 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T01:25:58.262 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-03-18T01:25:58.279 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T01:25:58.279 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-18T01:25:58.279 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-03-18T01:25:58.280 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T01:25:58.302 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T01:25:58.302 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-03-18T01:25:58.321 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T01:25:58.321 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-18T01:25:58.321 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-03-18T01:25:58.321 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-18T01:25:58.321 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr to start... 2024-03-18T01:25:58.322 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr... 2024-03-18T01:25:58.525 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:58 smithi007 systemd[1]: Started Ceph mgr.x for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:25:59.031 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:58 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:25:58.953+0000 7eff222b7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T01:25:59.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:25:59 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1271826269' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T01:25:59.302 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:59 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:25:59.087+0000 7eff222b7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T01:25:59.694 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:25:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:25:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:25:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T01:25:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T01:25:59.695 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T01:25:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T01:25:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:25:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T01:25:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T01:25:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-03-18T01:25:59.697 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:25:59.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T01:25:59.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T01:25:59.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:25:59.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-03-18T01:25:59.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T01:25:59.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:25:59.698 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T01:25:59.699 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T01:25:59.699 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:25:59.699 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T01:25:59.699 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:25:59.700 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T01:25:59.700 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T01:25:59.700 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T01:25:59.700 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T01:25:59.700 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T01:25:59.700 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T01:25:59.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:25:59.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T01:25:59.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T01:25:59.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T01:25:59.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T01:25:59.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T01:25:59.704 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T01:25:59.705 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T01:25:48.474114+0000", 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:25:59.706 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.923+0000 7f26da200700 1 Processor -- start 2024-03-18T01:25:59.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.923+0000 7f26da200700 1 -- start start 2024-03-18T01:25:59.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.923+0000 7f26da200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d4106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:59.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.923+0000 7f26da200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f26d4075950 con 0x7f26d4106510 2024-03-18T01:25:59.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.924+0000 7f26d3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d4106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:59.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.924+0000 7f26d3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d4106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44588/0 (socket says 172.21.15.7:44588) 2024-03-18T01:25:59.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.924+0000 7f26d3400700 1 -- 172.21.15.7:0/115544286 learned_addr learned my addr 172.21.15.7:0/115544286 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:59.707 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.924+0000 7f26d3400700 1 -- 172.21.15.7:0/115544286 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d4106e70 con 0x7f26d4106510 2024-03-18T01:25:59.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.925+0000 7f26d3400700 1 --2- 172.21.15.7:0/115544286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d4106930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f26c40098a0 tx=0x7f26c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6a1c44d72544dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:59.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.925+0000 7f26d2a00700 1 -- 172.21.15.7:0/115544286 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c4004160 con 0x7f26d4106510 2024-03-18T01:25:59.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.925+0000 7f26d2a00700 1 -- 172.21.15.7:0/115544286 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f26c4004300 con 0x7f26d4106510 2024-03-18T01:25:59.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.925+0000 7f26d2a00700 1 -- 172.21.15.7:0/115544286 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c4011b50 con 0x7f26d4106510 2024-03-18T01:25:59.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.925+0000 7f26da200700 1 -- 172.21.15.7:0/115544286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 msgr2=0x7f26d4106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.925+0000 7f26da200700 1 --2- 172.21.15.7:0/115544286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d4106930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f26c40098a0 tx=0x7f26c4009bb0 comp rx=0 tx=0).stop 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.926+0000 7f26da200700 1 -- 172.21.15.7:0/115544286 shutdown_connections 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.926+0000 7f26da200700 1 --2- 172.21.15.7:0/115544286 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d4106930 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.926+0000 7f26da200700 1 -- 172.21.15.7:0/115544286 >> 172.21.15.7:0/115544286 conn(0x7f26d41020d0 msgr2=0x7f26d41044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.926+0000 7f26da200700 1 -- 172.21.15.7:0/115544286 shutdown_connections 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.926+0000 7f26da200700 1 -- 172.21.15.7:0/115544286 wait complete. 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.927+0000 7f26da200700 1 Processor -- start 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.927+0000 7f26da200700 1 -- start start 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.927+0000 7f26da200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.927+0000 7f26da200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f26d4107e90 con 0x7f26d4106510 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.928+0000 7f26d3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.928+0000 7f26d3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d419dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44600/0 (socket says 172.21.15.7:44600) 2024-03-18T01:25:59.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.928+0000 7f26d3400700 1 -- 172.21.15.7:0/1271826269 learned_addr learned my addr 172.21.15.7:0/1271826269 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.928+0000 7f26d3400700 1 -- 172.21.15.7:0/1271826269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d419e250 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.928+0000 7f26d3400700 1 --2- 172.21.15.7:0/1271826269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d419dd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f26c4011670 tx=0x7f26c4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.929+0000 7f26d0c00700 1 -- 172.21.15.7:0/1271826269 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c40039c0 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.929+0000 7f26d0c00700 1 -- 172.21.15.7:0/1271826269 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f26c4011e50 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.929+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d419e4e0 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.929+0000 7f26d0c00700 1 -- 172.21.15.7:0/1271826269 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f26c401a5c0 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.929+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d41a11c0 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.929+0000 7f26d0c00700 1 -- 172.21.15.7:0/1271826269 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f26c401ac00 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.929+0000 7f26d0c00700 1 -- 172.21.15.7:0/1271826269 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f26c4019700 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.930+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26d4063900 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:58.932+0000 7f26d0c00700 1 -- 172.21.15.7:0/1271826269 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f26c4003b60 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.012+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f26d402ddc0 con 0x7f26d4106510 2024-03-18T01:25:59.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.013+0000 7f26d0c00700 1 -- 172.21.15.7:0/1271826269 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f26c40199a0 con 0x7f26d4106510 2024-03-18T01:25:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.014+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 msgr2=0x7f26d419dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:25:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.014+0000 7f26da200700 1 --2- 172.21.15.7:0/1271826269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d419dd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f26c4011670 tx=0x7f26c4011b60 comp rx=0 tx=0).stop 2024-03-18T01:25:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.014+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 shutdown_connections 2024-03-18T01:25:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.014+0000 7f26da200700 1 --2- 172.21.15.7:0/1271826269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f26d4106510 0x7f26d419dd10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:25:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.014+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 >> 172.21.15.7:0/1271826269 conn(0x7f26d41020d0 msgr2=0x7f26d4194970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:25:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.015+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 shutdown_connections 2024-03-18T01:25:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:25:59.015+0000 7f26da200700 1 -- 172.21.15.7:0/1271826269 wait complete. 2024-03-18T01:25:59.714 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (1/15)... 2024-03-18T01:26:00.044 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:25:59 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:25:59.918+0000 7eff222b7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T01:26:00.301 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:00 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:00.043+0000 7eff222b7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T01:26:00.302 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:00 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:00.223+0000 7eff222b7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T01:26:00.801 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:00 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:00.351+0000 7eff222b7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T01:26:00.802 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:00 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:00.463+0000 7eff222b7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T01:26:01.069 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:00 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:00.808+0000 7eff222b7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T01:26:01.069 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:00 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:00.953+0000 7eff222b7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T01:26:01.416 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:01 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:01.068+0000 7eff222b7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T01:26:01.416 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:01 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:01.194+0000 7eff222b7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T01:26:01.416 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:01 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:01.415+0000 7eff222b7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T01:26:02.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:02 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1592437914' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T01:26:03.051 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:02 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:02.791+0000 7eff222b7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T01:26:03.052 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:02 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:02.986+0000 7eff222b7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T01:26:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:26:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:26:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:26:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T01:26:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T01:26:03.156 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T01:26:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T01:26:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T01:26:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T01:26:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-03-18T01:26:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:03.157 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:03.159 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T01:26:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T01:26:03.160 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T01:26:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T01:26:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T01:26:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T01:26:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T01:26:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T01:26:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T01:26:03.164 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T01:26:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T01:25:48.474114+0000", 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:03.166 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T01:26:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:26:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.349+0000 7f9756000700 1 Processor -- start 2024-03-18T01:26:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.349+0000 7f9756000700 1 -- start start 2024-03-18T01:26:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.350+0000 7f9756000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f9750108080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.350+0000 7f9756000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9750108650 con 0x7f9750107c60 2024-03-18T01:26:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.350+0000 7f974f400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f9750108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:03.167 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.350+0000 7f974f400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f9750108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44614/0 (socket says 172.21.15.7:44614) 2024-03-18T01:26:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.350+0000 7f974f400700 1 -- 172.21.15.7:0/3231619583 learned_addr learned my addr 172.21.15.7:0/3231619583 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.351+0000 7f974f400700 1 -- 172.21.15.7:0/3231619583 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9750108e60 con 0x7f9750107c60 2024-03-18T01:26:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.351+0000 7f974f400700 1 --2- 172.21.15.7:0/3231619583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f9750108080 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f97400098a0 tx=0x7f9740009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd9f7fe96eb981f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.351+0000 7f974e000700 1 -- 172.21.15.7:0/3231619583 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9740004160 con 0x7f9750107c60 2024-03-18T01:26:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.351+0000 7f974e000700 1 -- 172.21.15.7:0/3231619583 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9740004300 con 0x7f9750107c60 2024-03-18T01:26:03.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.351+0000 7f974e000700 1 -- 172.21.15.7:0/3231619583 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9740011b50 con 0x7f9750107c60 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.352+0000 7f9756000700 1 -- 172.21.15.7:0/3231619583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 msgr2=0x7f9750108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.352+0000 7f9756000700 1 --2- 172.21.15.7:0/3231619583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f9750108080 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f97400098a0 tx=0x7f9740009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.352+0000 7f9756000700 1 -- 172.21.15.7:0/3231619583 shutdown_connections 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.352+0000 7f9756000700 1 --2- 172.21.15.7:0/3231619583 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f9750108080 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.352+0000 7f9756000700 1 -- 172.21.15.7:0/3231619583 >> 172.21.15.7:0/3231619583 conn(0x7f9750103200 msgr2=0x7f9750105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.352+0000 7f9756000700 1 -- 172.21.15.7:0/3231619583 shutdown_connections 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.352+0000 7f9756000700 1 -- 172.21.15.7:0/3231619583 wait complete. 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.354+0000 7f9756000700 1 Processor -- start 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.354+0000 7f9756000700 1 -- start start 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.354+0000 7f9756000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f975019bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.354+0000 7f9756000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f975010c210 con 0x7f9750107c60 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.355+0000 7f974f400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f975019bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:03.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.355+0000 7f974f400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f975019bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44628/0 (socket says 172.21.15.7:44628) 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.355+0000 7f974f400700 1 -- 172.21.15.7:0/1592437914 learned_addr learned my addr 172.21.15.7:0/1592437914 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.355+0000 7f974f400700 1 -- 172.21.15.7:0/1592437914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975019c0c0 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.355+0000 7f974f400700 1 --2- 172.21.15.7:0/1592437914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f975019bb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9740000c00 tx=0x7f9740011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.356+0000 7f9747e00700 1 -- 172.21.15.7:0/1592437914 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9740003b20 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.356+0000 7f9747e00700 1 -- 172.21.15.7:0/1592437914 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9740003cc0 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.356+0000 7f9747e00700 1 -- 172.21.15.7:0/1592437914 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9740024910 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.356+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975019c350 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.356+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f975019f040 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.356+0000 7f9747e00700 1 -- 172.21.15.7:0/1592437914 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9740022020 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.356+0000 7f9747e00700 1 -- 172.21.15.7:0/1592437914 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9740011e00 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.357+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9750063900 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.360+0000 7f9747e00700 1 -- 172.21.15.7:0/1592437914 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f974001acd0 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.438+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f975019f210 con 0x7f9750107c60 2024-03-18T01:26:03.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.439+0000 7f9747e00700 1 -- 172.21.15.7:0/1592437914 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9740032030 con 0x7f9750107c60 2024-03-18T01:26:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.440+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 msgr2=0x7f975019bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.440+0000 7f9756000700 1 --2- 172.21.15.7:0/1592437914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f975019bb80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9740000c00 tx=0x7f9740011c10 comp rx=0 tx=0).stop 2024-03-18T01:26:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.440+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 shutdown_connections 2024-03-18T01:26:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.440+0000 7f9756000700 1 --2- 172.21.15.7:0/1592437914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9750107c60 0x7f975019bb80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.440+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 >> 172.21.15.7:0/1592437914 conn(0x7f9750103200 msgr2=0x7f9750103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.441+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 shutdown_connections 2024-03-18T01:26:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:02.441+0000 7f9756000700 1 -- 172.21.15.7:0/1592437914 wait complete. 2024-03-18T01:26:03.174 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (2/15)... 2024-03-18T01:26:03.801 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:03 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:03.430+0000 7eff222b7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T01:26:03.801 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:03 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:03.551+0000 7eff222b7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T01:26:04.066 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:03 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:03.815+0000 7eff222b7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:26:04.067 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:04 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:04.066+0000 7eff222b7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T01:26:04.768 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:04 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:04.446+0000 7eff222b7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:26:04.768 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:04 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:04.767+0000 7eff222b7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T01:26:05.447 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:05 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:05.079+0000 7eff222b7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T01:26:06.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:05 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2914772618' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T01:26:06.533 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:06 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:06.176+0000 7eff222b7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T01:26:06.544 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:26:06.544 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:26:06.544 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-18T01:26:06.545 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T01:26:06.546 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:06.553 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:06.554 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T01:25:48.474114+0000", 2024-03-18T01:26:06.555 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.712+0000 7f93a1c00700 1 Processor -- start 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.712+0000 7f93a1c00700 1 -- start start 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.713+0000 7f93a1c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c07b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.713+0000 7f93a1c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f939c07b730 con 0x7f939c07cd00 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.713+0000 7f939aa00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c07b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.713+0000 7f939aa00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c07b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44640/0 (socket says 172.21.15.7:44640) 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.713+0000 7f939aa00700 1 -- 172.21.15.7:0/1821399124 learned_addr learned my addr 172.21.15.7:0/1821399124 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.714+0000 7f939aa00700 1 -- 172.21.15.7:0/1821399124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c07b8b0 con 0x7f939c07cd00 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.714+0000 7f939aa00700 1 --2- 172.21.15.7:0/1821399124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c07b160 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f938c0098a0 tx=0x7f938c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e22118f54d5a4fe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.714+0000 7f9399600700 1 -- 172.21.15.7:0/1821399124 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f938c004160 con 0x7f939c07cd00 2024-03-18T01:26:06.556 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.714+0000 7f9399600700 1 -- 172.21.15.7:0/1821399124 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f938c004300 con 0x7f939c07cd00 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.714+0000 7f9399600700 1 -- 172.21.15.7:0/1821399124 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f938c011b50 con 0x7f939c07cd00 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.715+0000 7f93a1c00700 1 -- 172.21.15.7:0/1821399124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 msgr2=0x7f939c07b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.715+0000 7f93a1c00700 1 --2- 172.21.15.7:0/1821399124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c07b160 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f938c0098a0 tx=0x7f938c009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.715+0000 7f93a1c00700 1 -- 172.21.15.7:0/1821399124 shutdown_connections 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.715+0000 7f93a1c00700 1 --2- 172.21.15.7:0/1821399124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c07b160 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.715+0000 7f93a1c00700 1 -- 172.21.15.7:0/1821399124 >> 172.21.15.7:0/1821399124 conn(0x7f939c1031e0 msgr2=0x7f939c105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.716+0000 7f93a1c00700 1 -- 172.21.15.7:0/1821399124 shutdown_connections 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.716+0000 7f93a1c00700 1 -- 172.21.15.7:0/1821399124 wait complete. 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.717+0000 7f93a1c00700 1 Processor -- start 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.717+0000 7f93a1c00700 1 -- start start 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.718+0000 7f93a1c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.718+0000 7f93a1c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f939c10e1f0 con 0x7f939c07cd00 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.718+0000 7f939aa00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:06.557 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.718+0000 7f939aa00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44648/0 (socket says 172.21.15.7:44648) 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.718+0000 7f939aa00700 1 -- 172.21.15.7:0/2914772618 learned_addr learned my addr 172.21.15.7:0/2914772618 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.718+0000 7f939aa00700 1 -- 172.21.15.7:0/2914772618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939c19e2e0 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.719+0000 7f939aa00700 1 --2- 172.21.15.7:0/2914772618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c19dda0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f938c011b60 tx=0x7f938c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.719+0000 7f9393400700 1 -- 172.21.15.7:0/2914772618 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f938c003970 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.719+0000 7f9393400700 1 -- 172.21.15.7:0/2914772618 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f938c003b10 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.719+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939c19e570 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.719+0000 7f9393400700 1 -- 172.21.15.7:0/2914772618 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f938c024610 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.719+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939c1a1260 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.720+0000 7f9393400700 1 -- 172.21.15.7:0/2914772618 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f938c024910 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.720+0000 7f9393400700 1 -- 172.21.15.7:0/2914772618 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f938c011e00 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.720+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f939c063900 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.723+0000 7f9393400700 1 -- 172.21.15.7:0/2914772618 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f938c01a7c0 con 0x7f939c07cd00 2024-03-18T01:26:06.558 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.802+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f939c1a1430 con 0x7f939c07cd00 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.803+0000 7f9393400700 1 -- 172.21.15.7:0/2914772618 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f938c02de10 con 0x7f939c07cd00 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.804+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 msgr2=0x7f939c19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.804+0000 7f93a1c00700 1 --2- 172.21.15.7:0/2914772618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c19dda0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f938c011b60 tx=0x7f938c011c40 comp rx=0 tx=0).stop 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.805+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 shutdown_connections 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.805+0000 7f93a1c00700 1 --2- 172.21.15.7:0/2914772618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f939c07cd00 0x7f939c19dda0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.805+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 >> 172.21.15.7:0/2914772618 conn(0x7f939c1031e0 msgr2=0x7f939c103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.805+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 shutdown_connections 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:05.805+0000 7f93a1c00700 1 -- 172.21.15.7:0/2914772618 wait complete. 2024-03-18T01:26:06.559 INFO:teuthology.orchestra.run.smithi007.stdout:mgr not available, waiting (3/15)... 2024-03-18T01:26:07.199 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:06 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:06.900+0000 7eff222b7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T01:26:07.199 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:07.031+0000 7eff222b7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T01:26:07.551 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:07.325+0000 7eff222b7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: Activating manager daemon x 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: mgrmap e2: x(active, starting, since 0.0103246s) 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: Manager daemon x is now available 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T01:26:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' 2024-03-18T01:26:07.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' 2024-03-18T01:26:07.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:07 smithi007 ceph-mon[31996]: from='mgr.14100 172.21.15.7:0/1069101053' entity='mgr.x' 2024-03-18T01:26:09.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:09 smithi007 ceph-mon[31996]: mgrmap e3: x(active, since 1.02114s) 2024-03-18T01:26:10.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:26:10.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:26:10.250 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:26:10.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T01:26:10.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T01:26:10.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T01:26:10.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T01:26:10.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:10.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T01:26:10.252 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 0 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T01:26:10.253 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T01:26:10.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:10.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T01:26:10.254 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:10.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T01:26:10.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T01:26:10.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T01:26:10.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T01:26:10.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T01:26:10.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T01:26:10.255 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:10.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T01:26:10.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T01:26:10.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T01:26:10.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T01:26:10.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T01:26:10.258 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T01:26:10.259 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ], 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T01:25:48.474114+0000", 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout }, 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:26:10.260 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.209+0000 7f2d16800700 1 Processor -- start 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.210+0000 7f2d16800700 1 -- start start 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.210+0000 7f2d16800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d10110c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.210+0000 7f2d16800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2d10111240 con 0x7f2d10082240 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.211+0000 7f2d15400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d10110c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.211+0000 7f2d15400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d10110c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50130/0 (socket says 172.21.15.7:50130) 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.211+0000 7f2d15400700 1 -- 172.21.15.7:0/3586394071 learned_addr learned my addr 172.21.15.7:0/3586394071 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.211+0000 7f2d15400700 1 -- 172.21.15.7:0/3586394071 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d101113c0 con 0x7f2d10082240 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.211+0000 7f2d15400700 1 --2- 172.21.15.7:0/3586394071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d10110c70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2d040098a0 tx=0x7f2d04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e04883e1f0db46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.212+0000 7f2d0fe00700 1 -- 172.21.15.7:0/3586394071 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d04004160 con 0x7f2d10082240 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.212+0000 7f2d0fe00700 1 -- 172.21.15.7:0/3586394071 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2d04004300 con 0x7f2d10082240 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.212+0000 7f2d0fe00700 1 -- 172.21.15.7:0/3586394071 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d04011b50 con 0x7f2d10082240 2024-03-18T01:26:10.266 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.213+0000 7f2d16800700 1 -- 172.21.15.7:0/3586394071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 msgr2=0x7f2d10110c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.213+0000 7f2d16800700 1 --2- 172.21.15.7:0/3586394071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d10110c70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2d040098a0 tx=0x7f2d04009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.213+0000 7f2d16800700 1 -- 172.21.15.7:0/3586394071 shutdown_connections 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.213+0000 7f2d16800700 1 --2- 172.21.15.7:0/3586394071 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d10110c70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.213+0000 7f2d16800700 1 -- 172.21.15.7:0/3586394071 >> 172.21.15.7:0/3586394071 conn(0x7f2d1007d090 msgr2=0x7f2d1007f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.213+0000 7f2d16800700 1 -- 172.21.15.7:0/3586394071 shutdown_connections 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.213+0000 7f2d16800700 1 -- 172.21.15.7:0/3586394071 wait complete. 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.214+0000 7f2d16800700 1 Processor -- start 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.214+0000 7f2d16800700 1 -- start start 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.215+0000 7f2d16800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d101a2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.215+0000 7f2d16800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2d101123e0 con 0x7f2d10082240 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.215+0000 7f2d15400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d101a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.215+0000 7f2d15400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d101a2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50140/0 (socket says 172.21.15.7:50140) 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.215+0000 7f2d15400700 1 -- 172.21.15.7:0/3202905925 learned_addr learned my addr 172.21.15.7:0/3202905925 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.215+0000 7f2d15400700 1 -- 172.21.15.7:0/3202905925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d101a2550 con 0x7f2d10082240 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.216+0000 7f2d15400700 1 --2- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d101a2010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2d04011850 tx=0x7f2d04011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:10.267 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.216+0000 7f2d0e000700 1 -- 172.21.15.7:0/3202905925 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d04003af0 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.216+0000 7f2d0e000700 1 -- 172.21.15.7:0/3202905925 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2d04020070 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.216+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d101a27e0 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.216+0000 7f2d0e000700 1 -- 172.21.15.7:0/3202905925 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2d040195a0 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.216+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d101a54d0 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.217+0000 7f2d0e000700 1 -- 172.21.15.7:0/3202905925 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 3) ==== 45989+0+0 (secure 0 0 0) 0x7f2d040198a0 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.217+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d10063900 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.217+0000 7f2d0e000700 1 --2- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f2cf8038e00 0x7f2cf803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.218+0000 7f2d0e000700 1 -- 172.21.15.7:0/3202905925 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2d04011e00 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.218+0000 7f2d14a00700 1 --2- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f2cf8038e00 0x7f2cf803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.219+0000 7f2d14a00700 1 --2- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f2cf8038e00 0x7f2cf803b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc006fd0 tx=0x7f2cfc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.224+0000 7f2d0e000700 1 -- 172.21.15.7:0/3202905925 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d04050030 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.349+0000 7f2d0e000700 1 -- 172.21.15.7:0/3202905925 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f2d0402bd00 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.540+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2d101a56a0 con 0x7f2d10082240 2024-03-18T01:26:10.268 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.541+0000 7f2d0e000700 1 -- 172.21.15.7:0/3202905925 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2d101a56a0 con 0x7f2d10082240 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.543+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f2cf8038e00 msgr2=0x7f2cf803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.543+0000 7f2d16800700 1 --2- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f2cf8038e00 0x7f2cf803b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc006fd0 tx=0x7f2cfc006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.543+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 msgr2=0x7f2d101a2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.543+0000 7f2d16800700 1 --2- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d101a2010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2d04011850 tx=0x7f2d04011c10 comp rx=0 tx=0).stop 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.544+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 shutdown_connections 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.544+0000 7f2d16800700 1 --2- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f2cf8038e00 0x7f2cf803b2c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.544+0000 7f2d16800700 1 --2- 172.21.15.7:0/3202905925 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d10082240 0x7f2d101a2010 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.544+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 >> 172.21.15.7:0/3202905925 conn(0x7f2d1007d090 msgr2=0x7f2d1007dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.544+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 shutdown_connections 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:09.544+0000 7f2d16800700 1 -- 172.21.15.7:0/3202905925 wait complete. 2024-03-18T01:26:10.269 INFO:teuthology.orchestra.run.smithi007.stdout:mgr is available 2024-03-18T01:26:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:10 smithi007 ceph-mon[31996]: mgrmap e4: x(active, since 2s) 2024-03-18T01:26:10.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3202905925' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T01:26:11.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:11 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/276335285' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T01:26:11.776 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:26:11.776 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T01:26:11.776 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout fsid = 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:26:11.776 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T01:26:11.776 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T01:26:11.777 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T01:26:11.777 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T01:26:11.777 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:26:11.777 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T01:26:11.777 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T01:26:11.777 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 2024-03-18T01:26:11.777 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T01:26:11.777 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-18T01:26:11.779 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T01:26:11.779 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T01:26:11.779 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T01:26:11.779 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T01:26:11.779 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T01:26:11.779 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.848+0000 7ff3e9a00700 1 Processor -- start 2024-03-18T01:26:11.779 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.849+0000 7ff3e9a00700 1 -- start start 2024-03-18T01:26:11.780 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.849+0000 7ff3e9a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e410a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:11.780 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.849+0000 7ff3e9a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410a760 con 0x7ff3e4109e00 2024-03-18T01:26:11.780 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.849+0000 7ff3e2a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e410a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:11.780 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.850+0000 7ff3e2a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e410a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50152/0 (socket says 172.21.15.7:50152) 2024-03-18T01:26:11.780 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.850+0000 7ff3e2a00700 1 -- 172.21.15.7:0/837063111 learned_addr learned my addr 172.21.15.7:0/837063111 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.850+0000 7ff3e2a00700 1 -- 172.21.15.7:0/837063111 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e410af60 con 0x7ff3e4109e00 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.850+0000 7ff3e2a00700 1 --2- 172.21.15.7:0/837063111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e410a220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40098a0 tx=0x7ff3d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b527a4b7a6d502f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.851+0000 7ff3e1600700 1 -- 172.21.15.7:0/837063111 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3d4004160 con 0x7ff3e4109e00 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.851+0000 7ff3e1600700 1 -- 172.21.15.7:0/837063111 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff3d4004300 con 0x7ff3e4109e00 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.851+0000 7ff3e1600700 1 -- 172.21.15.7:0/837063111 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3d4011b50 con 0x7ff3e4109e00 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.851+0000 7ff3e9a00700 1 -- 172.21.15.7:0/837063111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 msgr2=0x7ff3e410a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.851+0000 7ff3e9a00700 1 --2- 172.21.15.7:0/837063111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e410a220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff3d40098a0 tx=0x7ff3d4009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.851+0000 7ff3e9a00700 1 -- 172.21.15.7:0/837063111 shutdown_connections 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.851+0000 7ff3e9a00700 1 --2- 172.21.15.7:0/837063111 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e410a220 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.851+0000 7ff3e9a00700 1 -- 172.21.15.7:0/837063111 >> 172.21.15.7:0/837063111 conn(0x7ff3e407d210 msgr2=0x7ff3e407d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:11.782 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.852+0000 7ff3e9a00700 1 -- 172.21.15.7:0/837063111 shutdown_connections 2024-03-18T01:26:11.783 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.852+0000 7ff3e9a00700 1 -- 172.21.15.7:0/837063111 wait complete. 2024-03-18T01:26:11.783 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.853+0000 7ff3e9a00700 1 Processor -- start 2024-03-18T01:26:11.783 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.853+0000 7ff3e9a00700 1 -- start start 2024-03-18T01:26:11.783 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.853+0000 7ff3e9a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e419dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:11.783 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.853+0000 7ff3e9a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e410e220 con 0x7ff3e4109e00 2024-03-18T01:26:11.786 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.853+0000 7ff3e2a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e419dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:11.786 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.853+0000 7ff3e2a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e419dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50168/0 (socket says 172.21.15.7:50168) 2024-03-18T01:26:11.786 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.853+0000 7ff3e2a00700 1 -- 172.21.15.7:0/276335285 learned_addr learned my addr 172.21.15.7:0/276335285 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:11.786 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.854+0000 7ff3e2a00700 1 -- 172.21.15.7:0/276335285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e419e2e0 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.854+0000 7ff3e2a00700 1 --2- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e419dda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4011b60 tx=0x7ff3d4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.854+0000 7ff3db400700 1 -- 172.21.15.7:0/276335285 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3d4003970 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.854+0000 7ff3db400700 1 -- 172.21.15.7:0/276335285 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff3d4003b10 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.854+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e419e570 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.854+0000 7ff3db400700 1 -- 172.21.15.7:0/276335285 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3d4024610 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.855+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e41a1260 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.856+0000 7ff3db400700 1 -- 172.21.15.7:0/276335285 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7ff3d401a9f0 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.856+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e410e3a0 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.856+0000 7ff3db400700 1 --2- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff3c8038f20 0x7ff3c803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.856+0000 7ff3db400700 1 -- 172.21.15.7:0/276335285 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff3d4011e00 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.856+0000 7ff3e2000700 1 --2- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff3c8038f20 0x7ff3c803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.857+0000 7ff3e2000700 1 --2- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff3c8038f20 0x7ff3c803b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0006fd0 tx=0x7ff3d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:10.862+0000 7ff3db400700 1 -- 172.21.15.7:0/276335285 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff3d4019e10 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.087+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff3e4063900 con 0x7ff3e4109e00 2024-03-18T01:26:11.787 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.089+0000 7ff3db400700 1 -- 172.21.15.7:0/276335285 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7ff3d404c9f0 con 0x7ff3e4109e00 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.092+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff3c8038f20 msgr2=0x7ff3c803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.092+0000 7ff3e9a00700 1 --2- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff3c8038f20 0x7ff3c803b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0006fd0 tx=0x7ff3d0006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.092+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 msgr2=0x7ff3e419dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.092+0000 7ff3e9a00700 1 --2- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e419dda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff3d4011b60 tx=0x7ff3d4011c40 comp rx=0 tx=0).stop 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.093+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 shutdown_connections 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.093+0000 7ff3e9a00700 1 --2- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff3c8038f20 0x7ff3c803b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.093+0000 7ff3e9a00700 1 --2- 172.21.15.7:0/276335285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff3e4109e00 0x7ff3e419dda0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.093+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 >> 172.21.15.7:0/276335285 conn(0x7ff3e407d210 msgr2=0x7ff3e41074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.093+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 shutdown_connections 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:11.093+0000 7ff3e9a00700 1 -- 172.21.15.7:0/276335285 wait complete. 2024-03-18T01:26:11.788 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling cephadm module... 2024-03-18T01:26:13.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:12 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2714565957' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T01:26:13.052 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:12 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: ignoring --setuser ceph since I am not root 2024-03-18T01:26:13.052 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:12 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: ignoring --setgroup ceph since I am not root 2024-03-18T01:26:13.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.393+0000 7fe7e1400700 1 Processor -- start 2024-03-18T01:26:13.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.393+0000 7fe7e1400700 1 -- start start 2024-03-18T01:26:13.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.393+0000 7fe7e1400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc110c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:13.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.394+0000 7fe7e1400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc111240 con 0x7fe7dc082230 2024-03-18T01:26:13.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.394+0000 7fe7dbe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc110c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:13.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.394+0000 7fe7dbe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc110c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50170/0 (socket says 172.21.15.7:50170) 2024-03-18T01:26:13.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.394+0000 7fe7dbe00700 1 -- 172.21.15.7:0/2139339338 learned_addr learned my addr 172.21.15.7:0/2139339338 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.394+0000 7fe7dbe00700 1 -- 172.21.15.7:0/2139339338 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc1113c0 con 0x7fe7dc082230 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.395+0000 7fe7dbe00700 1 --2- 172.21.15.7:0/2139339338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc110c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0098a0 tx=0x7fe7cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fbba45d71976a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.395+0000 7fe7db400700 1 -- 172.21.15.7:0/2139339338 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7cc004160 con 0x7fe7dc082230 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.395+0000 7fe7db400700 1 -- 172.21.15.7:0/2139339338 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe7cc004300 con 0x7fe7dc082230 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.395+0000 7fe7db400700 1 -- 172.21.15.7:0/2139339338 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7cc011b50 con 0x7fe7dc082230 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.396+0000 7fe7e1400700 1 -- 172.21.15.7:0/2139339338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 msgr2=0x7fe7dc110c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.396+0000 7fe7e1400700 1 --2- 172.21.15.7:0/2139339338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc110c70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0098a0 tx=0x7fe7cc009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.396+0000 7fe7e1400700 1 -- 172.21.15.7:0/2139339338 shutdown_connections 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.396+0000 7fe7e1400700 1 --2- 172.21.15.7:0/2139339338 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc110c70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.396+0000 7fe7e1400700 1 -- 172.21.15.7:0/2139339338 >> 172.21.15.7:0/2139339338 conn(0x7fe7dc07d060 msgr2=0x7fe7dc07f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.396+0000 7fe7e1400700 1 -- 172.21.15.7:0/2139339338 shutdown_connections 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.396+0000 7fe7e1400700 1 -- 172.21.15.7:0/2139339338 wait complete. 2024-03-18T01:26:13.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.397+0000 7fe7e1400700 1 Processor -- start 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.397+0000 7fe7e1400700 1 -- start start 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.398+0000 7fe7e1400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc1a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.398+0000 7fe7e1400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe7dc1123e0 con 0x7fe7dc082230 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.398+0000 7fe7dbe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc1a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.398+0000 7fe7dbe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc1a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50180/0 (socket says 172.21.15.7:50180) 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.398+0000 7fe7dbe00700 1 -- 172.21.15.7:0/2714565957 learned_addr learned my addr 172.21.15.7:0/2714565957 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.398+0000 7fe7dbe00700 1 -- 172.21.15.7:0/2714565957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7dc1a24e0 con 0x7fe7dc082230 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.399+0000 7fe7dbe00700 1 --2- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc1a1fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc011b60 tx=0x7fe7cc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.399+0000 7fe7d9600700 1 -- 172.21.15.7:0/2714565957 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7cc003970 con 0x7fe7dc082230 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.399+0000 7fe7d9600700 1 -- 172.21.15.7:0/2714565957 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe7cc003b10 con 0x7fe7dc082230 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.399+0000 7fe7d9600700 1 -- 172.21.15.7:0/2714565957 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7cc024610 con 0x7fe7dc082230 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.399+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7dc1a2770 con 0x7fe7dc082230 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.399+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7dc1a5460 con 0x7fe7dc082230 2024-03-18T01:26:13.413 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.400+0000 7fe7d9600700 1 -- 172.21.15.7:0/2714565957 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fe7cc01a9f0 con 0x7fe7dc082230 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.400+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7dc063900 con 0x7fe7dc082230 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.401+0000 7fe7d9600700 1 --2- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7fe7c0038f20 0x7fe7c003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.401+0000 7fe7d9600700 1 -- 172.21.15.7:0/2714565957 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe7cc011e00 con 0x7fe7dc082230 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.401+0000 7fe7d3400700 1 --2- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7fe7c0038f20 0x7fe7c003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.402+0000 7fe7d3400700 1 --2- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7fe7c0038f20 0x7fe7c003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006fd0 tx=0x7fe7c4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.406+0000 7fe7d9600700 1 -- 172.21.15.7:0/2714565957 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe7cc019e10 con 0x7fe7dc082230 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.696+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fe7dc1a5630 con 0x7fe7dc082230 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.750+0000 7fe7d9600700 1 -- 172.21.15.7:0/2714565957 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fe7dc1a5630 con 0x7fe7dc082230 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.754+0000 7fe7d9600700 1 -- 172.21.15.7:0/2714565957 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fe7cc024c40 con 0x7fe7dc082230 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7fe7c0038f20 msgr2=0x7fe7c003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 --2- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7fe7c0038f20 0x7fe7c003b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006fd0 tx=0x7fe7c4006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:13.414 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 msgr2=0x7fe7dc1a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:13.415 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 --2- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc1a1fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc011b60 tx=0x7fe7cc011c40 comp rx=0 tx=0).stop 2024-03-18T01:26:13.415 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 shutdown_connections 2024-03-18T01:26:13.415 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 --2- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7fe7c0038f20 0x7fe7c003b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:13.415 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 --2- 172.21.15.7:0/2714565957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7dc082230 0x7fe7dc1a1fa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:13.415 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 >> 172.21.15.7:0/2714565957 conn(0x7fe7dc07d060 msgr2=0x7fe7dc07dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:13.415 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.755+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 shutdown_connections 2024-03-18T01:26:13.415 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:12.756+0000 7fe7e1400700 1 -- 172.21.15.7:0/2714565957 wait complete. 2024-03-18T01:26:13.694 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:13 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:13.568+0000 7f5354947200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T01:26:14.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:13 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2714565957' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T01:26:14.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:13 smithi007 ceph-mon[31996]: mgrmap e5: x(active, since 5s) 2024-03-18T01:26:14.000 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:13 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:13.694+0000 7f5354947200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T01:26:14.551 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:14 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:14.519+0000 7f5354947200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T01:26:14.949 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:14 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2482676252' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T01:26:14.950 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:14 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:14.651+0000 7f5354947200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T01:26:14.950 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:14 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:14.834+0000 7f5354947200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T01:26:14.967 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:13.999+0000 7f15a2a00700 1 Processor -- start 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:13.999+0000 7f15a2a00700 1 -- start start 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.000+0000 7f15a2a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c10aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.000+0000 7f15a2a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f159c0759f0 con 0x7f159c108680 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.000+0000 7f159be00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c10aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:14.969 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.000+0000 7f159be00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c10aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50214/0 (socket says 172.21.15.7:50214) 2024-03-18T01:26:14.974 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.000+0000 7f159be00700 1 -- 172.21.15.7:0/4217076223 learned_addr learned my addr 172.21.15.7:0/4217076223 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:14.974 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.001+0000 7f159be00700 1 -- 172.21.15.7:0/4217076223 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c10afb0 con 0x7f159c108680 2024-03-18T01:26:14.974 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.001+0000 7f159be00700 1 --2- 172.21.15.7:0/4217076223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c10aa70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f158c0098a0 tx=0x7f158c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4633bcd0c4949460 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.001+0000 7f159aa00700 1 -- 172.21.15.7:0/4217076223 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f158c004160 con 0x7f159c108680 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.001+0000 7f159aa00700 1 -- 172.21.15.7:0/4217076223 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f158c004300 con 0x7f159c108680 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.001+0000 7f159aa00700 1 -- 172.21.15.7:0/4217076223 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f158c011b50 con 0x7f159c108680 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.002+0000 7f15a2a00700 1 -- 172.21.15.7:0/4217076223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 msgr2=0x7f159c10aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.002+0000 7f15a2a00700 1 --2- 172.21.15.7:0/4217076223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c10aa70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f158c0098a0 tx=0x7f158c009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.002+0000 7f15a2a00700 1 -- 172.21.15.7:0/4217076223 shutdown_connections 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.002+0000 7f15a2a00700 1 --2- 172.21.15.7:0/4217076223 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c10aa70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.002+0000 7f15a2a00700 1 -- 172.21.15.7:0/4217076223 >> 172.21.15.7:0/4217076223 conn(0x7f159c1020d0 msgr2=0x7f159c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.002+0000 7f15a2a00700 1 -- 172.21.15.7:0/4217076223 shutdown_connections 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.002+0000 7f15a2a00700 1 -- 172.21.15.7:0/4217076223 wait complete. 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.003+0000 7f15a2a00700 1 Processor -- start 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.003+0000 7f15a2a00700 1 -- start start 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.004+0000 7f15a2a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c107f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.004+0000 7f15a2a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f159c10bf70 con 0x7f159c108680 2024-03-18T01:26:14.975 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.004+0000 7f159be00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c107f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.004+0000 7f159be00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c107f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50220/0 (socket says 172.21.15.7:50220) 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.004+0000 7f159be00700 1 -- 172.21.15.7:0/2482676252 learned_addr learned my addr 172.21.15.7:0/2482676252 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.004+0000 7f159be00700 1 -- 172.21.15.7:0/2482676252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c108440 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.005+0000 7f159be00700 1 --2- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c107f00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f158c011b60 tx=0x7f158c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.005+0000 7f1598c00700 1 -- 172.21.15.7:0/2482676252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f158c003b20 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.005+0000 7f1598c00700 1 -- 172.21.15.7:0/2482676252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f158c003cc0 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.005+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159c106630 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.005+0000 7f1598c00700 1 -- 172.21.15.7:0/2482676252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f158c024840 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.005+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f159c106b70 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.006+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f159c063900 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.006+0000 7f1598c00700 1 -- 172.21.15.7:0/2482676252 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f158c022020 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.007+0000 7f1598c00700 1 --2- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f1580038f70 0x7f158003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.007+0000 7f1598c00700 1 -- 172.21.15.7:0/2482676252 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f158c011e00 con 0x7f159c108680 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.007+0000 7f159b400700 1 -- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f1580038f70 msgr2=0x7f158003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1791220024 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.007+0000 7f159b400700 1 --2- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f1580038f70 0x7f158003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:26:14.976 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.012+0000 7f1598c00700 1 -- 172.21.15.7:0/2482676252 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f158c019c70 con 0x7f159c108680 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.208+0000 7f159b400700 1 -- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f1580038f70 msgr2=0x7f158003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1791220024 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.208+0000 7f159b400700 1 --2- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f1580038f70 0x7f158003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.293+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f159c1074a0 con 0x7f159c108680 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.293+0000 7f1598c00700 1 -- 172.21.15.7:0/2482676252 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f158c00ba30 con 0x7f159c108680 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f1580038f70 msgr2=0x7f158003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 --2- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f1580038f70 0x7f158003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 msgr2=0x7f159c107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 --2- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c107f00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f158c011b60 tx=0x7f158c011c40 comp rx=0 tx=0).stop 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 shutdown_connections 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 --2- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7f1580038f70 0x7f158003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 --2- 172.21.15.7:0/2482676252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f159c108680 0x7f159c107f00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 >> 172.21.15.7:0/2482676252 conn(0x7f159c1020d0 msgr2=0x7f159c102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.296+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 shutdown_connections 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:14.297+0000 7f15a2a00700 1 -- 172.21.15.7:0/2482676252 wait complete. 2024-03-18T01:26:14.977 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-03-18T01:26:14.978 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 5... 2024-03-18T01:26:15.286 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:14 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:14.965+0000 7f5354947200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T01:26:15.286 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:15 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:15.081+0000 7f5354947200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T01:26:15.551 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:15 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:15.445+0000 7f5354947200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T01:26:15.843 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:15 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:15.599+0000 7f5354947200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T01:26:15.843 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:15 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:15.714+0000 7f5354947200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T01:26:15.843 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:15 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:15.842+0000 7f5354947200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T01:26:16.169 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:16 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:16.064+0000 7f5354947200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T01:26:17.801 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:17 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:17.379+0000 7f5354947200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T01:26:17.802 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:17 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:17.555+0000 7f5354947200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T01:26:18.301 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:17 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:17.981+0000 7f5354947200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T01:26:18.302 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:18 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:18.104+0000 7f5354947200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T01:26:18.621 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:18 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:18.367+0000 7f5354947200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:26:19.008 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:18 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:18.620+0000 7f5354947200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T01:26:19.301 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:19 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:19.007+0000 7f5354947200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:26:19.644 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:19 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:19.329+0000 7f5354947200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T01:26:20.051 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:19 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:19.643+0000 7f5354947200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T01:26:21.051 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:20 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:20.707+0000 7f5354947200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T01:26:21.773 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:21.422+0000 7f5354947200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T01:26:21.774 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:21.557+0000 7f5354947200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T01:26:22.052 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:21.846+0000 7f5354947200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T01:26:22.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: Active manager daemon x restarted 2024-03-18T01:26:22.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: Activating manager daemon x 2024-03-18T01:26:22.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T01:26:22.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: mgrmap e6: x(active, starting, since 0.00481415s) 2024-03-18T01:26:22.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:26:22.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T01:26:22.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T01:26:22.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T01:26:22.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T01:26:22.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: Manager daemon x is now available 2024-03-18T01:26:22.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:22.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:21 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:23.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: Found migration_current of "None". Setting to last migration. 2024-03-18T01:26:23.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:26:23.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:26:23.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:26:23.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: [18/Mar/2024:01:26:22] ENGINE Bus STARTING 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: [18/Mar/2024:01:26:22] ENGINE Serving on http://172.21.15.7:8765 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: [18/Mar/2024:01:26:22] ENGINE Serving on https://172.21.15.7:7150 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: [18/Mar/2024:01:26:22] ENGINE Bus STARTED 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:26:23.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:22 smithi007 ceph-mon[31996]: mgrmap e7: x(active, since 1.01663s) 2024-03-18T01:26:23.522 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:26:23.522 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T01:26:23.522 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T01:26:23.522 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:26:23.522 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.559+0000 7ff253800700 1 Processor -- start 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.560+0000 7ff253800700 1 -- start start 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.560+0000 7ff253800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.560+0000 7ff253800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c108630 con 0x7ff24c107c40 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.560+0000 7ff250e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.560+0000 7ff250e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50236/0 (socket says 172.21.15.7:50236) 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.560+0000 7ff250e00700 1 -- 172.21.15.7:0/1514979367 learned_addr learned my addr 172.21.15.7:0/1514979367 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.561+0000 7ff250e00700 1 -- 172.21.15.7:0/1514979367 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c108e40 con 0x7ff24c107c40 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.561+0000 7ff250e00700 1 --2- 172.21.15.7:0/1514979367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c108060 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff2400098a0 tx=0x7ff240009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba7cf052d71abd79 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.561+0000 7ff24b400700 1 -- 172.21.15.7:0/1514979367 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff240004160 con 0x7ff24c107c40 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.561+0000 7ff24b400700 1 -- 172.21.15.7:0/1514979367 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff240004300 con 0x7ff24c107c40 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.561+0000 7ff24b400700 1 -- 172.21.15.7:0/1514979367 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff240011b50 con 0x7ff24c107c40 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.562+0000 7ff253800700 1 -- 172.21.15.7:0/1514979367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 msgr2=0x7ff24c108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.562+0000 7ff253800700 1 --2- 172.21.15.7:0/1514979367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c108060 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff2400098a0 tx=0x7ff240009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:23.523 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.562+0000 7ff253800700 1 -- 172.21.15.7:0/1514979367 shutdown_connections 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.562+0000 7ff253800700 1 --2- 172.21.15.7:0/1514979367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c108060 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.562+0000 7ff253800700 1 -- 172.21.15.7:0/1514979367 >> 172.21.15.7:0/1514979367 conn(0x7ff24c103240 msgr2=0x7ff24c105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.562+0000 7ff253800700 1 -- 172.21.15.7:0/1514979367 shutdown_connections 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.562+0000 7ff253800700 1 -- 172.21.15.7:0/1514979367 wait complete. 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.563+0000 7ff253800700 1 Processor -- start 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.563+0000 7ff253800700 1 -- start start 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.564+0000 7ff253800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c07db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.564+0000 7ff253800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c10c1f0 con 0x7ff24c107c40 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.564+0000 7ff250e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c07db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.564+0000 7ff250e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c07db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:50250/0 (socket says 172.21.15.7:50250) 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.564+0000 7ff250e00700 1 -- 172.21.15.7:0/353925208 learned_addr learned my addr 172.21.15.7:0/353925208 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.564+0000 7ff250e00700 1 -- 172.21.15.7:0/353925208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c07e040 con 0x7ff24c107c40 2024-03-18T01:26:23.524 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.565+0000 7ff250e00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c07db00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff240011850 tx=0x7ff240011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.565+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff240003af0 con 0x7ff24c107c40 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.565+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff240020070 con 0x7ff24c107c40 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.565+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2400195a0 con 0x7ff24c107c40 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.565+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24c07c230 con 0x7ff24c107c40 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.565+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24c07c770 con 0x7ff24c107c40 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.566+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7ff24001a430 con 0x7ff24c107c40 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.567+0000 7ff249600700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.567+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 --> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff23403bb40 con 0x7ff234038f70 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.567+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff240011e00 con 0x7ff24c107c40 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.567+0000 7ff24be00700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 msgr2=0x7ff23403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1791220024 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.567+0000 7ff24be00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.768+0000 7ff24be00700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 msgr2=0x7ff23403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1791220024 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:15.768+0000 7ff24be00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:26:23.525 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:16.169+0000 7ff24be00700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 msgr2=0x7ff23403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1791220024 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:16.169+0000 7ff24be00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:16.970+0000 7ff24be00700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 msgr2=0x7ff23403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1791220024 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:16.970+0000 7ff24be00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:18.572+0000 7ff24be00700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 msgr2=0x7ff23403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1791220024 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:18.572+0000 7ff24be00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:21.773+0000 7ff24be00700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 msgr2=0x7ff23403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1791220024 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:21.773+0000 7ff24be00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:21.855+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7ff24002bd00 con 0x7ff24c107c40 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:21.855+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 msgr2=0x7ff23403b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:21.855+0000 7ff249600700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.860+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7ff24004eaa0 con 0x7ff24c107c40 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.861+0000 7ff249600700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff23403c860 0x7ff23403ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.861+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff23403bb40 con 0x7ff23403c860 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.863+0000 7ff24be00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff23403c860 0x7ff23403ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.866+0000 7ff24be00700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff23403c860 0x7ff23403ec50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff238003a10 tx=0x7ff2380092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.868+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mgr.14120 v2:172.21.15.7:6800/1169030810 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ff23403bb40 con 0x7ff23403c860 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.878+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff24c063900 con 0x7ff23403c860 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff249600700 1 -- 172.21.15.7:0/353925208 <== mgr.14120 v2:172.21.15.7:6800/1169030810 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ff24c063900 con 0x7ff23403c860 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff23403c860 msgr2=0x7ff23403ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff253800700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff23403c860 0x7ff23403ec50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff238003a10 tx=0x7ff2380092b0 comp rx=0 tx=0).stop 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 msgr2=0x7ff24c07db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff253800700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c07db00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff240011850 tx=0x7ff240011c10 comp rx=0 tx=0).stop 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 shutdown_connections 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff253800700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff23403c860 0x7ff23403ec50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff253800700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff24c107c40 0x7ff24c07db00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.879+0000 7ff253800700 1 --2- 172.21.15.7:0/353925208 >> [v2:172.21.15.7:6800/1791220024,v1:172.21.15.7:6801/1791220024] conn(0x7ff234038f70 0x7ff23403b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:23.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.880+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 >> 172.21.15.7:0/353925208 conn(0x7ff24c103240 msgr2=0x7ff24c103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:23.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.880+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 shutdown_connections 2024-03-18T01:26:23.528 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:22.880+0000 7ff253800700 1 -- 172.21.15.7:0/353925208 wait complete. 2024-03-18T01:26:23.528 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 5 is available 2024-03-18T01:26:23.528 INFO:teuthology.orchestra.run.smithi007.stdout:Setting orchestrator backend to cephadm... 2024-03-18T01:26:24.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:23 smithi007 ceph-mon[31996]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T01:26:24.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:23 smithi007 ceph-mon[31996]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T01:26:25.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:24 smithi007 ceph-mon[31996]: mgrmap e8: x(active, since 2s) 2024-03-18T01:26:25.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:24 smithi007 ceph-mon[31996]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:26:25.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:24 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:25.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:24 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.127+0000 7f9a6c800700 1 Processor -- start 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.127+0000 7f9a6c800700 1 -- start start 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.128+0000 7f9a6c800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a64111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.128+0000 7f9a6c800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9a64082470 con 0x7f9a6410ec60 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.128+0000 7f9a69e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a64111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.128+0000 7f9a69e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a64111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43454/0 (socket says 172.21.15.7:43454) 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.128+0000 7f9a69e00700 1 -- 172.21.15.7:0/3408035520 learned_addr learned my addr 172.21.15.7:0/3408035520 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.129+0000 7f9a69e00700 1 -- 172.21.15.7:0/3408035520 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a64111590 con 0x7f9a6410ec60 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.129+0000 7f9a69e00700 1 --2- 172.21.15.7:0/3408035520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a64111050 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9a580098a0 tx=0x7f9a58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb0e8d9104e844 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.129+0000 7f9a68a00700 1 -- 172.21.15.7:0/3408035520 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a58004160 con 0x7f9a6410ec60 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.129+0000 7f9a68a00700 1 -- 172.21.15.7:0/3408035520 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a58004300 con 0x7f9a6410ec60 2024-03-18T01:26:25.171 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.129+0000 7f9a68a00700 1 -- 172.21.15.7:0/3408035520 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a58011b50 con 0x7f9a6410ec60 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.130+0000 7f9a6c800700 1 -- 172.21.15.7:0/3408035520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 msgr2=0x7f9a64111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.130+0000 7f9a6c800700 1 --2- 172.21.15.7:0/3408035520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a64111050 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9a580098a0 tx=0x7f9a58009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.130+0000 7f9a6c800700 1 -- 172.21.15.7:0/3408035520 shutdown_connections 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.130+0000 7f9a6c800700 1 --2- 172.21.15.7:0/3408035520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a64111050 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.130+0000 7f9a6c800700 1 -- 172.21.15.7:0/3408035520 >> 172.21.15.7:0/3408035520 conn(0x7f9a6407d210 msgr2=0x7f9a6407f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.130+0000 7f9a6c800700 1 -- 172.21.15.7:0/3408035520 shutdown_connections 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.130+0000 7f9a6c800700 1 -- 172.21.15.7:0/3408035520 wait complete. 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.131+0000 7f9a6c800700 1 Processor -- start 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.132+0000 7f9a6c800700 1 -- start start 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.132+0000 7f9a6c800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a641a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.132+0000 7f9a6c800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9a641125a0 con 0x7f9a6410ec60 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.132+0000 7f9a69e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a641a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.132+0000 7f9a69e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a641a2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43466/0 (socket says 172.21.15.7:43466) 2024-03-18T01:26:25.172 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.132+0000 7f9a69e00700 1 -- 172.21.15.7:0/3930021920 learned_addr learned my addr 172.21.15.7:0/3930021920 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.133+0000 7f9a69e00700 1 -- 172.21.15.7:0/3930021920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a641a2680 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.133+0000 7f9a69e00700 1 --2- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a641a2140 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9a58011b60 tx=0x7f9a58011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.133+0000 7f9a5ea00700 1 -- 172.21.15.7:0/3930021920 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a58003970 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.133+0000 7f9a5ea00700 1 -- 172.21.15.7:0/3930021920 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9a58003b10 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.133+0000 7f9a5ea00700 1 -- 172.21.15.7:0/3930021920 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a58024610 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.133+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a641a2910 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.134+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a641a5600 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.135+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a64063900 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.135+0000 7f9a5ea00700 1 -- 172.21.15.7:0/3930021920 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f9a5801a9f0 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.135+0000 7f9a5ea00700 1 --2- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f9a4c038f70 0x7f9a4c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.135+0000 7f9a5ea00700 1 -- 172.21.15.7:0/3930021920 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9a58011e00 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.136+0000 7f9a69400700 1 --2- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f9a4c038f70 0x7f9a4c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.136+0000 7f9a69400700 1 --2- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f9a4c038f70 0x7f9a4c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9a50006fd0 tx=0x7f9a50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.141+0000 7f9a5ea00700 1 -- 172.21.15.7:0/3930021920 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9a5800b3e0 con 0x7f9a6410ec60 2024-03-18T01:26:25.173 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.384+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f9a6407e410 con 0x7f9a4c038f70 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.401+0000 7f9a5ea00700 1 -- 172.21.15.7:0/3930021920 <== mgr.14120 v2:172.21.15.7:6800/1169030810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9a6407e410 con 0x7f9a4c038f70 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f9a4c038f70 msgr2=0x7f9a4c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 --2- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f9a4c038f70 0x7f9a4c03b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9a50006fd0 tx=0x7f9a50006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 msgr2=0x7f9a641a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 --2- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a641a2140 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9a58011b60 tx=0x7f9a58011c40 comp rx=0 tx=0).stop 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 shutdown_connections 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 --2- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f9a4c038f70 0x7f9a4c03b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 --2- 172.21.15.7:0/3930021920 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9a6410ec60 0x7f9a641a2140 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 >> 172.21.15.7:0/3930021920 conn(0x7f9a6407d210 msgr2=0x7f9a6407dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.404+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 shutdown_connections 2024-03-18T01:26:25.174 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:24.405+0000 7f9a6c800700 1 -- 172.21.15.7:0/3930021920 wait complete. 2024-03-18T01:26:26.680 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T01:26:26.680 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.791+0000 7fc1fbc00700 1 Processor -- start 2024-03-18T01:26:26.680 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.791+0000 7fc1fbc00700 1 -- start start 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.792+0000 7fc1fbc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f407b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.792+0000 7fc1fbc00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f407b750 con 0x7fc1f407cd20 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.792+0000 7fc1f9200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f407b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.792+0000 7fc1f9200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f407b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43468/0 (socket says 172.21.15.7:43468) 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.792+0000 7fc1f9200700 1 -- 172.21.15.7:0/3425953457 learned_addr learned my addr 172.21.15.7:0/3425953457 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.793+0000 7fc1f9200700 1 -- 172.21.15.7:0/3425953457 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f407b8d0 con 0x7fc1f407cd20 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.793+0000 7fc1f9200700 1 --2- 172.21.15.7:0/3425953457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f407b180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80098a0 tx=0x7fc1e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8d484decd825ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.793+0000 7fc1f3400700 1 -- 172.21.15.7:0/3425953457 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1e8004160 con 0x7fc1f407cd20 2024-03-18T01:26:26.682 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.793+0000 7fc1f3400700 1 -- 172.21.15.7:0/3425953457 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc1e8004300 con 0x7fc1f407cd20 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.794+0000 7fc1f3400700 1 -- 172.21.15.7:0/3425953457 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1e8011b50 con 0x7fc1f407cd20 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.794+0000 7fc1fbc00700 1 -- 172.21.15.7:0/3425953457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 msgr2=0x7fc1f407b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.794+0000 7fc1fbc00700 1 --2- 172.21.15.7:0/3425953457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f407b180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80098a0 tx=0x7fc1e8009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.794+0000 7fc1fbc00700 1 -- 172.21.15.7:0/3425953457 shutdown_connections 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.794+0000 7fc1fbc00700 1 --2- 172.21.15.7:0/3425953457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f407b180 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.794+0000 7fc1fbc00700 1 -- 172.21.15.7:0/3425953457 >> 172.21.15.7:0/3425953457 conn(0x7fc1f41031e0 msgr2=0x7fc1f4105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.794+0000 7fc1fbc00700 1 -- 172.21.15.7:0/3425953457 shutdown_connections 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.794+0000 7fc1fbc00700 1 -- 172.21.15.7:0/3425953457 wait complete. 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.795+0000 7fc1fbc00700 1 Processor -- start 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.796+0000 7fc1fbc00700 1 -- start start 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.796+0000 7fc1fbc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f41a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:26.683 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.796+0000 7fc1fbc00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f4109dc0 con 0x7fc1f407cd20 2024-03-18T01:26:26.686 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.796+0000 7fc1f9200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f41a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.796+0000 7fc1f9200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f41a21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43476/0 (socket says 172.21.15.7:43476) 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.796+0000 7fc1f9200700 1 -- 172.21.15.7:0/933848184 learned_addr learned my addr 172.21.15.7:0/933848184 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.797+0000 7fc1f9200700 1 -- 172.21.15.7:0/933848184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f41a2720 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.797+0000 7fc1f9200700 1 --2- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f41a21e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8011850 tx=0x7fc1e8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.797+0000 7fc1f1600700 1 -- 172.21.15.7:0/933848184 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1e8003af0 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.797+0000 7fc1f1600700 1 -- 172.21.15.7:0/933848184 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc1e8020070 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.797+0000 7fc1f1600700 1 -- 172.21.15.7:0/933848184 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1e80195a0 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.797+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f41a29b0 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.797+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f41a56a0 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.798+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1f4063900 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.798+0000 7fc1f1600700 1 -- 172.21.15.7:0/933848184 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fc1e8019850 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.799+0000 7fc1f1600700 1 --2- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fc1dc038f70 0x7fc1dc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.799+0000 7fc1f1600700 1 -- 172.21.15.7:0/933848184 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc1e8011e00 con 0x7fc1f407cd20 2024-03-18T01:26:26.687 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.799+0000 7fc1f3e00700 1 --2- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fc1dc038f70 0x7fc1dc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.800+0000 7fc1f3e00700 1 --2- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fc1dc038f70 0x7fc1dc03b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0006fd0 tx=0x7fc1e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:25.805+0000 7fc1f1600700 1 -- 172.21.15.7:0/933848184 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc1e8019b40 con 0x7fc1f407cd20 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.027+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fc1f4062710 con 0x7fc1dc038f70 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.029+0000 7fc1f1600700 1 -- 172.21.15.7:0/933848184 <== mgr.14120 v2:172.21.15.7:6800/1169030810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fc1f4062710 con 0x7fc1dc038f70 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fc1dc038f70 msgr2=0x7fc1dc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 --2- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fc1dc038f70 0x7fc1dc03b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0006fd0 tx=0x7fc1e0006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 msgr2=0x7fc1f41a21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 --2- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f41a21e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8011850 tx=0x7fc1e8011c10 comp rx=0 tx=0).stop 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 shutdown_connections 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 --2- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fc1dc038f70 0x7fc1dc03b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 --2- 172.21.15.7:0/933848184 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1f407cd20 0x7fc1f41a21e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 >> 172.21.15.7:0/933848184 conn(0x7fc1f41031e0 msgr2=0x7fc1f41055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 shutdown_connections 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:26.032+0000 7fc1fbc00700 1 -- 172.21.15.7:0/933848184 wait complete. 2024-03-18T01:26:26.688 INFO:teuthology.orchestra.run.smithi007.stdout:Generating ssh key... 2024-03-18T01:26:27.261 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:26 smithi007 ceph-mon[31996]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:26:28.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:27 smithi007 ceph-mon[31996]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:26:28.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:27 smithi007 ceph-mon[31996]: Generating ssh key... 2024-03-18T01:26:28.553 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: Generating public/private rsa key pair. 2024-03-18T01:26:28.553 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: Your identification has been saved in /tmp/tmpvbtfp8xm/key. 2024-03-18T01:26:28.553 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: Your public key has been saved in /tmp/tmpvbtfp8xm/key.pub. 2024-03-18T01:26:28.553 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: The key fingerprint is: 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: SHA256:bN5dk0bOKeIefZnHh6PNVKdRywHxlNF4dFOUsMtu+m0 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: The key's randomart image is: 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: +---[RSA 3072]----+ 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | +oB@| 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | *o=| 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | ..+.| 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | . .+o+o| 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | S . .oXoo| 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | o o +.+ X.| 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | . + ooO +| 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | . .o*.E.| 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: | ..o.+. | 2024-03-18T01:26:28.554 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:28 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: +----[SHA256]-----+ 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.421+0000 7fd00f200700 1 Processor -- start 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.421+0000 7fd00f200700 1 -- start start 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.422+0000 7fd00f200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008106510 0x7fd008106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.422+0000 7fd00f200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd008075950 con 0x7fd008106510 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.423+0000 7fd007e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008106510 0x7fd008106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.423+0000 7fd007e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008106510 0x7fd008106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37872/0 (socket says 172.21.15.7:37872) 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.423+0000 7fd007e00700 1 -- 172.21.15.7:0/438868464 learned_addr learned my addr 172.21.15.7:0/438868464 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.424+0000 7fd007e00700 1 -- 172.21.15.7:0/438868464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd008106e70 con 0x7fd008106510 2024-03-18T01:26:28.872 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.424+0000 7fd007e00700 1 --2- 172.21.15.7:0/438868464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008106510 0x7fd008106930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009e80 tx=0x7fcff800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792d894e546ea0bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.425+0000 7fd007400700 1 -- 172.21.15.7:0/438868464 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcff8017070 con 0x7fd008106510 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.425+0000 7fd007400700 1 -- 172.21.15.7:0/438868464 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcff800b8c0 con 0x7fd008106510 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.425+0000 7fd007400700 1 -- 172.21.15.7:0/438868464 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcff80045c0 con 0x7fd008106510 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.425+0000 7fd00f200700 1 -- 172.21.15.7:0/438868464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008106510 msgr2=0x7fd008106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.425+0000 7fd00f200700 1 --2- 172.21.15.7:0/438868464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008106510 0x7fd008106930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009e80 tx=0x7fcff800b330 comp rx=0 tx=0).stop 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.426+0000 7fd00f200700 1 -- 172.21.15.7:0/438868464 shutdown_connections 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.426+0000 7fd00f200700 1 --2- 172.21.15.7:0/438868464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd008106510 0x7fd008106930 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.426+0000 7fd00f200700 1 -- 172.21.15.7:0/438868464 >> 172.21.15.7:0/438868464 conn(0x7fd0081020d0 msgr2=0x7fd0081044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.426+0000 7fd00f200700 1 -- 172.21.15.7:0/438868464 shutdown_connections 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.426+0000 7fd00f200700 1 -- 172.21.15.7:0/438868464 wait complete. 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.427+0000 7fd00f200700 1 Processor -- start 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.427+0000 7fd00f200700 1 -- start start 2024-03-18T01:26:28.873 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.427+0000 7fd00f200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd00819def0 0x7fd00819e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.427+0000 7fd00f200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd008107e90 con 0x7fd00819def0 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.428+0000 7fd007e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd00819def0 0x7fd00819e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.428+0000 7fd007e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd00819def0 0x7fd00819e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37880/0 (socket says 172.21.15.7:37880) 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.428+0000 7fd007e00700 1 -- 172.21.15.7:0/3637538194 learned_addr learned my addr 172.21.15.7:0/3637538194 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.428+0000 7fd007e00700 1 -- 172.21.15.7:0/3637538194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00819e850 con 0x7fd00819def0 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.429+0000 7fd007e00700 1 --2- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd00819def0 0x7fd00819e310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcff8004350 tx=0x7fcff800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.429+0000 7fd005600700 1 -- 172.21.15.7:0/3637538194 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcff8021020 con 0x7fd00819def0 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.429+0000 7fd005600700 1 -- 172.21.15.7:0/3637538194 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcff8003b50 con 0x7fd00819def0 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.429+0000 7fd005600700 1 -- 172.21.15.7:0/3637538194 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcff8017040 con 0x7fd00819def0 2024-03-18T01:26:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.429+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00819eae0 con 0x7fd00819def0 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.429+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0081a1640 con 0x7fd00819def0 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.430+0000 7fd005600700 1 -- 172.21.15.7:0/3637538194 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fcff8024020 con 0x7fd00819def0 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.430+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd008063900 con 0x7fd00819def0 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.431+0000 7fd005600700 1 --2- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fcfec038f70 0x7fcfec03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.431+0000 7fd005600700 1 -- 172.21.15.7:0/3637538194 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcff800f050 con 0x7fd00819def0 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.431+0000 7fcfff400700 1 --2- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fcfec038f70 0x7fcfec03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.432+0000 7fcfff400700 1 --2- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fcfec038f70 0x7fcfec03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcff0006fd0 tx=0x7fcff0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.436+0000 7fd005600700 1 -- 172.21.15.7:0/3637538194 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcff802b020 con 0x7fd00819def0 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:27.666+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fd0080008d0 con 0x7fcfec038f70 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.211+0000 7fd005600700 1 -- 172.21.15.7:0/3637538194 <== mgr.14120 v2:172.21.15.7:6800/1169030810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd0080008d0 con 0x7fcfec038f70 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fcfec038f70 msgr2=0x7fcfec03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 --2- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fcfec038f70 0x7fcfec03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcff0006fd0 tx=0x7fcff0006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:28.875 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd00819def0 msgr2=0x7fd00819e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 --2- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd00819def0 0x7fd00819e310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcff8004350 tx=0x7fcff800bd50 comp rx=0 tx=0).stop 2024-03-18T01:26:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 shutdown_connections 2024-03-18T01:26:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 --2- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fcfec038f70 0x7fcfec03b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 --2- 172.21.15.7:0/3637538194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd00819def0 0x7fd00819e310 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 >> 172.21.15.7:0/3637538194 conn(0x7fd0081020d0 msgr2=0x7fd0081949c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 shutdown_connections 2024-03-18T01:26:28.876 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:28.215+0000 7fd00f200700 1 -- 172.21.15.7:0/3637538194 wait complete. 2024-03-18T01:26:29.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:29 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:29.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:29 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:30.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:30 smithi007 ceph-mon[31996]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:26:30.294 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDghiXkx3ALR1rBrj2i/XmE9cMTYhhZvVm7c89NK53W5wN0xrefSn1iGPho3Lv0WzvlgsyguyBSQ95mwd2BsL5qI4B5nugdKK+SLNKJli9WQk6pOvTlWQWt5iEd06Oqt1LfnwjMRBfj2RdZiOJaruXOoTse4hBys5dc31hfFbFJ3haGBNiWzxZBJMAHS9rLLYHQrLhDAAR2Oxcw3g3Jzx0hYkl4Iey6TFGbzGfg5/F8q1/0sGX2uHN7YK4DboNjrUJMPHRTFm/BQ7h52bw0X2zl9Q9X/0bY0XW6am5ZhkG8BE1PPupjY91W2iKXWXt3MapOShvhOXFQGSn9w5/F3d0aGSYSFOH67wJjp72TD8jddsppU7rgqNUcSddwSIxHcCofQwPFOLXhzC29NZ7lxPAiF6Otkz5hpkmarNKtHMxQ+obqhRhGcBD19TXjKarQIVvE/ca9nA/UYXoMuSDks6rPPA25vzVb83fmyodMyxMnqfRTlxqemeBUz1vB6wjK000= ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:26:30.300 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.407+0000 7f610cc00700 1 Processor -- start 2024-03-18T01:26:30.300 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.408+0000 7f610cc00700 1 -- start start 2024-03-18T01:26:30.300 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.408+0000 7f610cc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f6108111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.408+0000 7f610cc00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6108082470 con 0x7f610810ec60 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.408+0000 7f6105e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f6108111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.409+0000 7f6105e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f6108111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37894/0 (socket says 172.21.15.7:37894) 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.409+0000 7f6105e00700 1 -- 172.21.15.7:0/2354903124 learned_addr learned my addr 172.21.15.7:0/2354903124 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.409+0000 7f6105e00700 1 -- 172.21.15.7:0/2354903124 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6108111590 con 0x7f610810ec60 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.410+0000 7f6105e00700 1 --2- 172.21.15.7:0/2354903124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f6108111050 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f60f80098a0 tx=0x7f60f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8896056dc026f1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.410+0000 7f6104a00700 1 -- 172.21.15.7:0/2354903124 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60f8004160 con 0x7f610810ec60 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.410+0000 7f6104a00700 1 -- 172.21.15.7:0/2354903124 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f60f8004300 con 0x7f610810ec60 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.410+0000 7f6104a00700 1 -- 172.21.15.7:0/2354903124 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60f8011b50 con 0x7f610810ec60 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.411+0000 7f610cc00700 1 -- 172.21.15.7:0/2354903124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 msgr2=0x7f6108111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.411+0000 7f610cc00700 1 --2- 172.21.15.7:0/2354903124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f6108111050 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f60f80098a0 tx=0x7f60f8009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.411+0000 7f610cc00700 1 -- 172.21.15.7:0/2354903124 shutdown_connections 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.411+0000 7f610cc00700 1 --2- 172.21.15.7:0/2354903124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f6108111050 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:30.301 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.411+0000 7f610cc00700 1 -- 172.21.15.7:0/2354903124 >> 172.21.15.7:0/2354903124 conn(0x7f610807d210 msgr2=0x7f610807f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.411+0000 7f610cc00700 1 -- 172.21.15.7:0/2354903124 shutdown_connections 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.411+0000 7f610cc00700 1 -- 172.21.15.7:0/2354903124 wait complete. 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.412+0000 7f610cc00700 1 Processor -- start 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.412+0000 7f610cc00700 1 -- start start 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.413+0000 7f610cc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f61081a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.413+0000 7f610cc00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f61081125a0 con 0x7f610810ec60 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.413+0000 7f6105e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f61081a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.413+0000 7f6105e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f61081a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37900/0 (socket says 172.21.15.7:37900) 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.413+0000 7f6105e00700 1 -- 172.21.15.7:0/2656221751 learned_addr learned my addr 172.21.15.7:0/2656221751 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.413+0000 7f6105e00700 1 -- 172.21.15.7:0/2656221751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61081a26a0 con 0x7f610810ec60 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.414+0000 7f6105e00700 1 --2- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f61081a2160 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f60f8011b60 tx=0x7f60f8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:30.302 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.414+0000 7f60fea00700 1 -- 172.21.15.7:0/2656221751 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60f8003970 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.414+0000 7f60fea00700 1 -- 172.21.15.7:0/2656221751 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f60f8003b10 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.414+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61081a2930 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.414+0000 7f60fea00700 1 -- 172.21.15.7:0/2656221751 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60f8024610 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.414+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61081a5620 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.415+0000 7f60fea00700 1 -- 172.21.15.7:0/2656221751 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f60f801a430 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.415+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6108063900 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.416+0000 7f60fea00700 1 --2- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f60ec038f70 0x7f60ec03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.416+0000 7f60fea00700 1 -- 172.21.15.7:0/2656221751 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f60f8011e00 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.416+0000 7f6105400700 1 --2- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f60ec038f70 0x7f60ec03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.417+0000 7f6105400700 1 --2- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f60ec038f70 0x7f60ec03b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f60f4006fd0 tx=0x7f60f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.421+0000 7f60fea00700 1 -- 172.21.15.7:0/2656221751 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f60f800e260 con 0x7f610810ec60 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.648+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f610807e450 con 0x7f60ec038f70 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.650+0000 7f60fea00700 1 -- 172.21.15.7:0/2656221751 <== mgr.14120 v2:172.21.15.7:6800/1169030810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f610807e450 con 0x7f60ec038f70 2024-03-18T01:26:30.303 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f60ec038f70 msgr2=0x7f60ec03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 --2- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f60ec038f70 0x7f60ec03b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f60f4006fd0 tx=0x7f60f4006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 msgr2=0x7f61081a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 --2- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f61081a2160 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f60f8011b60 tx=0x7f60f8011c40 comp rx=0 tx=0).stop 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 shutdown_connections 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 --2- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f60ec038f70 0x7f60ec03b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 --2- 172.21.15.7:0/2656221751 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f610810ec60 0x7f61081a2160 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 >> 172.21.15.7:0/2656221751 conn(0x7f610807d210 msgr2=0x7f610807dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.653+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 shutdown_connections 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:29.654+0000 7f610cc00700 1 -- 172.21.15.7:0/2656221751 wait complete. 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T01:26:30.304 INFO:teuthology.orchestra.run.smithi007.stdout:Adding host smithi007... 2024-03-18T01:26:32.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:31 smithi007 ceph-mon[31996]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:26:33.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:32 smithi007 ceph-mon[31996]: Deploying cephadm binary to smithi007 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Added host 'smithi007' with addr '172.21.15.7' 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.910+0000 7ff9aa000700 1 Processor -- start 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.911+0000 7ff9aa000700 1 -- start start 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.911+0000 7ff9aa000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a4106910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.911+0000 7ff9aa000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a4075970 con 0x7ff9a41064f0 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.912+0000 7ff9a3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a4106910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.912+0000 7ff9a3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a4106910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37904/0 (socket says 172.21.15.7:37904) 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.912+0000 7ff9a3400700 1 -- 172.21.15.7:0/2174306841 learned_addr learned my addr 172.21.15.7:0/2174306841 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:38.513 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.912+0000 7ff9a3400700 1 -- 172.21.15.7:0/2174306841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a4106e50 con 0x7ff9a41064f0 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.912+0000 7ff9a3400700 1 --2- 172.21.15.7:0/2174306841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a4106910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff994009e80 tx=0x7ff99400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5668b79c103b4653 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.913+0000 7ff9a2a00700 1 -- 172.21.15.7:0/2174306841 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff994017070 con 0x7ff9a41064f0 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.913+0000 7ff9a2a00700 1 -- 172.21.15.7:0/2174306841 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff99400b8c0 con 0x7ff9a41064f0 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.913+0000 7ff9a2a00700 1 -- 172.21.15.7:0/2174306841 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9940045c0 con 0x7ff9a41064f0 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.914+0000 7ff9aa000700 1 -- 172.21.15.7:0/2174306841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 msgr2=0x7ff9a4106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.914+0000 7ff9aa000700 1 --2- 172.21.15.7:0/2174306841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a4106910 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff994009e80 tx=0x7ff99400b330 comp rx=0 tx=0).stop 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.914+0000 7ff9aa000700 1 -- 172.21.15.7:0/2174306841 shutdown_connections 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.914+0000 7ff9aa000700 1 --2- 172.21.15.7:0/2174306841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a4106910 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.914+0000 7ff9aa000700 1 -- 172.21.15.7:0/2174306841 >> 172.21.15.7:0/2174306841 conn(0x7ff9a41020d0 msgr2=0x7ff9a41044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.914+0000 7ff9aa000700 1 -- 172.21.15.7:0/2174306841 shutdown_connections 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.914+0000 7ff9aa000700 1 -- 172.21.15.7:0/2174306841 wait complete. 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.915+0000 7ff9aa000700 1 Processor -- start 2024-03-18T01:26:38.514 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.915+0000 7ff9aa000700 1 -- start start 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.916+0000 7ff9aa000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a419dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.916+0000 7ff9aa000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a4107e70 con 0x7ff9a41064f0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.916+0000 7ff9a3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a419dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.916+0000 7ff9a3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a419dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:37914/0 (socket says 172.21.15.7:37914) 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.916+0000 7ff9a3400700 1 -- 172.21.15.7:0/188575047 learned_addr learned my addr 172.21.15.7:0/188575047 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.916+0000 7ff9a3400700 1 -- 172.21.15.7:0/188575047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a419e200 con 0x7ff9a41064f0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.916+0000 7ff9a3400700 1 --2- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a419dcc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff994004350 tx=0x7ff99400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.917+0000 7ff9a0c00700 1 -- 172.21.15.7:0/188575047 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff994020020 con 0x7ff9a41064f0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.917+0000 7ff9a0c00700 1 -- 172.21.15.7:0/188575047 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff994003c10 con 0x7ff9a41064f0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.917+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a419e490 con 0x7ff9a41064f0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.917+0000 7ff9a0c00700 1 -- 172.21.15.7:0/188575047 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff994017040 con 0x7ff9a41064f0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.917+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a41a1180 con 0x7ff9a41064f0 2024-03-18T01:26:38.515 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.918+0000 7ff9a0c00700 1 -- 172.21.15.7:0/188575047 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7ff994023020 con 0x7ff9a41064f0 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.918+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a4063900 con 0x7ff9a41064f0 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.919+0000 7ff9a0c00700 1 --2- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff988038f20 0x7ff98803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.919+0000 7ff9a0c00700 1 -- 172.21.15.7:0/188575047 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff99400f050 con 0x7ff9a41064f0 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.919+0000 7ff99aa00700 1 --2- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff988038f20 0x7ff98803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.920+0000 7ff99aa00700 1 --2- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff988038f20 0x7ff98803b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff98c006fd0 tx=0x7ff98c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:30.925+0000 7ff9a0c00700 1 -- 172.21.15.7:0/188575047 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff994053080 con 0x7ff9a41064f0 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:31.160+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}) -- 0x7ff9a402df50 con 0x7ff988038f20 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.784+0000 7ff9a0c00700 1 -- 172.21.15.7:0/188575047 <== mgr.14120 v2:172.21.15.7:6800/1169030810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7ff9a402df50 con 0x7ff988038f20 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.788+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff988038f20 msgr2=0x7ff98803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:38.516 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.788+0000 7ff9aa000700 1 --2- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff988038f20 0x7ff98803b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff98c006fd0 tx=0x7ff98c006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.788+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 msgr2=0x7ff9a419dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.788+0000 7ff9aa000700 1 --2- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a419dcc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff994004350 tx=0x7ff99400bd50 comp rx=0 tx=0).stop 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.788+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 shutdown_connections 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.788+0000 7ff9aa000700 1 --2- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7ff988038f20 0x7ff98803b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.788+0000 7ff9aa000700 1 --2- 172.21.15.7:0/188575047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9a41064f0 0x7ff9a419dcc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.788+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 >> 172.21.15.7:0/188575047 conn(0x7ff9a41020d0 msgr2=0x7ff9a4194960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.789+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 shutdown_connections 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:37.789+0000 7ff9aa000700 1 -- 172.21.15.7:0/188575047 wait complete. 2024-03-18T01:26:38.517 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mon service... 2024-03-18T01:26:39.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:38 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:39.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:38 smithi007 ceph-mon[31996]: Added host smithi007 2024-03-18T01:26:39.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:38 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:26:40.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:40 smithi007 ceph-mon[31996]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:26:40.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:40 smithi007 ceph-mon[31996]: Saving service mon spec with placement count:5 2024-03-18T01:26:40.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:40 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:41.281 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T01:26:41.281 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.206+0000 7f37e7400700 1 Processor -- start 2024-03-18T01:26:41.281 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.206+0000 7f37e7400700 1 -- start start 2024-03-18T01:26:41.281 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.206+0000 7f37e7400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e010aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:41.281 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.206+0000 7f37e7400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f37e00759f0 con 0x7f37e0108680 2024-03-18T01:26:41.281 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.207+0000 7f37e4a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e010aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:41.281 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.207+0000 7f37e4a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e010aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49056/0 (socket says 172.21.15.7:49056) 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.207+0000 7f37e4a00700 1 -- 172.21.15.7:0/293998650 learned_addr learned my addr 172.21.15.7:0/293998650 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.208+0000 7f37e4a00700 1 -- 172.21.15.7:0/293998650 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e010afb0 con 0x7f37e0108680 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.208+0000 7f37e4a00700 1 --2- 172.21.15.7:0/293998650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e010aa70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f37d40098a0 tx=0x7f37d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a586a3a9c14eea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.209+0000 7f37df400700 1 -- 172.21.15.7:0/293998650 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37d4004160 con 0x7f37e0108680 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.209+0000 7f37df400700 1 -- 172.21.15.7:0/293998650 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f37d4004300 con 0x7f37e0108680 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.209+0000 7f37df400700 1 -- 172.21.15.7:0/293998650 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37d4011b50 con 0x7f37e0108680 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.209+0000 7f37e7400700 1 -- 172.21.15.7:0/293998650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 msgr2=0x7f37e010aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.209+0000 7f37e7400700 1 --2- 172.21.15.7:0/293998650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e010aa70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f37d40098a0 tx=0x7f37d4009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.210+0000 7f37e7400700 1 -- 172.21.15.7:0/293998650 shutdown_connections 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.210+0000 7f37e7400700 1 --2- 172.21.15.7:0/293998650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e010aa70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.210+0000 7f37e7400700 1 -- 172.21.15.7:0/293998650 >> 172.21.15.7:0/293998650 conn(0x7f37e01020d0 msgr2=0x7f37e01044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.210+0000 7f37e7400700 1 -- 172.21.15.7:0/293998650 shutdown_connections 2024-03-18T01:26:41.282 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.210+0000 7f37e7400700 1 -- 172.21.15.7:0/293998650 wait complete. 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.211+0000 7f37e7400700 1 Processor -- start 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.211+0000 7f37e7400700 1 -- start start 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.211+0000 7f37e7400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.211+0000 7f37e7400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f37e010bf70 con 0x7f37e0108680 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.212+0000 7f37e4a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.212+0000 7f37e4a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e019dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49066/0 (socket says 172.21.15.7:49066) 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.212+0000 7f37e4a00700 1 -- 172.21.15.7:0/171112132 learned_addr learned my addr 172.21.15.7:0/171112132 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.212+0000 7f37e4a00700 1 -- 172.21.15.7:0/171112132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e019e250 con 0x7f37e0108680 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.212+0000 7f37e4a00700 1 --2- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e019dd10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f37d4011670 tx=0x7f37d4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.213+0000 7f37dd600700 1 -- 172.21.15.7:0/171112132 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37d40039c0 con 0x7f37e0108680 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.213+0000 7f37dd600700 1 -- 172.21.15.7:0/171112132 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f37d4011e50 con 0x7f37e0108680 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.213+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e019e4e0 con 0x7f37e0108680 2024-03-18T01:26:41.283 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.213+0000 7f37dd600700 1 -- 172.21.15.7:0/171112132 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37d401a5c0 con 0x7f37e0108680 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.213+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e01a11c0 con 0x7f37e0108680 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.214+0000 7f37dd600700 1 -- 172.21.15.7:0/171112132 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f37d401a870 con 0x7f37e0108680 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.214+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e0063900 con 0x7f37e0108680 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.214+0000 7f37dd600700 1 --2- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f37c8038f20 0x7f37c803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.214+0000 7f37dd600700 1 -- 172.21.15.7:0/171112132 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f37d404d130 con 0x7f37e0108680 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.215+0000 7f37dfe00700 1 --2- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f37c8038f20 0x7f37c803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.216+0000 7f37dfe00700 1 --2- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f37c8038f20 0x7f37c803b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f37cc006fd0 tx=0x7f37cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.220+0000 7f37dd600700 1 -- 172.21.15.7:0/171112132 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37d4016260 con 0x7f37e0108680 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.505+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f37e01032d0 con 0x7f37c8038f20 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.516+0000 7f37dd600700 1 -- 172.21.15.7:0/171112132 <== mgr.14120 v2:172.21.15.7:6800/1169030810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f37e01032d0 con 0x7f37c8038f20 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f37c8038f20 msgr2=0x7f37c803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 --2- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f37c8038f20 0x7f37c803b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f37cc006fd0 tx=0x7f37cc006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:41.284 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 msgr2=0x7f37e019dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 --2- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e019dd10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f37d4011670 tx=0x7f37d4011b60 comp rx=0 tx=0).stop 2024-03-18T01:26:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 shutdown_connections 2024-03-18T01:26:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 --2- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f37c8038f20 0x7f37c803b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 --2- 172.21.15.7:0/171112132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37e0108680 0x7f37e019dd10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 >> 172.21.15.7:0/171112132 conn(0x7f37e01020d0 msgr2=0x7f37e0102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.519+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 shutdown_connections 2024-03-18T01:26:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:39.520+0000 7f37e7400700 1 -- 172.21.15.7:0/171112132 wait complete. 2024-03-18T01:26:41.285 INFO:teuthology.orchestra.run.smithi007.stdout:Deploying unmanaged mgr service... 2024-03-18T01:26:42.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:41 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:42.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:41 smithi007 ceph-mon[31996]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:26:42.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:41 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.832+0000 7f388e000700 1 Processor -- start 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.832+0000 7f388e000700 1 -- start start 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.833+0000 7f388e000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888106f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.833+0000 7f388e000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3888107500 con 0x7f3888106b10 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.833+0000 7f3887400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888106f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.833+0000 7f3887400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888106f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49068/0 (socket says 172.21.15.7:49068) 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.833+0000 7f3887400700 1 -- 172.21.15.7:0/112885427 learned_addr learned my addr 172.21.15.7:0/112885427 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.834+0000 7f3887400700 1 -- 172.21.15.7:0/112885427 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3888107d10 con 0x7f3888106b10 2024-03-18T01:26:42.711 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.834+0000 7f3887400700 1 --2- 172.21.15.7:0/112885427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888106f30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38780098a0 tx=0x7f3878009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62495d81740fb759 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.834+0000 7f3886000700 1 -- 172.21.15.7:0/112885427 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3878004160 con 0x7f3888106b10 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.834+0000 7f3886000700 1 -- 172.21.15.7:0/112885427 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3878004300 con 0x7f3888106b10 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.834+0000 7f3886000700 1 -- 172.21.15.7:0/112885427 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3878011b20 con 0x7f3888106b10 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.835+0000 7f388e000700 1 -- 172.21.15.7:0/112885427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 msgr2=0x7f3888106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.835+0000 7f388e000700 1 --2- 172.21.15.7:0/112885427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888106f30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f38780098a0 tx=0x7f3878009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.836+0000 7f388e000700 1 -- 172.21.15.7:0/112885427 shutdown_connections 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.836+0000 7f388e000700 1 --2- 172.21.15.7:0/112885427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888106f30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.836+0000 7f388e000700 1 -- 172.21.15.7:0/112885427 >> 172.21.15.7:0/112885427 conn(0x7f38881020d0 msgr2=0x7f38881044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.836+0000 7f388e000700 1 -- 172.21.15.7:0/112885427 shutdown_connections 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.836+0000 7f388e000700 1 -- 172.21.15.7:0/112885427 wait complete. 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.837+0000 7f388e000700 1 Processor -- start 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.837+0000 7f388e000700 1 -- start start 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.837+0000 7f388e000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888191c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.837+0000 7f388e000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f388810b0c0 con 0x7f3888106b10 2024-03-18T01:26:42.712 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.837+0000 7f3887400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888191c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.838+0000 7f3887400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888191c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49072/0 (socket says 172.21.15.7:49072) 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.838+0000 7f3887400700 1 -- 172.21.15.7:0/3996433557 learned_addr learned my addr 172.21.15.7:0/3996433557 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.838+0000 7f3887400700 1 -- 172.21.15.7:0/3996433557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3888192170 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.838+0000 7f3887400700 1 --2- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888191c30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38780117e0 tx=0x7f3878011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.839+0000 7f387fe00700 1 -- 172.21.15.7:0/3996433557 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3878003bf0 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.839+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38881903b0 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.839+0000 7f387fe00700 1 -- 172.21.15.7:0/3996433557 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f387801a910 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.839+0000 7f387fe00700 1 -- 172.21.15.7:0/3996433557 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3878024970 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.839+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3888190810 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.840+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3888063900 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.840+0000 7f387fe00700 1 -- 172.21.15.7:0/3996433557 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f3878022020 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.841+0000 7f387fe00700 1 --2- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f386c038f20 0x7f386c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.841+0000 7f387fe00700 1 -- 172.21.15.7:0/3996433557 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f387804d810 con 0x7f3888106b10 2024-03-18T01:26:42.713 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.841+0000 7f3886a00700 1 --2- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f386c038f20 0x7f386c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.842+0000 7f3886a00700 1 --2- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f386c038f20 0x7f386c03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3874006fd0 tx=0x7f3874006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:41.846+0000 7f387fe00700 1 -- 172.21.15.7:0/3996433557 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f387804d160 con 0x7f3888106b10 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.092+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f3888195930 con 0x7f386c038f20 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.099+0000 7f387fe00700 1 -- 172.21.15.7:0/3996433557 <== mgr.14120 v2:172.21.15.7:6800/1169030810 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3888195930 con 0x7f386c038f20 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.101+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f386c038f20 msgr2=0x7f386c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.101+0000 7f388e000700 1 --2- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f386c038f20 0x7f386c03b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3874006fd0 tx=0x7f3874006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.101+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 msgr2=0x7f3888191c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.101+0000 7f388e000700 1 --2- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888191c30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f38780117e0 tx=0x7f3878011ba0 comp rx=0 tx=0).stop 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.102+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 shutdown_connections 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.102+0000 7f388e000700 1 --2- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f386c038f20 0x7f386c03b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.102+0000 7f388e000700 1 --2- 172.21.15.7:0/3996433557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3888106b10 0x7f3888191c30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:42.714 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.102+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 >> 172.21.15.7:0/3996433557 conn(0x7f38881020d0 msgr2=0x7f3888102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:42.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.102+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 shutdown_connections 2024-03-18T01:26:42.715 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:42.102+0000 7f388e000700 1 -- 172.21.15.7:0/3996433557 wait complete. 2024-03-18T01:26:43.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:43 smithi007 ceph-mon[31996]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:26:43.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:43 smithi007 ceph-mon[31996]: Saving service mgr spec with placement count:2 2024-03-18T01:26:43.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:43 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:44.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.422+0000 7f1545200700 1 Processor -- start 2024-03-18T01:26:44.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.422+0000 7f1545200700 1 -- start start 2024-03-18T01:26:44.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.422+0000 7f1545200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f1540106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:44.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.422+0000 7f1545200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1540075950 con 0x7f1540106510 2024-03-18T01:26:44.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.423+0000 7f153de00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f1540106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:44.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.423+0000 7f153de00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f1540106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49074/0 (socket says 172.21.15.7:49074) 2024-03-18T01:26:44.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.423+0000 7f153de00700 1 -- 172.21.15.7:0/1976368823 learned_addr learned my addr 172.21.15.7:0/1976368823 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:44.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.423+0000 7f153de00700 1 -- 172.21.15.7:0/1976368823 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1540106e70 con 0x7f1540106510 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.424+0000 7f153de00700 1 --2- 172.21.15.7:0/1976368823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f1540106930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f15300098a0 tx=0x7f1530009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b821b3c0f8bc82da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.424+0000 7f153ca00700 1 -- 172.21.15.7:0/1976368823 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1530004160 con 0x7f1540106510 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.424+0000 7f153ca00700 1 -- 172.21.15.7:0/1976368823 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1530004300 con 0x7f1540106510 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.424+0000 7f153ca00700 1 -- 172.21.15.7:0/1976368823 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1530011b20 con 0x7f1540106510 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.425+0000 7f1545200700 1 -- 172.21.15.7:0/1976368823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 msgr2=0x7f1540106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.425+0000 7f1545200700 1 --2- 172.21.15.7:0/1976368823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f1540106930 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f15300098a0 tx=0x7f1530009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.425+0000 7f1545200700 1 -- 172.21.15.7:0/1976368823 shutdown_connections 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.425+0000 7f1545200700 1 --2- 172.21.15.7:0/1976368823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f1540106930 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.425+0000 7f1545200700 1 -- 172.21.15.7:0/1976368823 >> 172.21.15.7:0/1976368823 conn(0x7f15401020d0 msgr2=0x7f15401044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.425+0000 7f1545200700 1 -- 172.21.15.7:0/1976368823 shutdown_connections 2024-03-18T01:26:44.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.425+0000 7f1545200700 1 -- 172.21.15.7:0/1976368823 wait complete. 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.426+0000 7f1545200700 1 Processor -- start 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.427+0000 7f1545200700 1 -- start start 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.427+0000 7f1545200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f154010bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.427+0000 7f1545200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1540107f40 con 0x7f1540106510 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.428+0000 7f153de00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f154010bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.428+0000 7f153de00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f154010bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49076/0 (socket says 172.21.15.7:49076) 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.428+0000 7f153de00700 1 -- 172.21.15.7:0/1027969719 learned_addr learned my addr 172.21.15.7:0/1027969719 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.428+0000 7f153de00700 1 -- 172.21.15.7:0/1027969719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154010c4f0 con 0x7f1540106510 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.429+0000 7f153de00700 1 --2- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f154010bfb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f15300117e0 tx=0x7f1530011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.429+0000 7f1536a00700 1 -- 172.21.15.7:0/1027969719 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1530003bf0 con 0x7f1540106510 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.429+0000 7f1536a00700 1 -- 172.21.15.7:0/1027969719 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f153001a910 con 0x7f1540106510 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.429+0000 7f1536a00700 1 -- 172.21.15.7:0/1027969719 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1530024970 con 0x7f1540106510 2024-03-18T01:26:44.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.429+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154010c780 con 0x7f1540106510 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.429+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1540108cc0 con 0x7f1540106510 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.431+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1520005360 con 0x7f1540106510 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.431+0000 7f1536a00700 1 -- 172.21.15.7:0/1027969719 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f1530022020 con 0x7f1540106510 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.432+0000 7f1536a00700 1 --2- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f1524038f20 0x7f152403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.432+0000 7f1536a00700 1 -- 172.21.15.7:0/1027969719 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f153004d250 con 0x7f1540106510 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.432+0000 7f153d400700 1 --2- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f1524038f20 0x7f152403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.433+0000 7f153d400700 1 --2- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f1524038f20 0x7f152403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f152c006fd0 tx=0x7f152c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.438+0000 7f1536a00700 1 -- 172.21.15.7:0/1027969719 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1530011e20 con 0x7f1540106510 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.702+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f1520005650 con 0x7f1540106510 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.708+0000 7f1536a00700 1 -- 172.21.15.7:0/1027969719 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f153002ddb0 con 0x7f1540106510 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.711+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f1524038f20 msgr2=0x7f152403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.711+0000 7f1545200700 1 --2- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f1524038f20 0x7f152403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f152c006fd0 tx=0x7f152c006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.711+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 msgr2=0x7f154010bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:44.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.711+0000 7f1545200700 1 --2- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f154010bfb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f15300117e0 tx=0x7f1530011c10 comp rx=0 tx=0).stop 2024-03-18T01:26:44.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.711+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 shutdown_connections 2024-03-18T01:26:44.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.711+0000 7f1545200700 1 --2- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f1524038f20 0x7f152403b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:44.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.711+0000 7f1545200700 1 --2- 172.21.15.7:0/1027969719 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1540106510 0x7f154010bfb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:44.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.711+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 >> 172.21.15.7:0/1027969719 conn(0x7f15401020d0 msgr2=0x7f1540194930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:44.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.712+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 shutdown_connections 2024-03-18T01:26:44.412 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:43.712+0000 7f1545200700 1 -- 172.21.15.7:0/1027969719 wait complete. 2024-03-18T01:26:45.039 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:44 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1027969719' entity='client.admin' 2024-03-18T01:26:45.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:44 smithi007 ceph-mon[31996]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:26:46.465 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.464+0000 7fec9e000700 1 Processor -- start 2024-03-18T01:26:46.466 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.464+0000 7fec9e000700 1 -- start start 2024-03-18T01:26:46.466 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.464+0000 7fec9e000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec9810aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:46.466 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.464+0000 7fec9e000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fec980759f0 con 0x7fec98108680 2024-03-18T01:26:46.466 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.465+0000 7fec97400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec9810aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:46.466 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.465+0000 7fec97400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec9810aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49080/0 (socket says 172.21.15.7:49080) 2024-03-18T01:26:46.467 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.465+0000 7fec97400700 1 -- 172.21.15.7:0/591654995 learned_addr learned my addr 172.21.15.7:0/591654995 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:46.467 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.465+0000 7fec97400700 1 -- 172.21.15.7:0/591654995 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9810afb0 con 0x7fec98108680 2024-03-18T01:26:46.467 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.466+0000 7fec97400700 1 --2- 172.21.15.7:0/591654995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec9810aa70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec880098a0 tx=0x7fec88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeabcc01e710bef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:46.467 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.466+0000 7fec96a00700 1 -- 172.21.15.7:0/591654995 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec88004160 con 0x7fec98108680 2024-03-18T01:26:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.466+0000 7fec96a00700 1 -- 172.21.15.7:0/591654995 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec88004300 con 0x7fec98108680 2024-03-18T01:26:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.466+0000 7fec96a00700 1 -- 172.21.15.7:0/591654995 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec88011b20 con 0x7fec98108680 2024-03-18T01:26:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.467+0000 7fec9e000700 1 -- 172.21.15.7:0/591654995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 msgr2=0x7fec9810aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.467+0000 7fec9e000700 1 --2- 172.21.15.7:0/591654995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec9810aa70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fec880098a0 tx=0x7fec88009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.467+0000 7fec9e000700 1 -- 172.21.15.7:0/591654995 shutdown_connections 2024-03-18T01:26:46.468 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.467+0000 7fec9e000700 1 --2- 172.21.15.7:0/591654995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec9810aa70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.467+0000 7fec9e000700 1 -- 172.21.15.7:0/591654995 >> 172.21.15.7:0/591654995 conn(0x7fec981020d0 msgr2=0x7fec981044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.467+0000 7fec9e000700 1 -- 172.21.15.7:0/591654995 shutdown_connections 2024-03-18T01:26:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.467+0000 7fec9e000700 1 -- 172.21.15.7:0/591654995 wait complete. 2024-03-18T01:26:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.468+0000 7fec9e000700 1 Processor -- start 2024-03-18T01:26:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.469+0000 7fec9e000700 1 -- start start 2024-03-18T01:26:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.469+0000 7fec9e000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec98107f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:46.470 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.469+0000 7fec9e000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fec9810c020 con 0x7fec98108680 2024-03-18T01:26:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.469+0000 7fec97400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec98107f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.469+0000 7fec97400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec98107f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49090/0 (socket says 172.21.15.7:49090) 2024-03-18T01:26:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.469+0000 7fec97400700 1 -- 172.21.15.7:0/983311584 learned_addr learned my addr 172.21.15.7:0/983311584 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.470+0000 7fec97400700 1 -- 172.21.15.7:0/983311584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec98108440 con 0x7fec98108680 2024-03-18T01:26:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.470+0000 7fec97400700 1 --2- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec98107f00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fec880117e0 tx=0x7fec88011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.470+0000 7fec94c00700 1 -- 172.21.15.7:0/983311584 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec88003ba0 con 0x7fec98108680 2024-03-18T01:26:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.470+0000 7fec94c00700 1 -- 172.21.15.7:0/983311584 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec8801a910 con 0x7fec98108680 2024-03-18T01:26:46.471 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.470+0000 7fec94c00700 1 -- 172.21.15.7:0/983311584 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec880248f0 con 0x7fec98108680 2024-03-18T01:26:46.472 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.470+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec98106630 con 0x7fec98108680 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.471+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec98106af0 con 0x7fec98108680 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.472+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec98063900 con 0x7fec98108680 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.472+0000 7fec94c00700 1 -- 172.21.15.7:0/983311584 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fec88022020 con 0x7fec98108680 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.473+0000 7fec94c00700 1 --2- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fec7c038f70 0x7fec7c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.473+0000 7fec94c00700 1 -- 172.21.15.7:0/983311584 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fec8804d3c0 con 0x7fec98108680 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.474+0000 7fec8fe00700 1 --2- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fec7c038f70 0x7fec7c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.474+0000 7fec8fe00700 1 --2- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fec7c038f70 0x7fec7c03b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.479+0000 7fec94c00700 1 -- 172.21.15.7:0/983311584 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec8801ab70 con 0x7fec98108680 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.708+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fec98106d70 con 0x7fec98108680 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.714+0000 7fec94c00700 1 -- 172.21.15.7:0/983311584 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fec88017ca0 con 0x7fec98108680 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.717+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fec7c038f70 msgr2=0x7fec7c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.717+0000 7fec9e000700 1 --2- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fec7c038f70 0x7fec7c03b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.717+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 msgr2=0x7fec98107f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:46.473 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.717+0000 7fec9e000700 1 --2- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec98107f00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fec880117e0 tx=0x7fec88011ba0 comp rx=0 tx=0).stop 2024-03-18T01:26:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.717+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 shutdown_connections 2024-03-18T01:26:46.474 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.717+0000 7fec9e000700 1 --2- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fec7c038f70 0x7fec7c03b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:46.475 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.717+0000 7fec9e000700 1 --2- 172.21.15.7:0/983311584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec98108680 0x7fec98107f00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:46.475 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.717+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 >> 172.21.15.7:0/983311584 conn(0x7fec981020d0 msgr2=0x7fec98102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:46.475 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.718+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 shutdown_connections 2024-03-18T01:26:46.475 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:45.719+0000 7fec9e000700 1 -- 172.21.15.7:0/983311584 wait complete. 2024-03-18T01:26:46.475 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling the dashboard module... 2024-03-18T01:26:46.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:46 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:46.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:46 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/983311584' entity='client.admin' 2024-03-18T01:26:46.754 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:46 smithi007 ceph-mon[31996]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:26:48.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:47 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1218908039' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T01:26:48.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:47 smithi007 ceph-mon[31996]: from='mgr.14120 172.21.15.7:0/560079525' entity='mgr.x' 2024-03-18T01:26:48.052 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:47 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: ignoring --setuser ceph since I am not root 2024-03-18T01:26:48.052 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:47 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: ignoring --setgroup ceph since I am not root 2024-03-18T01:26:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.045+0000 7f0506a00700 1 Processor -- start 2024-03-18T01:26:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.045+0000 7f0506a00700 1 -- start start 2024-03-18T01:26:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.046+0000 7f0506a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050007b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.046+0000 7f0506a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f050007b750 con 0x7f050007cd20 2024-03-18T01:26:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.046+0000 7f04ffe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050007b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.047+0000 7f04ffe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050007b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55874/0 (socket says 172.21.15.7:55874) 2024-03-18T01:26:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.047+0000 7f04ffe00700 1 -- 172.21.15.7:0/3486323649 learned_addr learned my addr 172.21.15.7:0/3486323649 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:48.433 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.047+0000 7f04ffe00700 1 -- 172.21.15.7:0/3486323649 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050007b8d0 con 0x7f050007cd20 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.047+0000 7f04ffe00700 1 --2- 172.21.15.7:0/3486323649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050007b180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f04f00098a0 tx=0x7f04f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93b081f383878dec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.048+0000 7f04fea00700 1 -- 172.21.15.7:0/3486323649 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f0004160 con 0x7f050007cd20 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.048+0000 7f04fea00700 1 -- 172.21.15.7:0/3486323649 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f0004300 con 0x7f050007cd20 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.048+0000 7f04fea00700 1 -- 172.21.15.7:0/3486323649 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f0011b20 con 0x7f050007cd20 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.048+0000 7f0506a00700 1 -- 172.21.15.7:0/3486323649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 msgr2=0x7f050007b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.048+0000 7f0506a00700 1 --2- 172.21.15.7:0/3486323649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050007b180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f04f00098a0 tx=0x7f04f0009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.049+0000 7f0506a00700 1 -- 172.21.15.7:0/3486323649 shutdown_connections 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.049+0000 7f0506a00700 1 --2- 172.21.15.7:0/3486323649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050007b180 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.049+0000 7f0506a00700 1 -- 172.21.15.7:0/3486323649 >> 172.21.15.7:0/3486323649 conn(0x7f05001031e0 msgr2=0x7f0500105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.049+0000 7f0506a00700 1 -- 172.21.15.7:0/3486323649 shutdown_connections 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.049+0000 7f0506a00700 1 -- 172.21.15.7:0/3486323649 wait complete. 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.050+0000 7f0506a00700 1 Processor -- start 2024-03-18T01:26:48.434 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.050+0000 7f0506a00700 1 -- start start 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.050+0000 7f0506a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.050+0000 7f0506a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f050010e280 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.050+0000 7f04ffe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.051+0000 7f04ffe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55878/0 (socket says 172.21.15.7:55878) 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.051+0000 7f04ffe00700 1 -- 172.21.15.7:0/1218908039 learned_addr learned my addr 172.21.15.7:0/1218908039 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.051+0000 7f04ffe00700 1 -- 172.21.15.7:0/1218908039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050019e2e0 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.051+0000 7f04ffe00700 1 --2- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050019dda0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f04f0000c00 tx=0x7f04f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.052+0000 7f04fcc00700 1 -- 172.21.15.7:0/1218908039 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f0003b60 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.052+0000 7f04fcc00700 1 -- 172.21.15.7:0/1218908039 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04f001a910 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.052+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050019e570 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.052+0000 7f04fcc00700 1 -- 172.21.15.7:0/1218908039 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f04f0024930 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.052+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05001a1220 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.053+0000 7f04fcc00700 1 -- 172.21.15.7:0/1218908039 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f04f0022020 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.053+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0500063900 con 0x7f050007cd20 2024-03-18T01:26:48.435 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.054+0000 7f04fcc00700 1 --2- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f04e4038fd0 0x7f04e403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.054+0000 7f04fcc00700 1 -- 172.21.15.7:0/1218908039 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f04f004d0e0 con 0x7f050007cd20 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.054+0000 7f04ff400700 1 --2- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f04e4038fd0 0x7f04e403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.055+0000 7f04ff400700 1 --2- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f04e4038fd0 0x7f04e403b490 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f04ec006fd0 tx=0x7f04ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.059+0000 7f04fcc00700 1 -- 172.21.15.7:0/1218908039 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04f00172b0 con 0x7f050007cd20 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.347+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f050019e9d0 con 0x7f050007cd20 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.718+0000 7f04fcc00700 1 -- 172.21.15.7:0/1218908039 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f04f002c510 con 0x7f050007cd20 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.720+0000 7f04fcc00700 1 -- 172.21.15.7:0/1218908039 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f04f004f0e0 con 0x7f050007cd20 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.722+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f04e4038fd0 msgr2=0x7f04e403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.722+0000 7f0506a00700 1 --2- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f04e4038fd0 0x7f04e403b490 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f04ec006fd0 tx=0x7f04ec006e40 comp rx=0 tx=0).stop 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.723+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 msgr2=0x7f050019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.723+0000 7f0506a00700 1 --2- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050019dda0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f04f0000c00 tx=0x7f04f0011c10 comp rx=0 tx=0).stop 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.723+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 shutdown_connections 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.723+0000 7f0506a00700 1 --2- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f04e4038fd0 0x7f04e403b490 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:48.436 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.723+0000 7f0506a00700 1 --2- 172.21.15.7:0/1218908039 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f050007cd20 0x7f050019dda0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:48.437 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.723+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 >> 172.21.15.7:0/1218908039 conn(0x7f05001031e0 msgr2=0x7f0500103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:48.437 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.723+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 shutdown_connections 2024-03-18T01:26:48.437 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:47.723+0000 7f0506a00700 1 -- 172.21.15.7:0/1218908039 wait complete. 2024-03-18T01:26:48.715 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:48 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:48.593+0000 7faa549c9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T01:26:49.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:48 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1218908039' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T01:26:49.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:48 smithi007 ceph-mon[31996]: mgrmap e9: x(active, since 25s) 2024-03-18T01:26:49.048 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:48 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:48.720+0000 7faa549c9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T01:26:49.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:49 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3841034857' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T01:26:49.910 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:49 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:49.590+0000 7faa549c9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T01:26:49.910 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:49 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:49.716+0000 7faa549c9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T01:26:50.162 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:49 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:49.907+0000 7faa549c9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T01:26:50.162 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:50 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:50.046+0000 7faa549c9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T01:26:50.511 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:50 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:50.163+0000 7faa549c9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T01:26:50.619 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:26:50.619 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T01:26:50.619 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.046+0000 7f4f42400700 1 Processor -- start 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.046+0000 7f4f42400700 1 -- start start 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.047+0000 7f4f42400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c07b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.047+0000 7f4f42400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c07b730 con 0x7f4f3c07cd00 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.047+0000 7f4f3b400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c07b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.047+0000 7f4f3b400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c07b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55902/0 (socket says 172.21.15.7:55902) 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.048+0000 7f4f3b400700 1 -- 172.21.15.7:0/2956547851 learned_addr learned my addr 172.21.15.7:0/2956547851 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.048+0000 7f4f3b400700 1 -- 172.21.15.7:0/2956547851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c07b8b0 con 0x7f4f3c07cd00 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.048+0000 7f4f3b400700 1 --2- 172.21.15.7:0/2956547851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c07b160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0098a0 tx=0x7f4f2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfb49bfed635f83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.049+0000 7f4f3a000700 1 -- 172.21.15.7:0/2956547851 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f2c004160 con 0x7f4f3c07cd00 2024-03-18T01:26:50.620 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.049+0000 7f4f3a000700 1 -- 172.21.15.7:0/2956547851 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2c004300 con 0x7f4f3c07cd00 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.049+0000 7f4f3a000700 1 -- 172.21.15.7:0/2956547851 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f2c011b20 con 0x7f4f3c07cd00 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.050+0000 7f4f42400700 1 -- 172.21.15.7:0/2956547851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 msgr2=0x7f4f3c07b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.050+0000 7f4f42400700 1 --2- 172.21.15.7:0/2956547851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c07b160 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c0098a0 tx=0x7f4f2c009bb0 comp rx=0 tx=0).stop 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.050+0000 7f4f42400700 1 -- 172.21.15.7:0/2956547851 shutdown_connections 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.050+0000 7f4f42400700 1 --2- 172.21.15.7:0/2956547851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c07b160 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.050+0000 7f4f42400700 1 -- 172.21.15.7:0/2956547851 >> 172.21.15.7:0/2956547851 conn(0x7f4f3c1031e0 msgr2=0x7f4f3c105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.050+0000 7f4f42400700 1 -- 172.21.15.7:0/2956547851 shutdown_connections 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.050+0000 7f4f42400700 1 -- 172.21.15.7:0/2956547851 wait complete. 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.052+0000 7f4f42400700 1 Processor -- start 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.052+0000 7f4f42400700 1 -- start start 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.052+0000 7f4f42400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.052+0000 7f4f42400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c10e2a0 con 0x7f4f3c07cd00 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.053+0000 7f4f3b400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.053+0000 7f4f3b400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55908/0 (socket says 172.21.15.7:55908) 2024-03-18T01:26:50.621 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.053+0000 7f4f3b400700 1 -- 172.21.15.7:0/3841034857 learned_addr learned my addr 172.21.15.7:0/3841034857 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.053+0000 7f4f3b400700 1 -- 172.21.15.7:0/3841034857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c19e250 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.053+0000 7f4f3b400700 1 --2- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c19dd10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c011b60 tx=0x7f4f2c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.054+0000 7f4f33e00700 1 -- 172.21.15.7:0/3841034857 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f2c003b60 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.054+0000 7f4f33e00700 1 -- 172.21.15.7:0/3841034857 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f2c01a910 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.054+0000 7f4f33e00700 1 -- 172.21.15.7:0/3841034857 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f2c024930 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.054+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3c19e4e0 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.054+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3c19e940 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.055+0000 7f4f33e00700 1 -- 172.21.15.7:0/3841034857 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f4f2c022020 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.055+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f3c063900 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.056+0000 7f4f33e00700 1 --2- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f4f20038f70 0x7f4f2003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.056+0000 7f4f33e00700 1 -- 172.21.15.7:0/3841034857 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4f2c04d280 con 0x7f4f3c07cd00 2024-03-18T01:26:50.622 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.056+0000 7f4f3aa00700 1 -- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f4f20038f70 msgr2=0x7f4f2003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1169030810 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.056+0000 7f4f3aa00700 1 --2- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f4f20038f70 0x7f4f2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.061+0000 7f4f33e00700 1 -- 172.21.15.7:0/3841034857 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4f2c04cbd0 con 0x7f4f3c07cd00 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.257+0000 7f4f3aa00700 1 -- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f4f20038f70 msgr2=0x7f4f2003b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1169030810 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.257+0000 7f4f3aa00700 1 --2- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f4f20038f70 0x7f4f2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.384+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f4f3c1a13e0 con 0x7f4f3c07cd00 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.385+0000 7f4f33e00700 1 -- 172.21.15.7:0/3841034857 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f4f2c01a4e0 con 0x7f4f3c07cd00 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.388+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f4f20038f70 msgr2=0x7f4f2003b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.388+0000 7f4f42400700 1 --2- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f4f20038f70 0x7f4f2003b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.388+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 msgr2=0x7f4f3c19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.388+0000 7f4f42400700 1 --2- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c19dd10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c011b60 tx=0x7f4f2c011c40 comp rx=0 tx=0).stop 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.388+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 shutdown_connections 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.388+0000 7f4f42400700 1 --2- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7f4f20038f70 0x7f4f2003b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.388+0000 7f4f42400700 1 --2- 172.21.15.7:0/3841034857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4f3c07cd00 0x7f4f3c19dd10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:50.623 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.388+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 >> 172.21.15.7:0/3841034857 conn(0x7f4f3c1031e0 msgr2=0x7f4f3c103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:50.624 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.389+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 shutdown_connections 2024-03-18T01:26:50.624 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:49.389+0000 7f4f42400700 1 -- 172.21.15.7:0/3841034857 wait complete. 2024-03-18T01:26:50.624 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for the mgr to restart... 2024-03-18T01:26:50.624 INFO:teuthology.orchestra.run.smithi007.stdout:Waiting for mgr epoch 9... 2024-03-18T01:26:50.787 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:50 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:50.509+0000 7faa549c9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T01:26:50.787 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:50 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:50.669+0000 7faa549c9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T01:26:51.046 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:50 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:50.785+0000 7faa549c9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T01:26:51.046 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:50 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:50.919+0000 7faa549c9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T01:26:51.301 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:51 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:51.151+0000 7faa549c9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T01:26:52.801 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:52 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:52.470+0000 7faa549c9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T01:26:52.802 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:52 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:52.648+0000 7faa549c9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T01:26:53.471 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:53 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:53.077+0000 7faa549c9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T01:26:53.471 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:53 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:53.200+0000 7faa549c9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T01:26:53.471 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:53 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:53.469+0000 7faa549c9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:26:54.051 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:53 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:53.724+0000 7faa549c9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T01:26:54.444 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:54 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:54.117+0000 7faa549c9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:26:54.444 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:54 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:54.442+0000 7faa549c9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T01:26:55.051 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:54 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:54.759+0000 7faa549c9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T01:26:56.052 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:55 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:55.860+0000 7faa549c9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T01:26:57.022 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:56 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:56.590+0000 7faa549c9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T01:26:57.022 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:56 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:56.727+0000 7faa549c9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T01:26:57.301 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:26:57.019+0000 7faa549c9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: Active manager daemon x restarted 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: Activating manager daemon x 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: mgrmap e10: x(active, starting, since 0.0106179s) 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T01:26:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:57 smithi007 ceph-mon[31996]: Manager daemon x is now available 2024-03-18T01:26:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:26:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:26:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T01:26:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T01:26:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:58 smithi007 ceph-mon[31996]: mgrmap e11: x(active, since 1.01202s) 2024-03-18T01:26:58.723 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout { 2024-03-18T01:26:58.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T01:26:58.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T01:26:58.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout } 2024-03-18T01:26:58.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.191+0000 7fe900400700 1 Processor -- start 2024-03-18T01:26:58.724 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.191+0000 7fe900400700 1 -- start start 2024-03-18T01:26:58.725 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.191+0000 7fe900400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f807b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:58.725 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.192+0000 7fe900400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f807b730 con 0x7fe8f807cd00 2024-03-18T01:26:58.726 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.192+0000 7fe8fda00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f807b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:58.726 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.192+0000 7fe8fda00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f807b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55922/0 (socket says 172.21.15.7:55922) 2024-03-18T01:26:58.726 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.192+0000 7fe8fda00700 1 -- 172.21.15.7:0/3554542245 learned_addr learned my addr 172.21.15.7:0/3554542245 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:58.726 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.192+0000 7fe8fda00700 1 -- 172.21.15.7:0/3554542245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f807b8b0 con 0x7fe8f807cd00 2024-03-18T01:26:58.726 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.193+0000 7fe8fda00700 1 --2- 172.21.15.7:0/3554542245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f807b160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009e80 tx=0x7fe8ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873399157c4296f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.193+0000 7fe8f7e00700 1 -- 172.21.15.7:0/3554542245 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8ec017070 con 0x7fe8f807cd00 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.193+0000 7fe8f7e00700 1 -- 172.21.15.7:0/3554542245 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8ec00b8c0 con 0x7fe8f807cd00 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.193+0000 7fe8f7e00700 1 -- 172.21.15.7:0/3554542245 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8ec0045c0 con 0x7fe8f807cd00 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.194+0000 7fe900400700 1 -- 172.21.15.7:0/3554542245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 msgr2=0x7fe8f807b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.194+0000 7fe900400700 1 --2- 172.21.15.7:0/3554542245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f807b160 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009e80 tx=0x7fe8ec00b330 comp rx=0 tx=0).stop 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.194+0000 7fe900400700 1 -- 172.21.15.7:0/3554542245 shutdown_connections 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.194+0000 7fe900400700 1 --2- 172.21.15.7:0/3554542245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f807b160 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.194+0000 7fe900400700 1 -- 172.21.15.7:0/3554542245 >> 172.21.15.7:0/3554542245 conn(0x7fe8f81031e0 msgr2=0x7fe8f8105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.194+0000 7fe900400700 1 -- 172.21.15.7:0/3554542245 shutdown_connections 2024-03-18T01:26:58.728 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.194+0000 7fe900400700 1 -- 172.21.15.7:0/3554542245 wait complete. 2024-03-18T01:26:58.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.195+0000 7fe900400700 1 Processor -- start 2024-03-18T01:26:58.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.195+0000 7fe900400700 1 -- start start 2024-03-18T01:26:58.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.195+0000 7fe900400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:58.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.195+0000 7fe900400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f810e2a0 con 0x7fe8f807cd00 2024-03-18T01:26:58.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.196+0000 7fe8fda00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:58.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.196+0000 7fe8fda00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f819dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55938/0 (socket says 172.21.15.7:55938) 2024-03-18T01:26:58.729 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.196+0000 7fe8fda00700 1 -- 172.21.15.7:0/476350252 learned_addr learned my addr 172.21.15.7:0/476350252 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.196+0000 7fe8fda00700 1 -- 172.21.15.7:0/476350252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f819e270 con 0x7fe8f807cd00 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.196+0000 7fe8fda00700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f819dd30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009530 tx=0x7fe8ec003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.197+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8ec020020 con 0x7fe8f807cd00 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.197+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8ec00bba0 con 0x7fe8f807cd00 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.197+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f819e500 con 0x7fe8f807cd00 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.197+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8ec017040 con 0x7fe8f807cd00 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.197+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f81a11f0 con 0x7fe8f807cd00 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.198+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fe8ec023020 con 0x7fe8f807cd00 2024-03-18T01:26:58.732 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.199+0000 7fe8f6000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 0x7fe8e003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.199+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 --> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe8e003bba0 con 0x7fe8e0038fd0 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.199+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe8ec00f050 con 0x7fe8f807cd00 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.199+0000 7fe8fd000700 1 -- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 msgr2=0x7fe8e003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1169030810 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.199+0000 7fe8fd000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 0x7fe8e003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.400+0000 7fe8fd000700 1 -- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 msgr2=0x7fe8e003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1169030810 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.400+0000 7fe8fd000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 0x7fe8e003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.801+0000 7fe8fd000700 1 -- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 msgr2=0x7fe8e003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1169030810 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:51.801+0000 7fe8fd000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 0x7fe8e003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:52.602+0000 7fe8fd000700 1 -- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 msgr2=0x7fe8e003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1169030810 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:52.602+0000 7fe8fd000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 0x7fe8e003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:54.204+0000 7fe8fd000700 1 -- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 msgr2=0x7fe8e003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.7:6800/1169030810 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:54.204+0000 7fe8fd000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 0x7fe8e003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T01:26:58.733 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:57.029+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7fe8ec012430 con 0x7fe8f807cd00 2024-03-18T01:26:58.734 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:57.029+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 msgr2=0x7fe8e003b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:26:58.734 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:57.029+0000 7fe8f6000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 0x7fe8e003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:58.736 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.030+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fe8ec04e770 con 0x7fe8f807cd00 2024-03-18T01:26:58.736 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.031+0000 7fe8f6000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe8e003ca80 0x7fe8e003ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:26:58.736 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.031+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe8ec023ec0 con 0x7fe8e003ca80 2024-03-18T01:26:58.736 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.032+0000 7fe8fd000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe8e003ca80 0x7fe8e003ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:26:58.736 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.034+0000 7fe8fd000700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe8e003ca80 0x7fe8e003ee70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4003a10 tx=0x7fe8e4009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:26:58.736 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.036+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fe8ec023ec0 con 0x7fe8e003ca80 2024-03-18T01:26:58.736 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.045+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe8f8063900 con 0x7fe8e003ca80 2024-03-18T01:26:58.736 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.045+0000 7fe8f6000700 1 -- 172.21.15.7:0/476350252 <== mgr.14152 v2:172.21.15.7:6800/2743163723 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fe8f8063900 con 0x7fe8e003ca80 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe8e003ca80 msgr2=0x7fe8e003ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe8e003ca80 0x7fe8e003ee70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4003a10 tx=0x7fe8e4009270 comp rx=0 tx=0).stop 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 msgr2=0x7fe8f819dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f819dd30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009530 tx=0x7fe8ec003d00 comp rx=0 tx=0).stop 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 shutdown_connections 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe8e003ca80 0x7fe8e003ee70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe8f807cd00 0x7fe8f819dd30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 --2- 172.21.15.7:0/476350252 >> [v2:172.21.15.7:6800/1169030810,v1:172.21.15.7:6801/1169030810] conn(0x7fe8e0038fd0 0x7fe8e003b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 >> 172.21.15.7:0/476350252 conn(0x7fe8f81031e0 msgr2=0x7fe8f8103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 shutdown_connections 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:58.046+0000 7fe900400700 1 -- 172.21.15.7:0/476350252 wait complete. 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:mgr epoch 9 is available 2024-03-18T01:26:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:Generating a dashboard self-signed certificate... 2024-03-18T01:26:59.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:59 smithi007 ceph-mon[31996]: [18/Mar/2024:01:26:58] ENGINE Bus STARTING 2024-03-18T01:26:59.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:59 smithi007 ceph-mon[31996]: [18/Mar/2024:01:26:58] ENGINE Serving on https://172.21.15.7:7150 2024-03-18T01:26:59.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:59 smithi007 ceph-mon[31996]: [18/Mar/2024:01:26:58] ENGINE Serving on http://172.21.15.7:8765 2024-03-18T01:26:59.280 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:26:59 smithi007 ceph-mon[31996]: [18/Mar/2024:01:26:58] ENGINE Bus STARTED 2024-03-18T01:27:00.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:00 smithi007 ceph-mon[31996]: mgrmap e12: x(active, since 2s) 2024-03-18T01:27:00.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:00 smithi007 ceph-mon[31996]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:27:00.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:00 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:00.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:00 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:00.241 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:00 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:00.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T01:27:00.526 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.501+0000 7f6801000700 1 Processor -- start 2024-03-18T01:27:00.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.501+0000 7f6801000700 1 -- start start 2024-03-18T01:27:00.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.501+0000 7f6801000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc07c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:00.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.501+0000 7f6801000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc07c820 con 0x7f67fc07ddf0 2024-03-18T01:27:00.527 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.502+0000 7f67f9e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc07c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:00.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.502+0000 7f67f9e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc07c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33148/0 (socket says 172.21.15.7:33148) 2024-03-18T01:27:00.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.502+0000 7f67f9e00700 1 -- 172.21.15.7:0/1385447579 learned_addr learned my addr 172.21.15.7:0/1385447579 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:00.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.502+0000 7f67f9e00700 1 -- 172.21.15.7:0/1385447579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67fc07c9a0 con 0x7f67fc07ddf0 2024-03-18T01:27:00.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.503+0000 7f67f9e00700 1 --2- 172.21.15.7:0/1385447579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc07c250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0098a0 tx=0x7f67ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40b7c81ec0c500b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:00.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.503+0000 7f67f8a00700 1 -- 172.21.15.7:0/1385447579 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67ec004160 con 0x7f67fc07ddf0 2024-03-18T01:27:00.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.503+0000 7f67f8a00700 1 -- 172.21.15.7:0/1385447579 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec004300 con 0x7f67fc07ddf0 2024-03-18T01:27:00.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.503+0000 7f67f8a00700 1 -- 172.21.15.7:0/1385447579 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67ec011b20 con 0x7f67fc07ddf0 2024-03-18T01:27:00.532 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.504+0000 7f6801000700 1 -- 172.21.15.7:0/1385447579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 msgr2=0x7f67fc07c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.504+0000 7f6801000700 1 --2- 172.21.15.7:0/1385447579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc07c250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0098a0 tx=0x7f67ec009bb0 comp rx=0 tx=0).stop 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.504+0000 7f6801000700 1 -- 172.21.15.7:0/1385447579 shutdown_connections 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.504+0000 7f6801000700 1 --2- 172.21.15.7:0/1385447579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc07c250 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.504+0000 7f6801000700 1 -- 172.21.15.7:0/1385447579 >> 172.21.15.7:0/1385447579 conn(0x7f67fc103170 msgr2=0x7f67fc1055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.504+0000 7f6801000700 1 -- 172.21.15.7:0/1385447579 shutdown_connections 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.504+0000 7f6801000700 1 -- 172.21.15.7:0/1385447579 wait complete. 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.505+0000 7f6801000700 1 Processor -- start 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.505+0000 7f6801000700 1 -- start start 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.505+0000 7f6801000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.505+0000 7f6801000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f67fc10e250 con 0x7f67fc07ddf0 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.506+0000 7f67f9e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.506+0000 7f67f9e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc19dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33158/0 (socket says 172.21.15.7:33158) 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.506+0000 7f67f9e00700 1 -- 172.21.15.7:0/3728352104 learned_addr learned my addr 172.21.15.7:0/3728352104 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:00.533 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.506+0000 7f67f9e00700 1 -- 172.21.15.7:0/3728352104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67fc19e240 con 0x7f67fc07ddf0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.506+0000 7f67f9e00700 1 --2- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc19dd00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0116a0 tx=0x7f67ec011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.507+0000 7f67f2a00700 1 -- 172.21.15.7:0/3728352104 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67ec003a40 con 0x7f67fc07ddf0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.507+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67fc19e4d0 con 0x7f67fc07ddf0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.507+0000 7f67f2a00700 1 -- 172.21.15.7:0/3728352104 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec003be0 con 0x7f67fc07ddf0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.507+0000 7f67f2a00700 1 -- 172.21.15.7:0/3728352104 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67ec01a750 con 0x7f67fc07ddf0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.507+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67fc19e930 con 0x7f67fc07ddf0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.508+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67fc063900 con 0x7f67fc07ddf0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.508+0000 7f67f2a00700 1 -- 172.21.15.7:0/3728352104 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f67ec024aa0 con 0x7f67fc07ddf0 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.509+0000 7f67f2a00700 1 --2- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67e0038f80 0x7f67e003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:00.534 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.509+0000 7f67f2a00700 1 -- 172.21.15.7:0/3728352104 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f67ec04d610 con 0x7f67fc07ddf0 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.509+0000 7f67f9400700 1 --2- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67e0038f80 0x7f67e003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.510+0000 7f67f9400700 1 --2- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67e0038f80 0x7f67e003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f67e8006fd0 tx=0x7f67e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.514+0000 7f67f2a00700 1 -- 172.21.15.7:0/3728352104 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f67ec011e20 con 0x7f67fc07ddf0 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.745+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f67fc104400 con 0x7f67e0038f80 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.889+0000 7f67f2a00700 1 -- 172.21.15.7:0/3728352104 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f67fc104400 con 0x7f67e0038f80 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.892+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67e0038f80 msgr2=0x7f67e003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.892+0000 7f6801000700 1 --2- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67e0038f80 0x7f67e003b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f67e8006fd0 tx=0x7f67e8006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.892+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 msgr2=0x7f67fc19dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.892+0000 7f6801000700 1 --2- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc19dd00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0116a0 tx=0x7f67ec011ba0 comp rx=0 tx=0).stop 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.892+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 shutdown_connections 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.892+0000 7f6801000700 1 --2- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67e0038f80 0x7f67e003b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.892+0000 7f6801000700 1 --2- 172.21.15.7:0/3728352104 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f67fc07ddf0 0x7f67fc19dd00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.892+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 >> 172.21.15.7:0/3728352104 conn(0x7f67fc103170 msgr2=0x7f67fc103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:00.535 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.893+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 shutdown_connections 2024-03-18T01:27:00.536 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:26:59.893+0000 7f6801000700 1 -- 172.21.15.7:0/3728352104 wait complete. 2024-03-18T01:27:00.536 INFO:teuthology.orchestra.run.smithi007.stdout:Creating initial admin user... 2024-03-18T01:27:02.450 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$AZ4SDrSgxW86OjFkXkwKTOeD6E/GiCHR6kswWoz1da.sLWit6nrvi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710725221, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T01:27:02.450 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.156+0000 7fd32f600700 1 Processor -- start 2024-03-18T01:27:02.450 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.156+0000 7fd32f600700 1 -- start start 2024-03-18T01:27:02.450 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.156+0000 7fd32f600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd328111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.156+0000 7fd32f600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd328082460 con 0x7fd32810ec60 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.157+0000 7fd32cc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd328111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.157+0000 7fd32cc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd328111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33170/0 (socket says 172.21.15.7:33170) 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.157+0000 7fd32cc00700 1 -- 172.21.15.7:0/4181934350 learned_addr learned my addr 172.21.15.7:0/4181934350 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.157+0000 7fd32cc00700 1 -- 172.21.15.7:0/4181934350 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd328111590 con 0x7fd32810ec60 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.158+0000 7fd32cc00700 1 --2- 172.21.15.7:0/4181934350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd328111050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0098a0 tx=0x7fd31c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e9e8d907c4b1db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.158+0000 7fd327400700 1 -- 172.21.15.7:0/4181934350 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd31c004160 con 0x7fd32810ec60 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.158+0000 7fd327400700 1 -- 172.21.15.7:0/4181934350 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd31c004300 con 0x7fd32810ec60 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.158+0000 7fd327400700 1 -- 172.21.15.7:0/4181934350 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd31c011b20 con 0x7fd32810ec60 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.159+0000 7fd32f600700 1 -- 172.21.15.7:0/4181934350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 msgr2=0x7fd328111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.159+0000 7fd32f600700 1 --2- 172.21.15.7:0/4181934350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd328111050 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0098a0 tx=0x7fd31c009bb0 comp rx=0 tx=0).stop 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.159+0000 7fd32f600700 1 -- 172.21.15.7:0/4181934350 shutdown_connections 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.159+0000 7fd32f600700 1 --2- 172.21.15.7:0/4181934350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd328111050 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.159+0000 7fd32f600700 1 -- 172.21.15.7:0/4181934350 >> 172.21.15.7:0/4181934350 conn(0x7fd32807d220 msgr2=0x7fd32807f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.159+0000 7fd32f600700 1 -- 172.21.15.7:0/4181934350 shutdown_connections 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.159+0000 7fd32f600700 1 -- 172.21.15.7:0/4181934350 wait complete. 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.160+0000 7fd32f600700 1 Processor -- start 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.160+0000 7fd32f600700 1 -- start start 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.160+0000 7fd32f600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd3281a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.160+0000 7fd32f600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd328112600 con 0x7fd32810ec60 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.161+0000 7fd32cc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd3281a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.161+0000 7fd32cc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd3281a2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33180/0 (socket says 172.21.15.7:33180) 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.161+0000 7fd32cc00700 1 -- 172.21.15.7:0/1805049554 learned_addr learned my addr 172.21.15.7:0/1805049554 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.161+0000 7fd32cc00700 1 -- 172.21.15.7:0/1805049554 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3281a2670 con 0x7fd32810ec60 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.161+0000 7fd32cc00700 1 --2- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd3281a2130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd31c011b60 tx=0x7fd31c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.162+0000 7fd325600700 1 -- 172.21.15.7:0/1805049554 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd31c003b60 con 0x7fd32810ec60 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.162+0000 7fd325600700 1 -- 172.21.15.7:0/1805049554 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd31c01a910 con 0x7fd32810ec60 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.162+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3281a2900 con 0x7fd32810ec60 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.162+0000 7fd325600700 1 -- 172.21.15.7:0/1805049554 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd31c024930 con 0x7fd32810ec60 2024-03-18T01:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.162+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3281a2d60 con 0x7fd32810ec60 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.163+0000 7fd325600700 1 -- 172.21.15.7:0/1805049554 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fd31c022020 con 0x7fd32810ec60 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.163+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd328063900 con 0x7fd32810ec60 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.164+0000 7fd325600700 1 --2- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd310038fd0 0x7fd31003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.164+0000 7fd325600700 1 -- 172.21.15.7:0/1805049554 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd31c04d1e0 con 0x7fd32810ec60 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.164+0000 7fd327e00700 1 --2- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd310038fd0 0x7fd31003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.165+0000 7fd327e00700 1 --2- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd310038fd0 0x7fd31003b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd314006fd0 tx=0x7fd314006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.170+0000 7fd325600700 1 -- 172.21.15.7:0/1805049554 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd31c0212c0 con 0x7fd32810ec60 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.426+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fd3281a5790 con 0x7fd310038fd0 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.691+0000 7fd325600700 1 -- 172.21.15.7:0/1805049554 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fd3281a5790 con 0x7fd310038fd0 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.694+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd310038fd0 msgr2=0x7fd31003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.694+0000 7fd32f600700 1 --2- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd310038fd0 0x7fd31003b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd314006fd0 tx=0x7fd314006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.694+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 msgr2=0x7fd3281a2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.694+0000 7fd32f600700 1 --2- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd3281a2130 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd31c011b60 tx=0x7fd31c011c40 comp rx=0 tx=0).stop 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.695+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 shutdown_connections 2024-03-18T01:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.695+0000 7fd32f600700 1 --2- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd310038fd0 0x7fd31003b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:02.454 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.695+0000 7fd32f600700 1 --2- 172.21.15.7:0/1805049554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd32810ec60 0x7fd3281a2130 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:02.454 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.695+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 >> 172.21.15.7:0/1805049554 conn(0x7fd32807d220 msgr2=0x7fd32807dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:02.454 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.695+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 shutdown_connections 2024-03-18T01:27:02.454 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:01.695+0000 7fd32f600700 1 -- 172.21.15.7:0/1805049554 wait complete. 2024-03-18T01:27:02.454 INFO:teuthology.orchestra.run.smithi007.stdout:Fetching dashboard port number... 2024-03-18T01:27:02.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:02 smithi007 ceph-mon[31996]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:27:02.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:02.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:03.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:03 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1943615256' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T01:27:03.992 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T01:27:03.992 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.026+0000 7f5855a00700 1 Processor -- start 2024-03-18T01:27:03.993 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.026+0000 7f5855a00700 1 -- start start 2024-03-18T01:27:03.993 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.027+0000 7f5855a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f585010aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:03.993 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.027+0000 7f5855a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5850075990 con 0x7f58501086e0 2024-03-18T01:27:03.993 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.027+0000 7f584ea00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f585010aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:03.993 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.027+0000 7f584ea00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f585010aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33196/0 (socket says 172.21.15.7:33196) 2024-03-18T01:27:03.993 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.027+0000 7f584ea00700 1 -- 172.21.15.7:0/2283917182 learned_addr learned my addr 172.21.15.7:0/2283917182 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:03.993 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.027+0000 7f584ea00700 1 -- 172.21.15.7:0/2283917182 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f585010b010 con 0x7f58501086e0 2024-03-18T01:27:03.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.028+0000 7f584ea00700 1 --2- 172.21.15.7:0/2283917182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f585010aad0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58400098a0 tx=0x7f5840009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b2e0fff0d9656d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:03.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.028+0000 7f584d600700 1 -- 172.21.15.7:0/2283917182 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5840004160 con 0x7f58501086e0 2024-03-18T01:27:03.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.028+0000 7f584d600700 1 -- 172.21.15.7:0/2283917182 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5840004300 con 0x7f58501086e0 2024-03-18T01:27:03.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.028+0000 7f584d600700 1 -- 172.21.15.7:0/2283917182 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5840011b20 con 0x7f58501086e0 2024-03-18T01:27:03.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.029+0000 7f5855a00700 1 -- 172.21.15.7:0/2283917182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 msgr2=0x7f585010aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:03.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.029+0000 7f5855a00700 1 --2- 172.21.15.7:0/2283917182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f585010aad0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58400098a0 tx=0x7f5840009bb0 comp rx=0 tx=0).stop 2024-03-18T01:27:03.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.029+0000 7f5855a00700 1 -- 172.21.15.7:0/2283917182 shutdown_connections 2024-03-18T01:27:03.998 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.029+0000 7f5855a00700 1 --2- 172.21.15.7:0/2283917182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f585010aad0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.029+0000 7f5855a00700 1 -- 172.21.15.7:0/2283917182 >> 172.21.15.7:0/2283917182 conn(0x7f58501020d0 msgr2=0x7f58501044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.029+0000 7f5855a00700 1 -- 172.21.15.7:0/2283917182 shutdown_connections 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.029+0000 7f5855a00700 1 -- 172.21.15.7:0/2283917182 wait complete. 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.030+0000 7f5855a00700 1 Processor -- start 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.030+0000 7f5855a00700 1 -- start start 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.031+0000 7f5855a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f5850194b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.031+0000 7f5855a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f585010c020 con 0x7f58501086e0 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.031+0000 7f584ea00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f5850194b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.031+0000 7f584ea00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f5850194b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33198/0 (socket says 172.21.15.7:33198) 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.031+0000 7f584ea00700 1 -- 172.21.15.7:0/1943615256 learned_addr learned my addr 172.21.15.7:0/1943615256 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.031+0000 7f584ea00700 1 -- 172.21.15.7:0/1943615256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5850195090 con 0x7f58501086e0 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.032+0000 7f584ea00700 1 --2- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f5850194b50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58400117e0 tx=0x7f5840011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:03.999 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.032+0000 7f5847400700 1 -- 172.21.15.7:0/1943615256 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5840003ba0 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.032+0000 7f5847400700 1 -- 172.21.15.7:0/1943615256 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f584001a910 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.032+0000 7f5847400700 1 -- 172.21.15.7:0/1943615256 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58400248f0 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.032+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5850195320 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.032+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5850191860 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.033+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58501954f0 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.034+0000 7f5847400700 1 -- 172.21.15.7:0/1943615256 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f5840022020 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.034+0000 7f5847400700 1 --2- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5834038fd0 0x7f583403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.034+0000 7f5847400700 1 -- 172.21.15.7:0/1943615256 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f584004d440 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.034+0000 7f584e000700 1 --2- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5834038fd0 0x7f583403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.035+0000 7f584e000700 1 --2- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5834038fd0 0x7f583403b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f583c006fd0 tx=0x7f583c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.040+0000 7f5847400700 1 -- 172.21.15.7:0/1943615256 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f58501954f0 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.280+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f5850191eb0 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.281+0000 7f5847400700 1 -- 172.21.15.7:0/1943615256 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f584001ab70 con 0x7f58501086e0 2024-03-18T01:27:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.283+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5834038fd0 msgr2=0x7f583403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.283+0000 7f5855a00700 1 --2- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5834038fd0 0x7f583403b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f583c006fd0 tx=0x7f583c006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.283+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 msgr2=0x7f5850194b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.283+0000 7f5855a00700 1 --2- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f5850194b50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58400117e0 tx=0x7f5840011ba0 comp rx=0 tx=0).stop 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.283+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 shutdown_connections 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.283+0000 7f5855a00700 1 --2- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5834038fd0 0x7f583403b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.283+0000 7f5855a00700 1 --2- 172.21.15.7:0/1943615256 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f58501086e0 0x7f5850194b50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.283+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 >> 172.21.15.7:0/1943615256 conn(0x7f58501020d0 msgr2=0x7f5850102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.284+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 shutdown_connections 2024-03-18T01:27:04.001 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:03.284+0000 7f5855a00700 1 -- 172.21.15.7:0/1943615256 wait complete. 2024-03-18T01:27:04.030 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T01:27:04.030 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout disabled 2024-03-18T01:27:04.052 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T01:27:04.052 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stdout inactive 2024-03-18T01:27:04.052 INFO:teuthology.orchestra.run.smithi007.stdout:firewalld.service is not enabled 2024-03-18T01:27:04.052 INFO:teuthology.orchestra.run.smithi007.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T01:27:04.056 INFO:teuthology.orchestra.run.smithi007.stdout:Ceph Dashboard is now available at: 2024-03-18T01:27:04.056 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:04.056 INFO:teuthology.orchestra.run.smithi007.stdout: URL: https://smithi007.front.sepia.ceph.com:8443/ 2024-03-18T01:27:04.056 INFO:teuthology.orchestra.run.smithi007.stdout: User: admin 2024-03-18T01:27:04.056 INFO:teuthology.orchestra.run.smithi007.stdout: Password: 9lrzz8drd4 2024-03-18T01:27:04.057 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:04.057 INFO:teuthology.orchestra.run.smithi007.stdout:Saving cluster configuration to /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config directory 2024-03-18T01:27:04.058 INFO:teuthology.orchestra.run.smithi007.stdout:Enabling autotune for osd_memory_target 2024-03-18T01:27:04.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:04 smithi007 ceph-mon[31996]: mgrmap e13: x(active, since 6s) 2024-03-18T01:27:06.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.569+0000 7f376a600700 1 Processor -- start 2024-03-18T01:27:06.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.569+0000 7f376a600700 1 -- start start 2024-03-18T01:27:06.404 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.570+0000 7f376a600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f3764110c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.570+0000 7f376a600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3764111240 con 0x7f3764082240 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.570+0000 7f3769200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f3764110c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.570+0000 7f3769200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f3764110c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33204/0 (socket says 172.21.15.7:33204) 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.570+0000 7f3769200700 1 -- 172.21.15.7:0/3187911875 learned_addr learned my addr 172.21.15.7:0/3187911875 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.571+0000 7f3769200700 1 -- 172.21.15.7:0/3187911875 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37641113c0 con 0x7f3764082240 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.571+0000 7f3769200700 1 --2- 172.21.15.7:0/3187911875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f3764110c70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f37580098a0 tx=0x7f3758009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950da69b1235de3d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.571+0000 7f3763400700 1 -- 172.21.15.7:0/3187911875 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3758004160 con 0x7f3764082240 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.571+0000 7f3763400700 1 -- 172.21.15.7:0/3187911875 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3758004300 con 0x7f3764082240 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.571+0000 7f3763400700 1 -- 172.21.15.7:0/3187911875 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3758011b20 con 0x7f3764082240 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.572+0000 7f376a600700 1 -- 172.21.15.7:0/3187911875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 msgr2=0x7f3764110c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.572+0000 7f376a600700 1 --2- 172.21.15.7:0/3187911875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f3764110c70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f37580098a0 tx=0x7f3758009bb0 comp rx=0 tx=0).stop 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.572+0000 7f376a600700 1 -- 172.21.15.7:0/3187911875 shutdown_connections 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.572+0000 7f376a600700 1 --2- 172.21.15.7:0/3187911875 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f3764110c70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:06.405 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.572+0000 7f376a600700 1 -- 172.21.15.7:0/3187911875 >> 172.21.15.7:0/3187911875 conn(0x7f376407d090 msgr2=0x7f376407f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:06.406 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.573+0000 7f376a600700 1 -- 172.21.15.7:0/3187911875 shutdown_connections 2024-03-18T01:27:06.406 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.573+0000 7f376a600700 1 -- 172.21.15.7:0/3187911875 wait complete. 2024-03-18T01:27:06.408 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.574+0000 7f376a600700 1 Processor -- start 2024-03-18T01:27:06.408 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.574+0000 7f376a600700 1 -- start start 2024-03-18T01:27:06.408 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.574+0000 7f376a600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f37641a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:06.408 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.574+0000 7f376a600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3764112490 con 0x7f3764082240 2024-03-18T01:27:06.408 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.574+0000 7f3769200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f37641a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.574+0000 7f3769200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f37641a1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:33212/0 (socket says 172.21.15.7:33212) 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.574+0000 7f3769200700 1 -- 172.21.15.7:0/1991870216 learned_addr learned my addr 172.21.15.7:0/1991870216 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.575+0000 7f3769200700 1 -- 172.21.15.7:0/1991870216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37641a2520 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.575+0000 7f3769200700 1 --2- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f37641a1fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3758000c00 tx=0x7f3758011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.575+0000 7f3761600700 1 -- 172.21.15.7:0/1991870216 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3758003b60 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.575+0000 7f3761600700 1 -- 172.21.15.7:0/1991870216 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f375801a910 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.575+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37641a27b0 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.575+0000 7f3761600700 1 -- 172.21.15.7:0/1991870216 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3758024930 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.576+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37641a2c10 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.577+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3764063900 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.577+0000 7f3761600700 1 -- 172.21.15.7:0/1991870216 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f3758022020 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.578+0000 7f3761600700 1 --2- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f374c039010 0x7f374c03b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.578+0000 7f3761600700 1 -- 172.21.15.7:0/1991870216 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f375804d610 con 0x7f3764082240 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.578+0000 7f3763e00700 1 --2- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f374c039010 0x7f374c03b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:06.409 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.579+0000 7f3763e00700 1 --2- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f374c039010 0x7f374c03b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3750006fd0 tx=0x7f3750006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:06.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.583+0000 7f3761600700 1 -- 172.21.15.7:0/1991870216 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f375802c3a0 con 0x7f3764082240 2024-03-18T01:27:06.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.806+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f37641a55e0 con 0x7f3764082240 2024-03-18T01:27:06.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.807+0000 7f3761600700 1 -- 172.21.15.7:0/1991870216 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f3758032030 con 0x7f3764082240 2024-03-18T01:27:06.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.809+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f374c039010 msgr2=0x7f374c03b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:06.410 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.809+0000 7f376a600700 1 --2- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f374c039010 0x7f374c03b4d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3750006fd0 tx=0x7f3750006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:06.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.809+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 msgr2=0x7f37641a1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:06.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.809+0000 7f376a600700 1 --2- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f37641a1fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3758000c00 tx=0x7f3758011c10 comp rx=0 tx=0).stop 2024-03-18T01:27:06.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.810+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 shutdown_connections 2024-03-18T01:27:06.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.810+0000 7f376a600700 1 --2- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f374c039010 0x7f374c03b4d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:06.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.810+0000 7f376a600700 1 --2- 172.21.15.7:0/1991870216 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3764082240 0x7f37641a1fe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:06.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.810+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 >> 172.21.15.7:0/1991870216 conn(0x7f376407d090 msgr2=0x7f376407dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:06.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.810+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 shutdown_connections 2024-03-18T01:27:06.411 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:05.810+0000 7f376a600700 1 -- 172.21.15.7:0/1991870216 wait complete. 2024-03-18T01:27:08.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.047+0000 7fd264400700 1 Processor -- start 2024-03-18T01:27:08.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.048+0000 7fd264400700 1 -- start start 2024-03-18T01:27:08.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.048+0000 7fd264400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c10aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:08.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.048+0000 7fd264400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c075990 con 0x7fd25c1086e0 2024-03-18T01:27:08.065 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.049+0000 7fd261a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c10aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:08.067 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.049+0000 7fd261a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c10aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60254/0 (socket says 172.21.15.7:60254) 2024-03-18T01:27:08.067 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.049+0000 7fd261a00700 1 -- 172.21.15.7:0/3385238635 learned_addr learned my addr 172.21.15.7:0/3385238635 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:08.067 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.049+0000 7fd261a00700 1 -- 172.21.15.7:0/3385238635 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c10b010 con 0x7fd25c1086e0 2024-03-18T01:27:08.067 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.049+0000 7fd261a00700 1 --2- 172.21.15.7:0/3385238635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c10aad0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd2500098a0 tx=0x7fd250009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2512ee33675d9d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:08.067 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.050+0000 7fd25be00700 1 -- 172.21.15.7:0/3385238635 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd250004160 con 0x7fd25c1086e0 2024-03-18T01:27:08.067 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.050+0000 7fd25be00700 1 -- 172.21.15.7:0/3385238635 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd250004300 con 0x7fd25c1086e0 2024-03-18T01:27:08.068 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.050+0000 7fd25be00700 1 -- 172.21.15.7:0/3385238635 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd250011b20 con 0x7fd25c1086e0 2024-03-18T01:27:08.068 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.050+0000 7fd264400700 1 -- 172.21.15.7:0/3385238635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 msgr2=0x7fd25c10aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:08.068 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.050+0000 7fd264400700 1 --2- 172.21.15.7:0/3385238635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c10aad0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd2500098a0 tx=0x7fd250009bb0 comp rx=0 tx=0).stop 2024-03-18T01:27:08.068 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.051+0000 7fd264400700 1 -- 172.21.15.7:0/3385238635 shutdown_connections 2024-03-18T01:27:08.068 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.051+0000 7fd264400700 1 --2- 172.21.15.7:0/3385238635 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c10aad0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:08.068 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.051+0000 7fd264400700 1 -- 172.21.15.7:0/3385238635 >> 172.21.15.7:0/3385238635 conn(0x7fd25c1020d0 msgr2=0x7fd25c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:08.068 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.051+0000 7fd264400700 1 -- 172.21.15.7:0/3385238635 shutdown_connections 2024-03-18T01:27:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.051+0000 7fd264400700 1 -- 172.21.15.7:0/3385238635 wait complete. 2024-03-18T01:27:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.052+0000 7fd264400700 1 Processor -- start 2024-03-18T01:27:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.052+0000 7fd264400700 1 -- start start 2024-03-18T01:27:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.052+0000 7fd264400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c19dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.052+0000 7fd264400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c10c020 con 0x7fd25c1086e0 2024-03-18T01:27:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.053+0000 7fd261a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c19dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.053+0000 7fd261a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c19dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60258/0 (socket says 172.21.15.7:60258) 2024-03-18T01:27:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.053+0000 7fd261a00700 1 -- 172.21.15.7:0/4188014593 learned_addr learned my addr 172.21.15.7:0/4188014593 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.053+0000 7fd261a00700 1 -- 172.21.15.7:0/4188014593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c19e1e0 con 0x7fd25c1086e0 2024-03-18T01:27:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.053+0000 7fd261a00700 1 --2- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c19dca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd2500117e0 tx=0x7fd250011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.054+0000 7fd25a000700 1 -- 172.21.15.7:0/4188014593 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd250003bf0 con 0x7fd25c1086e0 2024-03-18T01:27:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.054+0000 7fd25a000700 1 -- 172.21.15.7:0/4188014593 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25001a910 con 0x7fd25c1086e0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.054+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25c19e470 con 0x7fd25c1086e0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.054+0000 7fd25a000700 1 -- 172.21.15.7:0/4188014593 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd250024970 con 0x7fd25c1086e0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.054+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25c19e8d0 con 0x7fd25c1086e0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.055+0000 7fd25a000700 1 -- 172.21.15.7:0/4188014593 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7fd250022020 con 0x7fd25c1086e0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.055+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd25c063900 con 0x7fd25c1086e0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.056+0000 7fd25a000700 1 --2- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd244039020 0x7fd24403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.056+0000 7fd25a000700 1 -- 172.21.15.7:0/4188014593 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd250052660 con 0x7fd25c1086e0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.056+0000 7fd261000700 1 --2- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd244039020 0x7fd24403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.057+0000 7fd261000700 1 --2- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd244039020 0x7fd24403b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd248006fd0 tx=0x7fd248006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.061+0000 7fd25a000700 1 -- 172.21.15.7:0/4188014593 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd250011e20 con 0x7fd25c1086e0 2024-03-18T01:27:08.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.415+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fd25c1a1340 con 0x7fd25c1086e0 2024-03-18T01:27:08.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.420+0000 7fd25a000700 1 -- 172.21.15.7:0/4188014593 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fd250018350 con 0x7fd25c1086e0 2024-03-18T01:27:08.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd244039020 msgr2=0x7fd24403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:08.074 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 --2- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd244039020 0x7fd24403b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd248006fd0 tx=0x7fd248006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 msgr2=0x7fd25c19dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 --2- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c19dca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd2500117e0 tx=0x7fd250011c10 comp rx=0 tx=0).stop 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 shutdown_connections 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 --2- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd244039020 0x7fd24403b4e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 --2- 172.21.15.7:0/4188014593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd25c1086e0 0x7fd25c19dca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 >> 172.21.15.7:0/4188014593 conn(0x7fd25c1020d0 msgr2=0x7fd25c102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.423+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 shutdown_connections 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr 2024-03-18T01:27:07.424+0000 7fd264400700 1 -- 172.21.15.7:0/4188014593 wait complete. 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:08.075 INFO:teuthology.orchestra.run.smithi007.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: ceph telemetry on 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout:For more information see: 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:08.076 INFO:teuthology.orchestra.run.smithi007.stdout:Bootstrap complete. 2024-03-18T01:27:08.202 INFO:tasks.cephadm:Fetching config... 2024-03-18T01:27:08.203 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:27:08.203 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T01:27:08.255 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T01:27:08.255 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:27:08.255 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T01:27:08.295 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T01:27:08.295 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:27:08.295 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-18T01:27:08.433 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T01:27:08.433 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:27:08.433 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T01:27:08.480 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T01:27:08.480 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDghiXkx3ALR1rBrj2i/XmE9cMTYhhZvVm7c89NK53W5wN0xrefSn1iGPho3Lv0WzvlgsyguyBSQ95mwd2BsL5qI4B5nugdKK+SLNKJli9WQk6pOvTlWQWt5iEd06Oqt1LfnwjMRBfj2RdZiOJaruXOoTse4hBys5dc31hfFbFJ3haGBNiWzxZBJMAHS9rLLYHQrLhDAAR2Oxcw3g3Jzx0hYkl4Iey6TFGbzGfg5/F8q1/0sGX2uHN7YK4DboNjrUJMPHRTFm/BQ7h52bw0X2zl9Q9X/0bY0XW6am5ZhkG8BE1PPupjY91W2iKXWXt3MapOShvhOXFQGSn9w5/F3d0aGSYSFOH67wJjp72TD8jddsppU7rgqNUcSddwSIxHcCofQwPFOLXhzC29NZ7lxPAiF6Otkz5hpkmarNKtHMxQ+obqhRhGcBD19TXjKarQIVvE/ca9nA/UYXoMuSDks6rPPA25vzVb83fmyodMyxMnqfRTlxqemeBUz1vB6wjK000= ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T01:27:08.647 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:08 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4188014593' entity='client.admin' 2024-03-18T01:27:08.654 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDghiXkx3ALR1rBrj2i/XmE9cMTYhhZvVm7c89NK53W5wN0xrefSn1iGPho3Lv0WzvlgsyguyBSQ95mwd2BsL5qI4B5nugdKK+SLNKJli9WQk6pOvTlWQWt5iEd06Oqt1LfnwjMRBfj2RdZiOJaruXOoTse4hBys5dc31hfFbFJ3haGBNiWzxZBJMAHS9rLLYHQrLhDAAR2Oxcw3g3Jzx0hYkl4Iey6TFGbzGfg5/F8q1/0sGX2uHN7YK4DboNjrUJMPHRTFm/BQ7h52bw0X2zl9Q9X/0bY0XW6am5ZhkG8BE1PPupjY91W2iKXWXt3MapOShvhOXFQGSn9w5/F3d0aGSYSFOH67wJjp72TD8jddsppU7rgqNUcSddwSIxHcCofQwPFOLXhzC29NZ7lxPAiF6Otkz5hpkmarNKtHMxQ+obqhRhGcBD19TXjKarQIVvE/ca9nA/UYXoMuSDks6rPPA25vzVb83fmyodMyxMnqfRTlxqemeBUz1vB6wjK000= ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:27:08.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDghiXkx3ALR1rBrj2i/XmE9cMTYhhZvVm7c89NK53W5wN0xrefSn1iGPho3Lv0WzvlgsyguyBSQ95mwd2BsL5qI4B5nugdKK+SLNKJli9WQk6pOvTlWQWt5iEd06Oqt1LfnwjMRBfj2RdZiOJaruXOoTse4hBys5dc31hfFbFJ3haGBNiWzxZBJMAHS9rLLYHQrLhDAAR2Oxcw3g3Jzx0hYkl4Iey6TFGbzGfg5/F8q1/0sGX2uHN7YK4DboNjrUJMPHRTFm/BQ7h52bw0X2zl9Q9X/0bY0XW6am5ZhkG8BE1PPupjY91W2iKXWXt3MapOShvhOXFQGSn9w5/F3d0aGSYSFOH67wJjp72TD8jddsppU7rgqNUcSddwSIxHcCofQwPFOLXhzC29NZ7lxPAiF6Otkz5hpkmarNKtHMxQ+obqhRhGcBD19TXjKarQIVvE/ca9nA/UYXoMuSDks6rPPA25vzVb83fmyodMyxMnqfRTlxqemeBUz1vB6wjK000= ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T01:27:08.780 INFO:teuthology.orchestra.run.smithi046.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDghiXkx3ALR1rBrj2i/XmE9cMTYhhZvVm7c89NK53W5wN0xrefSn1iGPho3Lv0WzvlgsyguyBSQ95mwd2BsL5qI4B5nugdKK+SLNKJli9WQk6pOvTlWQWt5iEd06Oqt1LfnwjMRBfj2RdZiOJaruXOoTse4hBys5dc31hfFbFJ3haGBNiWzxZBJMAHS9rLLYHQrLhDAAR2Oxcw3g3Jzx0hYkl4Iey6TFGbzGfg5/F8q1/0sGX2uHN7YK4DboNjrUJMPHRTFm/BQ7h52bw0X2zl9Q9X/0bY0XW6am5ZhkG8BE1PPupjY91W2iKXWXt3MapOShvhOXFQGSn9w5/F3d0aGSYSFOH67wJjp72TD8jddsppU7rgqNUcSddwSIxHcCofQwPFOLXhzC29NZ7lxPAiF6Otkz5hpkmarNKtHMxQ+obqhRhGcBD19TXjKarQIVvE/ca9nA/UYXoMuSDks6rPPA25vzVb83fmyodMyxMnqfRTlxqemeBUz1vB6wjK000= ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:27:08.799 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDghiXkx3ALR1rBrj2i/XmE9cMTYhhZvVm7c89NK53W5wN0xrefSn1iGPho3Lv0WzvlgsyguyBSQ95mwd2BsL5qI4B5nugdKK+SLNKJli9WQk6pOvTlWQWt5iEd06Oqt1LfnwjMRBfj2RdZiOJaruXOoTse4hBys5dc31hfFbFJ3haGBNiWzxZBJMAHS9rLLYHQrLhDAAR2Oxcw3g3Jzx0hYkl4Iey6TFGbzGfg5/F8q1/0sGX2uHN7YK4DboNjrUJMPHRTFm/BQ7h52bw0X2zl9Q9X/0bY0XW6am5ZhkG8BE1PPupjY91W2iKXWXt3MapOShvhOXFQGSn9w5/F3d0aGSYSFOH67wJjp72TD8jddsppU7rgqNUcSddwSIxHcCofQwPFOLXhzC29NZ7lxPAiF6Otkz5hpkmarNKtHMxQ+obqhRhGcBD19TXjKarQIVvE/ca9nA/UYXoMuSDks6rPPA25vzVb83fmyodMyxMnqfRTlxqemeBUz1vB6wjK000= ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T01:27:08.845 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDghiXkx3ALR1rBrj2i/XmE9cMTYhhZvVm7c89NK53W5wN0xrefSn1iGPho3Lv0WzvlgsyguyBSQ95mwd2BsL5qI4B5nugdKK+SLNKJli9WQk6pOvTlWQWt5iEd06Oqt1LfnwjMRBfj2RdZiOJaruXOoTse4hBys5dc31hfFbFJ3haGBNiWzxZBJMAHS9rLLYHQrLhDAAR2Oxcw3g3Jzx0hYkl4Iey6TFGbzGfg5/F8q1/0sGX2uHN7YK4DboNjrUJMPHRTFm/BQ7h52bw0X2zl9Q9X/0bY0XW6am5ZhkG8BE1PPupjY91W2iKXWXt3MapOShvhOXFQGSn9w5/F3d0aGSYSFOH67wJjp72TD8jddsppU7rgqNUcSddwSIxHcCofQwPFOLXhzC29NZ7lxPAiF6Otkz5hpkmarNKtHMxQ+obqhRhGcBD19TXjKarQIVvE/ca9nA/UYXoMuSDks6rPPA25vzVb83fmyodMyxMnqfRTlxqemeBUz1vB6wjK000= ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:27:08.860 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T01:27:09.305 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:27:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.091+0000 7f7985000700 1 -- 172.21.15.7:0/969961654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 msgr2=0x7f7980104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.091+0000 7f7985000700 1 --2- 172.21.15.7:0/969961654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 0x7f7980104c40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7970009910 tx=0x7f7970009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.091+0000 7f7985000700 1 -- 172.21.15.7:0/969961654 shutdown_connections 2024-03-18T01:27:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.091+0000 7f7985000700 1 --2- 172.21.15.7:0/969961654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 0x7f7980104c40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.091+0000 7f7985000700 1 -- 172.21.15.7:0/969961654 >> 172.21.15.7:0/969961654 conn(0x7f79800fc1a0 msgr2=0x7f79800fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.091+0000 7f7985000700 1 -- 172.21.15.7:0/969961654 shutdown_connections 2024-03-18T01:27:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.091+0000 7f7985000700 1 -- 172.21.15.7:0/969961654 wait complete. 2024-03-18T01:27:12.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.092+0000 7f7985000700 1 Processor -- start 2024-03-18T01:27:12.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.092+0000 7f7985000700 1 -- start start 2024-03-18T01:27:12.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.093+0000 7f7985000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 0x7f79801020d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:12.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.093+0000 7f7985000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7980107d90 con 0x7f7980102850 2024-03-18T01:27:12.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.093+0000 7f797de00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 0x7f79801020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:12.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.093+0000 7f797de00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 0x7f79801020d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60274/0 (socket says 172.21.15.7:60274) 2024-03-18T01:27:12.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.093+0000 7f797de00700 1 -- 172.21.15.7:0/3288751206 learned_addr learned my addr 172.21.15.7:0/3288751206 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:12.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.093+0000 7f797de00700 1 -- 172.21.15.7:0/3288751206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7980102610 con 0x7f7980102850 2024-03-18T01:27:12.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.094+0000 7f797de00700 1 --2- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 0x7f79801020d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7970005950 tx=0x7f7970005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:12.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.094+0000 7f7976a00700 1 -- 172.21.15.7:0/3288751206 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7970026070 con 0x7f7980102850 2024-03-18T01:27:12.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.094+0000 7f7976a00700 1 -- 172.21.15.7:0/3288751206 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f797000bc30 con 0x7f7980102850 2024-03-18T01:27:12.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.094+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7980100800 con 0x7f7980102850 2024-03-18T01:27:12.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.094+0000 7f7976a00700 1 -- 172.21.15.7:0/3288751206 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7970003b30 con 0x7f7980102850 2024-03-18T01:27:12.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.094+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7980100d40 con 0x7f7980102850 2024-03-18T01:27:12.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.095+0000 7f7976a00700 1 -- 172.21.15.7:0/3288751206 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f7970029070 con 0x7f7980102850 2024-03-18T01:27:12.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.096+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7980108510 con 0x7f7980102850 2024-03-18T01:27:12.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.096+0000 7f7976a00700 1 --2- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7964038fc0 0x7f796403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:12.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.096+0000 7f7976a00700 1 -- 172.21.15.7:0/3288751206 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f797001e070 con 0x7f7980102850 2024-03-18T01:27:12.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.096+0000 7f797d400700 1 --2- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7964038fc0 0x7f796403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:12.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.097+0000 7f797d400700 1 --2- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7964038fc0 0x7f796403b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f796c006fd0 tx=0x7f796c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:12.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.102+0000 7f7976a00700 1 -- 172.21.15.7:0/3288751206 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7970013070 con 0x7f7980102850 2024-03-18T01:27:12.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.333+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f7980190810 con 0x7f7980102850 2024-03-18T01:27:12.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.337+0000 7f7976a00700 1 -- 172.21.15.7:0/3288751206 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f7970030020 con 0x7f7980102850 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.340+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7964038fc0 msgr2=0x7f796403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.340+0000 7f7985000700 1 --2- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7964038fc0 0x7f796403b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f796c006fd0 tx=0x7f796c006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.340+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 msgr2=0x7f79801020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.340+0000 7f7985000700 1 --2- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 0x7f79801020d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7970005950 tx=0x7f7970005030 comp rx=0 tx=0).stop 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.340+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 shutdown_connections 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.340+0000 7f7985000700 1 --2- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7964038fc0 0x7f796403b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.340+0000 7f7985000700 1 --2- 172.21.15.7:0/3288751206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7980102850 0x7f79801020d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.340+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 >> 172.21.15.7:0/3288751206 conn(0x7f79800fc1a0 msgr2=0x7f79800fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.341+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 shutdown_connections 2024-03-18T01:27:12.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:12.341+0000 7f7985000700 1 -- 172.21.15.7:0/3288751206 wait complete. 2024-03-18T01:27:13.076 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T01:27:13.076 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T01:27:13.489 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:27:13.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:13 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3288751206' entity='client.admin' 2024-03-18T01:27:15.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.958+0000 7f5d6fe00700 1 -- 172.21.15.7:0/232057765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 msgr2=0x7f5d70108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:15.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.958+0000 7f5d6fe00700 1 --2- 172.21.15.7:0/232057765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 0x7f5d70108900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009910 tx=0x7f5d60009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:15.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.958+0000 7f5d6fe00700 1 -- 172.21.15.7:0/232057765 shutdown_connections 2024-03-18T01:27:15.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.958+0000 7f5d6fe00700 1 --2- 172.21.15.7:0/232057765 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 0x7f5d70108900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:15.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.958+0000 7f5d6fe00700 1 -- 172.21.15.7:0/232057765 >> 172.21.15.7:0/232057765 conn(0x7f5d700772c0 msgr2=0x7f5d70079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:15.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.959+0000 7f5d6fe00700 1 -- 172.21.15.7:0/232057765 shutdown_connections 2024-03-18T01:27:15.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.959+0000 7f5d6fe00700 1 -- 172.21.15.7:0/232057765 wait complete. 2024-03-18T01:27:15.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.960+0000 7f5d6fe00700 1 Processor -- start 2024-03-18T01:27:15.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.960+0000 7f5d6fe00700 1 -- start start 2024-03-18T01:27:15.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.960+0000 7f5d6fe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 0x7f5d7019d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:15.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.960+0000 7f5d6fe00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7010e210 con 0x7f5d701084e0 2024-03-18T01:27:15.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.961+0000 7f5d6ea00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 0x7f5d7019d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:15.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.961+0000 7f5d6ea00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 0x7f5d7019d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58976/0 (socket says 172.21.15.7:58976) 2024-03-18T01:27:15.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.961+0000 7f5d6ea00700 1 -- 172.21.15.7:0/1149073372 learned_addr learned my addr 172.21.15.7:0/1149073372 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:15.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.961+0000 7f5d6ea00700 1 -- 172.21.15.7:0/1149073372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7019dd60 con 0x7f5d701084e0 2024-03-18T01:27:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.961+0000 7f5d6ea00700 1 --2- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 0x7f5d7019d820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005850 tx=0x7f5d60005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.962+0000 7f5d67400700 1 -- 172.21.15.7:0/1149073372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d60026070 con 0x7f5d701084e0 2024-03-18T01:27:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.962+0000 7f5d67400700 1 -- 172.21.15.7:0/1149073372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d600044a0 con 0x7f5d701084e0 2024-03-18T01:27:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.962+0000 7f5d67400700 1 -- 172.21.15.7:0/1149073372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d60003ab0 con 0x7f5d701084e0 2024-03-18T01:27:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.962+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7019dff0 con 0x7f5d701084e0 2024-03-18T01:27:15.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.962+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d701a0ce0 con 0x7f5d701084e0 2024-03-18T01:27:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.963+0000 7f5d67400700 1 -- 172.21.15.7:0/1149073372 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f5d60029070 con 0x7f5d701084e0 2024-03-18T01:27:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.963+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7010e990 con 0x7f5d701084e0 2024-03-18T01:27:15.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.963+0000 7f5d67400700 1 --2- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5d54039020 0x7f5d5403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.963+0000 7f5d67400700 1 -- 172.21.15.7:0/1149073372 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5d6001e070 con 0x7f5d701084e0 2024-03-18T01:27:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.964+0000 7f5d6e000700 1 --2- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5d54039020 0x7f5d5403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:15.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.964+0000 7f5d6e000700 1 --2- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5d54039020 0x7f5d5403b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c006fd0 tx=0x7f5d5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:15.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:15.970+0000 7f5d67400700 1 -- 172.21.15.7:0/1149073372 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d60013070 con 0x7f5d701084e0 2024-03-18T01:27:16.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.199+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f5d7002dc80 con 0x7f5d54039020 2024-03-18T01:27:16.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.207+0000 7f5d67400700 1 -- 172.21.15.7:0/1149073372 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5d7002dc80 con 0x7f5d54039020 2024-03-18T01:27:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.209+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5d54039020 msgr2=0x7f5d5403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.209+0000 7f5d6fe00700 1 --2- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5d54039020 0x7f5d5403b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c006fd0 tx=0x7f5d5c006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:16.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.210+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 msgr2=0x7f5d7019d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.210+0000 7f5d6fe00700 1 --2- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 0x7f5d7019d820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005850 tx=0x7f5d60005030 comp rx=0 tx=0).stop 2024-03-18T01:27:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.210+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 shutdown_connections 2024-03-18T01:27:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.210+0000 7f5d6fe00700 1 --2- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5d54039020 0x7f5d5403b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.210+0000 7f5d6fe00700 1 --2- 172.21.15.7:0/1149073372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5d701084e0 0x7f5d7019d820 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.210+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 >> 172.21.15.7:0/1149073372 conn(0x7f5d700772c0 msgr2=0x7f5d70077cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.210+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 shutdown_connections 2024-03-18T01:27:16.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:16.210+0000 7f5d6fe00700 1 -- 172.21.15.7:0/1149073372 wait complete. 2024-03-18T01:27:16.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:16.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:16.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:27:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:27:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:27:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:27:16.994 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi046 2024-03-18T01:27:16.994 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:27:16.994 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/etc/ceph/ceph.conf 2024-03-18T01:27:17.013 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:27:17.013 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:27:17.071 INFO:tasks.cephadm:Adding host smithi046 to orchestrator... 2024-03-18T01:27:17.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch host add smithi046 2024-03-18T01:27:17.495 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:27:17.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:17 smithi007 ceph-mon[31996]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:27:17.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:17 smithi007 ceph-mon[31996]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-18T01:27:17.641 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:17 smithi007 ceph-mon[31996]: Updating smithi007:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:27:18.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:18 smithi007 ceph-mon[31996]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:18.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:18 smithi007 ceph-mon[31996]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:27:19.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.601+0000 7f3b7f200700 1 -- 172.21.15.7:0/2297845677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 msgr2=0x7f3b78103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:19.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.601+0000 7f3b7f200700 1 --2- 172.21.15.7:0/2297845677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 0x7f3b78103240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800b0a0 tx=0x7f3b6800b3b0 comp rx=0 tx=0).stop 2024-03-18T01:27:19.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.601+0000 7f3b7f200700 1 -- 172.21.15.7:0/2297845677 shutdown_connections 2024-03-18T01:27:19.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.601+0000 7f3b7f200700 1 --2- 172.21.15.7:0/2297845677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 0x7f3b78103240 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:19.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.601+0000 7f3b7f200700 1 -- 172.21.15.7:0/2297845677 >> 172.21.15.7:0/2297845677 conn(0x7f3b780fc690 msgr2=0x7f3b780feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:19.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.602+0000 7f3b7f200700 1 -- 172.21.15.7:0/2297845677 shutdown_connections 2024-03-18T01:27:19.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.602+0000 7f3b7f200700 1 -- 172.21.15.7:0/2297845677 wait complete. 2024-03-18T01:27:19.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.603+0000 7f3b7f200700 1 Processor -- start 2024-03-18T01:27:19.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.603+0000 7f3b7f200700 1 -- start start 2024-03-18T01:27:19.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.603+0000 7f3b7f200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 0x7f3b78100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:19.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.603+0000 7f3b7f200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7810c330 con 0x7f3b78102e60 2024-03-18T01:27:19.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.603+0000 7f3b77e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 0x7f3b78100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:19.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.604+0000 7f3b77e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 0x7f3b78100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58988/0 (socket says 172.21.15.7:58988) 2024-03-18T01:27:19.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.604+0000 7f3b77e00700 1 -- 172.21.15.7:0/15275783 learned_addr learned my addr 172.21.15.7:0/15275783 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:19.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.604+0000 7f3b77e00700 1 -- 172.21.15.7:0/15275783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b781029b0 con 0x7f3b78102e60 2024-03-18T01:27:19.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.604+0000 7f3b77e00700 1 --2- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 0x7f3b78100ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b68000f80 tx=0x7f3b6800bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:19.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.605+0000 7f3b75600700 1 -- 172.21.15.7:0/15275783 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b68020020 con 0x7f3b78102e60 2024-03-18T01:27:19.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.605+0000 7f3b75600700 1 -- 172.21.15.7:0/15275783 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b68018040 con 0x7f3b78102e60 2024-03-18T01:27:19.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.605+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b78101060 con 0x7f3b78102e60 2024-03-18T01:27:19.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.605+0000 7f3b75600700 1 -- 172.21.15.7:0/15275783 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3b68017070 con 0x7f3b78102e60 2024-03-18T01:27:19.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.605+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b78101520 con 0x7f3b78102e60 2024-03-18T01:27:19.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.606+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b7806a990 con 0x7f3b78102e60 2024-03-18T01:27:19.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.606+0000 7f3b75600700 1 -- 172.21.15.7:0/15275783 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f3b68023020 con 0x7f3b78102e60 2024-03-18T01:27:19.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.607+0000 7f3b75600700 1 --2- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3b5c039060 0x7f3b5c03b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:19.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.607+0000 7f3b75600700 1 -- 172.21.15.7:0/15275783 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3b6800f050 con 0x7f3b78102e60 2024-03-18T01:27:19.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.607+0000 7f3b6f400700 1 --2- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3b5c039060 0x7f3b5c03b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:19.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.608+0000 7f3b6f400700 1 --2- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3b5c039060 0x7f3b5c03b520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3b60006fd0 tx=0x7f3b60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:19.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.612+0000 7f3b75600700 1 -- 172.21.15.7:0/15275783 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3b6802a020 con 0x7f3b78102e60 2024-03-18T01:27:19.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:19.849+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi046", "target": ["mon-mgr", ""]}) -- 0x7f3b78101ff0 con 0x7f3b5c039060 2024-03-18T01:27:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:19 smithi007 ceph-mon[31996]: Updating smithi007:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T01:27:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:19 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:19 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:19 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:21.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:20 smithi007 ceph-mon[31996]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:21.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:20 smithi007 ceph-mon[31996]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi046", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:27:22.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:21 smithi007 ceph-mon[31996]: Deploying cephadm binary to smithi046 2024-03-18T01:27:23.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:22 smithi007 ceph-mon[31996]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:24.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.923+0000 7f3b75600700 1 -- 172.21.15.7:0/15275783 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f3b78101ff0 con 0x7f3b5c039060 2024-03-18T01:27:24.924 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi046' with addr '172.21.15.46' 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3b5c039060 msgr2=0x7f3b5c03b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 --2- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3b5c039060 0x7f3b5c03b520 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3b60006fd0 tx=0x7f3b60006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 msgr2=0x7f3b78100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 --2- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 0x7f3b78100ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b68000f80 tx=0x7f3b6800bd20 comp rx=0 tx=0).stop 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 shutdown_connections 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 --2- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3b5c039060 0x7f3b5c03b520 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 --2- 172.21.15.7:0/15275783 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3b78102e60 0x7f3b78100ac0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 >> 172.21.15.7:0/15275783 conn(0x7f3b780fc690 msgr2=0x7f3b78103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 shutdown_connections 2024-03-18T01:27:24.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:24.927+0000 7f3b7f200700 1 -- 172.21.15.7:0/15275783 wait complete. 2024-03-18T01:27:25.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:24 smithi007 ceph-mon[31996]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:25.755 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T01:27:26.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:26.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:25 smithi007 ceph-mon[31996]: Added host smithi046 2024-03-18T01:27:26.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:27:26.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:25 smithi007 ceph-mon[31996]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:26.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:26.200 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:27:27.149 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:28.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.203+0000 7ff9bd600700 1 -- 172.21.15.7:0/3314958364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 msgr2=0x7ff9b8104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:28.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.203+0000 7ff9bd600700 1 --2- 172.21.15.7:0/3314958364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 0x7ff9b8104500 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009910 tx=0x7ff9a8009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:28.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.203+0000 7ff9bd600700 1 -- 172.21.15.7:0/3314958364 shutdown_connections 2024-03-18T01:27:28.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.203+0000 7ff9bd600700 1 --2- 172.21.15.7:0/3314958364 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 0x7ff9b8104500 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:28.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.203+0000 7ff9bd600700 1 -- 172.21.15.7:0/3314958364 >> 172.21.15.7:0/3314958364 conn(0x7ff9b80ff9d0 msgr2=0x7ff9b8101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:28.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.203+0000 7ff9bd600700 1 -- 172.21.15.7:0/3314958364 shutdown_connections 2024-03-18T01:27:28.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.203+0000 7ff9bd600700 1 -- 172.21.15.7:0/3314958364 wait complete. 2024-03-18T01:27:28.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.204+0000 7ff9bd600700 1 Processor -- start 2024-03-18T01:27:28.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.205+0000 7ff9bd600700 1 -- start start 2024-03-18T01:27:28.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.205+0000 7ff9bd600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 0x7ff9b819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:28.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.205+0000 7ff9bd600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b810e4f0 con 0x7ff9b8104120 2024-03-18T01:27:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.205+0000 7ff9b6a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 0x7ff9b819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.205+0000 7ff9b6a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 0x7ff9b819f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58708/0 (socket says 172.21.15.7:58708) 2024-03-18T01:27:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.205+0000 7ff9b6a00700 1 -- 172.21.15.7:0/3443823068 learned_addr learned my addr 172.21.15.7:0/3443823068 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.206+0000 7ff9b6a00700 1 -- 172.21.15.7:0/3443823068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b819fbe0 con 0x7ff9b8104120 2024-03-18T01:27:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.206+0000 7ff9b6a00700 1 --2- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 0x7ff9b819f6a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8006010 tx=0x7ff9a8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.206+0000 7ff9af400700 1 -- 172.21.15.7:0/3443823068 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9a8026070 con 0x7ff9b8104120 2024-03-18T01:27:28.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.206+0000 7ff9af400700 1 -- 172.21.15.7:0/3443823068 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9a800bbd0 con 0x7ff9b8104120 2024-03-18T01:27:28.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.206+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b819fed0 con 0x7ff9b8104120 2024-03-18T01:27:28.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.206+0000 7ff9af400700 1 -- 172.21.15.7:0/3443823068 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff9a8003910 con 0x7ff9b8104120 2024-03-18T01:27:28.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.207+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b8199820 con 0x7ff9b8104120 2024-03-18T01:27:28.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.208+0000 7ff9af400700 1 -- 172.21.15.7:0/3443823068 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7ff9a8029070 con 0x7ff9b8104120 2024-03-18T01:27:28.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.208+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b8105600 con 0x7ff9b8104120 2024-03-18T01:27:28.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.208+0000 7ff9af400700 1 --2- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff99c038fc0 0x7ff99c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:28.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.208+0000 7ff9af400700 1 -- 172.21.15.7:0/3443823068 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff9a801e070 con 0x7ff9b8104120 2024-03-18T01:27:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.208+0000 7ff9b6000700 1 --2- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff99c038fc0 0x7ff99c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:28.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.209+0000 7ff9b6000700 1 --2- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff99c038fc0 0x7ff99c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4006fd0 tx=0x7ff9a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:28.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.214+0000 7ff9af400700 1 -- 172.21.15.7:0/3443823068 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff9a8013070 con 0x7ff9b8104120 2024-03-18T01:27:28.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.449+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9b819ad50 con 0x7ff99c038fc0 2024-03-18T01:27:28.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.451+0000 7ff9af400700 1 -- 172.21.15.7:0/3443823068 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+158 (secure 0 0 0) 0x7ff9b819ad50 con 0x7ff99c038fc0 2024-03-18T01:27:28.451 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:28.451 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}] 2024-03-18T01:27:28.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff99c038fc0 msgr2=0x7ff99c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:28.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 --2- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff99c038fc0 0x7ff99c03b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4006fd0 tx=0x7ff9a4006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:28.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 msgr2=0x7ff9b819f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:28.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 --2- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 0x7ff9b819f6a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8006010 tx=0x7ff9a8005030 comp rx=0 tx=0).stop 2024-03-18T01:27:28.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 shutdown_connections 2024-03-18T01:27:28.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 --2- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff99c038fc0 0x7ff99c03b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:28.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 --2- 172.21.15.7:0/3443823068 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff9b8104120 0x7ff9b819f6a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:28.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 >> 172.21.15.7:0/3443823068 conn(0x7ff9b80ff9d0 msgr2=0x7ff9b8105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:28.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 shutdown_connections 2024-03-18T01:27:28.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:28.454+0000 7ff9bd600700 1 -- 172.21.15.7:0/3443823068 wait complete. 2024-03-18T01:27:28.788 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:28 smithi007 ceph-mon[31996]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:28.788 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:29.122 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-03-18T01:27:29.122 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:27:29.122 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-03-18T01:27:29.141 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:27:29.141 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:27:29.199 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-03-18T01:27:29.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch host add smithi203 2024-03-18T01:27:29.657 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:27:29.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:29 smithi007 ceph-mon[31996]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:27:30.757 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:30 smithi007 ceph-mon[31996]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:31.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.586+0000 7f8567000700 1 -- 172.21.15.7:0/1961773690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 msgr2=0x7f8560103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:31.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.586+0000 7f8567000700 1 --2- 172.21.15.7:0/1961773690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 0x7f8560103240 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8550009910 tx=0x7f8550009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:31.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.586+0000 7f8567000700 1 -- 172.21.15.7:0/1961773690 shutdown_connections 2024-03-18T01:27:31.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.586+0000 7f8567000700 1 --2- 172.21.15.7:0/1961773690 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 0x7f8560103240 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:31.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.586+0000 7f8567000700 1 -- 172.21.15.7:0/1961773690 >> 172.21.15.7:0/1961773690 conn(0x7f8560077810 msgr2=0x7f85600feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:31.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.586+0000 7f8567000700 1 -- 172.21.15.7:0/1961773690 shutdown_connections 2024-03-18T01:27:31.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.586+0000 7f8567000700 1 -- 172.21.15.7:0/1961773690 wait complete. 2024-03-18T01:27:31.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.587+0000 7f8567000700 1 Processor -- start 2024-03-18T01:27:31.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.588+0000 7f8567000700 1 -- start start 2024-03-18T01:27:31.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.588+0000 7f8567000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 0x7f856019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:31.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.588+0000 7f8567000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f856010c350 con 0x7f8560102e60 2024-03-18T01:27:31.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.588+0000 7f855fe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 0x7f856019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:31.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.588+0000 7f855fe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 0x7f856019f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58730/0 (socket says 172.21.15.7:58730) 2024-03-18T01:27:31.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.588+0000 7f855fe00700 1 -- 172.21.15.7:0/1706241669 learned_addr learned my addr 172.21.15.7:0/1706241669 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:31.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.589+0000 7f855fe00700 1 -- 172.21.15.7:0/1706241669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856019fbe0 con 0x7f8560102e60 2024-03-18T01:27:31.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.589+0000 7f855fe00700 1 --2- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 0x7f856019f6a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8550006010 tx=0x7f8550005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:31.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.589+0000 7f855cc00700 1 -- 172.21.15.7:0/1706241669 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8550026070 con 0x7f8560102e60 2024-03-18T01:27:31.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.589+0000 7f855cc00700 1 -- 172.21.15.7:0/1706241669 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f855000bc30 con 0x7f8560102e60 2024-03-18T01:27:31.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.589+0000 7f8567000700 1 -- 172.21.15.7:0/1706241669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f856019fe70 con 0x7f8560102e60 2024-03-18T01:27:31.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.589+0000 7f855cc00700 1 -- 172.21.15.7:0/1706241669 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85500039a0 con 0x7f8560102e60 2024-03-18T01:27:31.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.590+0000 7f8567000700 1 -- 172.21.15.7:0/1706241669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8560199860 con 0x7f8560102e60 2024-03-18T01:27:31.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.591+0000 7f8556000700 1 -- 172.21.15.7:0/1706241669 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8560069fa0 con 0x7f8560102e60 2024-03-18T01:27:31.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.591+0000 7f855cc00700 1 -- 172.21.15.7:0/1706241669 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f8550029070 con 0x7f8560102e60 2024-03-18T01:27:31.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.591+0000 7f855cc00700 1 --2- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8544038fc0 0x7f854403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:31.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.592+0000 7f855cc00700 1 -- 172.21.15.7:0/1706241669 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f855001e070 con 0x7f8560102e60 2024-03-18T01:27:31.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.592+0000 7f855f400700 1 --2- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8544038fc0 0x7f854403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:31.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.592+0000 7f855f400700 1 --2- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8544038fc0 0x7f854403b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f854c006fd0 tx=0x7f854c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:31.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.597+0000 7f855cc00700 1 -- 172.21.15.7:0/1706241669 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8550013070 con 0x7f8560102e60 2024-03-18T01:27:31.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:31.832+0000 7f8556000700 1 -- 172.21.15.7:0/1706241669 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) -- 0x7f8560063900 con 0x7f8544038fc0 2024-03-18T01:27:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:27:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:27:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:31 smithi007 ceph-mon[31996]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-18T01:27:33.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:32 smithi007 ceph-mon[31996]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:32 smithi007 ceph-mon[31996]: Updating smithi046:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:27:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:32 smithi007 ceph-mon[31996]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:27:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:32 smithi007 ceph-mon[31996]: Updating smithi046:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T01:27:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:32 smithi007 ceph-mon[31996]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:27:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:34.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:33 smithi007 ceph-mon[31996]: Deploying cephadm binary to smithi203 2024-03-18T01:27:35.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:34 smithi007 ceph-mon[31996]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:36.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.656+0000 7f855cc00700 1 -- 172.21.15.7:0/1706241669 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8560063900 con 0x7f8544038fc0 2024-03-18T01:27:36.657 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-03-18T01:27:36.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 -- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8544038fc0 msgr2=0x7f854403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:36.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 --2- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8544038fc0 0x7f854403b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f854c006fd0 tx=0x7f854c006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:36.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 -- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 msgr2=0x7f856019f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:36.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 --2- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 0x7f856019f6a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8550006010 tx=0x7f8550005030 comp rx=0 tx=0).stop 2024-03-18T01:27:36.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 -- 172.21.15.7:0/1706241669 shutdown_connections 2024-03-18T01:27:36.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 --2- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8544038fc0 0x7f854403b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:36.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 --2- 172.21.15.7:0/1706241669 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8560102e60 0x7f856019f6a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:36.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 -- 172.21.15.7:0/1706241669 >> 172.21.15.7:0/1706241669 conn(0x7f8560077810 msgr2=0x7f8560108020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:36.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 -- 172.21.15.7:0/1706241669 shutdown_connections 2024-03-18T01:27:36.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:36.660+0000 7f8556000700 1 -- 172.21.15.7:0/1706241669 wait complete. 2024-03-18T01:27:36.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:36 smithi007 ceph-mon[31996]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:36.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:36.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:27:37.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T01:27:37.841 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:27:38.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:37 smithi007 ceph-mon[31996]: Added host smithi203 2024-03-18T01:27:38.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:37 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:38.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:37 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:38.993 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:38 smithi007 ceph-mon[31996]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.805+0000 7fd2cea00700 1 -- 172.21.15.7:0/1628068631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 msgr2=0x7fd2c8074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.805+0000 7fd2cea00700 1 --2- 172.21.15.7:0/1628068631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 0x7fd2c8074a80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009910 tx=0x7fd2b8009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.806+0000 7fd2cea00700 1 -- 172.21.15.7:0/1628068631 shutdown_connections 2024-03-18T01:27:39.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.806+0000 7fd2cea00700 1 --2- 172.21.15.7:0/1628068631 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 0x7fd2c8074a80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.806+0000 7fd2cea00700 1 -- 172.21.15.7:0/1628068631 >> 172.21.15.7:0/1628068631 conn(0x7fd2c80fd7b0 msgr2=0x7fd2c80ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.806+0000 7fd2cea00700 1 -- 172.21.15.7:0/1628068631 shutdown_connections 2024-03-18T01:27:39.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.806+0000 7fd2cea00700 1 -- 172.21.15.7:0/1628068631 wait complete. 2024-03-18T01:27:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.807+0000 7fd2cea00700 1 Processor -- start 2024-03-18T01:27:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.807+0000 7fd2cea00700 1 -- start start 2024-03-18T01:27:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.807+0000 7fd2cea00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 0x7fd2c819f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.807+0000 7fd2cea00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c810e540 con 0x7fd2c80746a0 2024-03-18T01:27:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.808+0000 7fd2c7e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 0x7fd2c819f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.808+0000 7fd2c7e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 0x7fd2c819f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46412/0 (socket says 172.21.15.7:46412) 2024-03-18T01:27:39.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.808+0000 7fd2c7e00700 1 -- 172.21.15.7:0/2085573884 learned_addr learned my addr 172.21.15.7:0/2085573884 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.808+0000 7fd2c7e00700 1 -- 172.21.15.7:0/2085573884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c819fbf0 con 0x7fd2c80746a0 2024-03-18T01:27:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.808+0000 7fd2c7e00700 1 --2- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 0x7fd2c819f6b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8006010 tx=0x7fd2b8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.809+0000 7fd2c4c00700 1 -- 172.21.15.7:0/2085573884 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2b8026070 con 0x7fd2c80746a0 2024-03-18T01:27:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.809+0000 7fd2c4c00700 1 -- 172.21.15.7:0/2085573884 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2b800bc30 con 0x7fd2c80746a0 2024-03-18T01:27:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.809+0000 7fd2c4c00700 1 -- 172.21.15.7:0/2085573884 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2b80039a0 con 0x7fd2c80746a0 2024-03-18T01:27:39.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.809+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c819fe80 con 0x7fd2c80746a0 2024-03-18T01:27:39.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.809+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c8199870 con 0x7fd2c80746a0 2024-03-18T01:27:39.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.810+0000 7fd2c4c00700 1 -- 172.21.15.7:0/2085573884 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7fd2b8029070 con 0x7fd2c80746a0 2024-03-18T01:27:39.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.810+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c8075d50 con 0x7fd2c80746a0 2024-03-18T01:27:39.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.810+0000 7fd2c4c00700 1 --2- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2ac039020 0x7fd2ac03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:39.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.810+0000 7fd2c4c00700 1 -- 172.21.15.7:0/2085573884 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd2b801e070 con 0x7fd2c80746a0 2024-03-18T01:27:39.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.811+0000 7fd2c7400700 1 --2- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2ac039020 0x7fd2ac03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:39.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.811+0000 7fd2c7400700 1 --2- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2ac039020 0x7fd2ac03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4006fd0 tx=0x7fd2b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:39.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:39.816+0000 7fd2c4c00700 1 -- 172.21.15.7:0/2085573884 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd2b8013070 con 0x7fd2c80746a0 2024-03-18T01:27:40.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.052+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd2c819ada0 con 0x7fd2ac039020 2024-03-18T01:27:40.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.054+0000 7fd2c4c00700 1 -- 172.21.15.7:0/2085573884 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7fd2c819ada0 con 0x7fd2ac039020 2024-03-18T01:27:40.056 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:27:40.056 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-03-18T01:27:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.057+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2ac039020 msgr2=0x7fd2ac03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.057+0000 7fd2cea00700 1 --2- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2ac039020 0x7fd2ac03b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4006fd0 tx=0x7fd2b4006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.057+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 msgr2=0x7fd2c819f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.057+0000 7fd2cea00700 1 --2- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 0x7fd2c819f6b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8006010 tx=0x7fd2b8005030 comp rx=0 tx=0).stop 2024-03-18T01:27:40.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.058+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 shutdown_connections 2024-03-18T01:27:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.058+0000 7fd2cea00700 1 --2- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2ac039020 0x7fd2ac03b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.058+0000 7fd2cea00700 1 --2- 172.21.15.7:0/2085573884 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd2c80746a0 0x7fd2c819f6b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.058+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 >> 172.21.15.7:0/2085573884 conn(0x7fd2c80fd7b0 msgr2=0x7fd2c8105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.058+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 shutdown_connections 2024-03-18T01:27:40.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:40.058+0000 7fd2cea00700 1 -- 172.21.15.7:0/2085573884 wait complete. 2024-03-18T01:27:40.687 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:40 smithi007 ceph-mon[31996]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:40.687 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:40.759 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T01:27:40.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T01:27:41.223 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:27:41.625 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:41 smithi007 ceph-mon[31996]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:27:42.664 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:42 smithi007 ceph-mon[31996]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:43.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.136+0000 7f275fe00700 1 -- 172.21.15.7:0/1927999052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 msgr2=0x7f27581011d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:43.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.136+0000 7f275fe00700 1 --2- 172.21.15.7:0/1927999052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 0x7f27581011d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f274c009910 tx=0x7f274c009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:43.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.137+0000 7f275fe00700 1 -- 172.21.15.7:0/1927999052 shutdown_connections 2024-03-18T01:27:43.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.137+0000 7f275fe00700 1 --2- 172.21.15.7:0/1927999052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 0x7f27581011d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:43.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.137+0000 7f275fe00700 1 -- 172.21.15.7:0/1927999052 >> 172.21.15.7:0/1927999052 conn(0x7f27580fc680 msgr2=0x7f27580feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:43.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.137+0000 7f275fe00700 1 -- 172.21.15.7:0/1927999052 shutdown_connections 2024-03-18T01:27:43.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.137+0000 7f275fe00700 1 -- 172.21.15.7:0/1927999052 wait complete. 2024-03-18T01:27:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.138+0000 7f275fe00700 1 Processor -- start 2024-03-18T01:27:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.138+0000 7f275fe00700 1 -- start start 2024-03-18T01:27:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.138+0000 7f275fe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 0x7f275819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.138+0000 7f275fe00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f275810b1c0 con 0x7f2758100df0 2024-03-18T01:27:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.139+0000 7f275d400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 0x7f275819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.139+0000 7f275d400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 0x7f275819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46430/0 (socket says 172.21.15.7:46430) 2024-03-18T01:27:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.139+0000 7f275d400700 1 -- 172.21.15.7:0/4161717077 learned_addr learned my addr 172.21.15.7:0/4161717077 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:27:43.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.139+0000 7f275d400700 1 -- 172.21.15.7:0/4161717077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275819d980 con 0x7f2758100df0 2024-03-18T01:27:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.139+0000 7f275d400700 1 --2- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 0x7f275819d440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f274c005850 tx=0x7f274c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.140+0000 7f2756000700 1 -- 172.21.15.7:0/4161717077 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f274c026070 con 0x7f2758100df0 2024-03-18T01:27:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.140+0000 7f2756000700 1 -- 172.21.15.7:0/4161717077 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f274c00bbd0 con 0x7f2758100df0 2024-03-18T01:27:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.140+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275819dc70 con 0x7f2758100df0 2024-03-18T01:27:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.140+0000 7f2756000700 1 -- 172.21.15.7:0/4161717077 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f274c003970 con 0x7f2758100df0 2024-03-18T01:27:43.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.140+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2758197600 con 0x7f2758100df0 2024-03-18T01:27:43.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.141+0000 7f2756000700 1 -- 172.21.15.7:0/4161717077 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f274c029070 con 0x7f2758100df0 2024-03-18T01:27:43.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.141+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27581022d0 con 0x7f2758100df0 2024-03-18T01:27:43.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.141+0000 7f2756000700 1 --2- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2740038fc0 0x7f274003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:43.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.142+0000 7f2756000700 1 -- 172.21.15.7:0/4161717077 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f274c01e070 con 0x7f2758100df0 2024-03-18T01:27:43.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.142+0000 7f275ca00700 1 --2- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2740038fc0 0x7f274003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:43.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.142+0000 7f275ca00700 1 --2- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2740038fc0 0x7f274003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2744006fd0 tx=0x7f2744006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:43.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.147+0000 7f2756000700 1 -- 172.21.15.7:0/4161717077 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f274c013070 con 0x7f2758100df0 2024-03-18T01:27:43.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.383+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f2758063900 con 0x7f2758100df0 2024-03-18T01:27:43.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.648+0000 7f2756000700 1 -- 172.21.15.7:0/4161717077 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f274c00fe20 con 0x7f2758100df0 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-18T01:27:43.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:43 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4161717077' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T01:27:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.653+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2740038fc0 msgr2=0x7f274003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.653+0000 7f275fe00700 1 --2- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2740038fc0 0x7f274003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2744006fd0 tx=0x7f2744006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.653+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 msgr2=0x7f275819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.653+0000 7f275fe00700 1 --2- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 0x7f275819d440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f274c005850 tx=0x7f274c005030 comp rx=0 tx=0).stop 2024-03-18T01:27:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.653+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 shutdown_connections 2024-03-18T01:27:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.653+0000 7f275fe00700 1 --2- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2740038fc0 0x7f274003b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.653+0000 7f275fe00700 1 --2- 172.21.15.7:0/4161717077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2758100df0 0x7f275819d440 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.653+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 >> 172.21.15.7:0/4161717077 conn(0x7f27580fc680 msgr2=0x7f2758069f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.654+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 shutdown_connections 2024-03-18T01:27:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:27:43.654+0000 7f275fe00700 1 -- 172.21.15.7:0/4161717077 wait complete. 2024-03-18T01:27:43.657 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-03-18T01:27:44.389 INFO:tasks.cephadm:Adding mon.a on smithi007 2024-03-18T01:27:44.389 INFO:tasks.cephadm:Adding mon.b on smithi046 2024-03-18T01:27:44.389 INFO:tasks.cephadm:Adding mon.c on smithi203 2024-03-18T01:27:44.389 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi046:172.21.15.46=b;smithi203:172.21.15.203=c' 2024-03-18T01:27:44.620 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:27:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: Updating smithi203:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:27:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:27:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: Updating smithi203:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T01:27:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4161717077' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T01:27:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:27:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.951+0000 7f57f8062700 1 -- 172.21.15.203:0/2022272507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 msgr2=0x7f57f0101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.951+0000 7f57f8062700 1 --2- 172.21.15.203:0/2022272507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 0x7f57f0101030 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009910 tx=0x7f57ec009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.951+0000 7f57f8062700 1 -- 172.21.15.203:0/2022272507 shutdown_connections 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.951+0000 7f57f8062700 1 --2- 172.21.15.203:0/2022272507 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 0x7f57f0101030 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.951+0000 7f57f8062700 1 -- 172.21.15.203:0/2022272507 >> 172.21.15.203:0/2022272507 conn(0x7f57f00fc690 msgr2=0x7f57f00feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.951+0000 7f57f8062700 1 -- 172.21.15.203:0/2022272507 shutdown_connections 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.951+0000 7f57f8062700 1 -- 172.21.15.203:0/2022272507 wait complete. 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.952+0000 7f57f8062700 1 Processor -- start 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.952+0000 7f57f8062700 1 -- start start 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.952+0000 7f57f8062700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 0x7f57f019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:45.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.952+0000 7f57f8062700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0112930 con 0x7f57f0100c50 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.953+0000 7f57f5dfe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 0x7f57f019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.953+0000 7f57f5dfe700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 0x7f57f019de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.203:44342/0 (socket says 172.21.15.203:44342) 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.953+0000 7f57f5dfe700 1 -- 172.21.15.203:0/1373139869 learned_addr learned my addr 172.21.15.203:0/1373139869 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.953+0000 7f57f5dfe700 1 -- 172.21.15.203:0/1373139869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f019e3b0 con 0x7f57f0100c50 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.954+0000 7f57f5dfe700 1 --2- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 0x7f57f019de70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f57ec006010 tx=0x7f57ec005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.954+0000 7f57e6ffd700 1 -- 172.21.15.203:0/1373139869 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57ec026070 con 0x7f57f0100c50 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.954+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f019e640 con 0x7f57f0100c50 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.954+0000 7f57e6ffd700 1 -- 172.21.15.203:0/1373139869 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57ec00bc30 con 0x7f57f0100c50 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.954+0000 7f57e6ffd700 1 -- 172.21.15.203:0/1373139869 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57ec003ab0 con 0x7f57f0100c50 2024-03-18T01:27:45.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.954+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f019eaa0 con 0x7f57f0100c50 2024-03-18T01:27:45.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.955+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f006a990 con 0x7f57f0100c50 2024-03-18T01:27:45.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.956+0000 7f57e6ffd700 1 -- 172.21.15.203:0/1373139869 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f57ec029070 con 0x7f57f0100c50 2024-03-18T01:27:45.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.956+0000 7f57e6ffd700 1 --2- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57dc038fc0 0x7f57dc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:45.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.956+0000 7f57e6ffd700 1 -- 172.21.15.203:0/1373139869 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f57ec01e070 con 0x7f57f0100c50 2024-03-18T01:27:45.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.957+0000 7f57f55fd700 1 --2- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57dc038fc0 0x7f57dc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:45.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.957+0000 7f57f55fd700 1 --2- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57dc038fc0 0x7f57dc03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f57e0006fd0 tx=0x7f57e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:45.972 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:45.971+0000 7f57e6ffd700 1 -- 172.21.15.203:0/1373139869 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f57ec013070 con 0x7f57f0100c50 2024-03-18T01:27:46.183 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.182+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi046:172.21.15.46=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}) -- 0x7f57f00fe6b0 con 0x7f57dc038fc0 2024-03-18T01:27:46.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.197+0000 7f57e6ffd700 1 -- 172.21.15.203:0/1373139869 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f57f00fe6b0 con 0x7f57dc038fc0 2024-03-18T01:27:46.198 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-03-18T01:27:46.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.199+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57dc038fc0 msgr2=0x7f57dc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:46.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.199+0000 7f57f8062700 1 --2- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57dc038fc0 0x7f57dc03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f57e0006fd0 tx=0x7f57e0006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:46.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.199+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 msgr2=0x7f57f019de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:46.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.199+0000 7f57f8062700 1 --2- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 0x7f57f019de70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f57ec006010 tx=0x7f57ec005050 comp rx=0 tx=0).stop 2024-03-18T01:27:46.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.199+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 shutdown_connections 2024-03-18T01:27:46.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.199+0000 7f57f8062700 1 --2- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57dc038fc0 0x7f57dc03b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:46.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.199+0000 7f57f8062700 1 --2- 172.21.15.203:0/1373139869 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57f0100c50 0x7f57f019de70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:46.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.199+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 >> 172.21.15.203:0/1373139869 conn(0x7f57f00fc690 msgr2=0x7f57f00fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:46.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.200+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 shutdown_connections 2024-03-18T01:27:46.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:46.200+0000 7f57f8062700 1 -- 172.21.15.203:0/1373139869 wait complete. 2024-03-18T01:27:46.813 DEBUG:teuthology.orchestra.run.smithi046:mon.b> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.b.service 2024-03-18T01:27:46.816 DEBUG:teuthology.orchestra.run.smithi203:mon.c> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.c.service 2024-03-18T01:27:46.818 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T01:27:46.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:27:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:46 smithi007 ceph-mon[31996]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:27:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:27:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:27:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:48.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:47 smithi007 ceph-mon[31996]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi046:172.21.15.46=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:27:48.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:47 smithi007 ceph-mon[31996]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi046:172.21.15.46=b;smithi203:172.21.15.203=c;count:3 2024-03-18T01:27:48.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:47 smithi007 ceph-mon[31996]: Deploying daemon mon.c on smithi203 2024-03-18T01:27:48.761 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:27:49.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:48 smithi007 ceph-mon[31996]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:50.512 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 podman[36297]: 2024-03-18 01:27:50.426233469 +0000 UTC m=+0.306986940 container create 459e01704c44f6672e945efba134ee3847269d0c96721f8194c861dc0c8d6ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-18T01:27:50.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.697+0000 7fedcc454700 1 -- 172.21.15.203:0/3255748186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 msgr2=0x7fedc4101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.697+0000 7fedcc454700 1 --2- 172.21.15.203:0/3255748186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 0x7fedc4101220 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009910 tx=0x7fedb4009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.697+0000 7fedcc454700 1 -- 172.21.15.203:0/3255748186 shutdown_connections 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.697+0000 7fedcc454700 1 --2- 172.21.15.203:0/3255748186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 0x7fedc4101220 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.697+0000 7fedcc454700 1 -- 172.21.15.203:0/3255748186 >> 172.21.15.203:0/3255748186 conn(0x7fedc40fc710 msgr2=0x7fedc40feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.697+0000 7fedcc454700 1 -- 172.21.15.203:0/3255748186 shutdown_connections 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.697+0000 7fedcc454700 1 -- 172.21.15.203:0/3255748186 wait complete. 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.698+0000 7fedcc454700 1 Processor -- start 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.698+0000 7fedcc454700 1 -- start start 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.698+0000 7fedcc454700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 0x7fedc419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:50.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.698+0000 7fedcc454700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fedc410b210 con 0x7fedc4100e40 2024-03-18T01:27:50.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.698+0000 7fedca1f0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 0x7fedc419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:50.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.699+0000 7fedca1f0700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 0x7fedc419d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.203:34244/0 (socket says 172.21.15.203:34244) 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.699+0000 7fedca1f0700 1 -- 172.21.15.203:0/1148459305 learned_addr learned my addr 172.21.15.203:0/1148459305 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.699+0000 7fedca1f0700 1 -- 172.21.15.203:0/1148459305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc419d950 con 0x7fedc4100e40 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.700+0000 7fedca1f0700 1 --2- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 0x7fedc419d410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fedb4005850 tx=0x7fedb4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.700+0000 7fedbb7fe700 1 -- 172.21.15.203:0/1148459305 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedb4026070 con 0x7fedc4100e40 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.700+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc419dc40 con 0x7fedc4100e40 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.700+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc41975d0 con 0x7fedc4100e40 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.700+0000 7fedbb7fe700 1 -- 172.21.15.203:0/1148459305 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb400b6f0 con 0x7fedc4100e40 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.700+0000 7fedbb7fe700 1 -- 172.21.15.203:0/1148459305 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedb4030060 con 0x7fedc4100e40 2024-03-18T01:27:50.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.701+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feda8005330 con 0x7fedc4100e40 2024-03-18T01:27:50.702 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.702+0000 7fedbb7fe700 1 -- 172.21.15.203:0/1148459305 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7fedb4029070 con 0x7fedc4100e40 2024-03-18T01:27:50.702 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.702+0000 7fedbb7fe700 1 --2- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fedb0038fc0 0x7fedb003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:50.702 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.702+0000 7fedbb7fe700 1 -- 172.21.15.203:0/1148459305 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fedb401e070 con 0x7fedc4100e40 2024-03-18T01:27:50.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.702+0000 7fedc99ef700 1 --2- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fedb0038fc0 0x7fedb003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:50.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.703+0000 7fedc99ef700 1 --2- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fedb0038fc0 0x7fedb003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fedc0006fd0 tx=0x7fedc0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:50.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.717+0000 7fedbb7fe700 1 -- 172.21.15.203:0/1148459305 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fedb407b740 con 0x7fedc4100e40 2024-03-18T01:27:50.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:50.831+0000 7fedbb7fe700 1 -- 172.21.15.203:0/1148459305 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fedb4021ca0 con 0x7fedc4100e40 2024-03-18T01:27:51.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:51.000+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7feda80058f0 con 0x7fedc4100e40 2024-03-18T01:27:51.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:50 smithi007 ceph-mon[31996]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 podman[36297]: 2024-03-18 01:27:50.763325305 +0000 UTC m=+0.644078780 container init 459e01704c44f6672e945efba134ee3847269d0c96721f8194c861dc0c8d6ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T01:27:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 podman[36297]: 2024-03-18 01:27:50.766682818 +0000 UTC m=+0.647436278 container start 459e01704c44f6672e945efba134ee3847269d0c96721f8194c861dc0c8d6ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T01:27:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T01:27:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T01:27:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pidfile_write: ignore empty --pid-file 2024-03-18T01:27:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: load: jerasure load: lrc 2024-03-18T01:27:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: RocksDB version: 7.9.2 2024-03-18T01:27:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Git sha 0 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: DB SUMMARY 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: DB Session ID: OVNZ1OKNEO0AQ5W7QSCJ 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: CURRENT file: CURRENT 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.error_if_exists: 0 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.create_if_missing: 0 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.paranoid_checks: 1 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T01:27:51.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T01:27:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.env: 0x55ffb7101c60 2024-03-18T01:27:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T01:27:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.info_log: 0x55ffb8125ac0 2024-03-18T01:27:51.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.statistics: (nil) 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.use_fsync: 0 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_log_file_size: 0 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.allow_fallocate: 1 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.use_direct_reads: 0 2024-03-18T01:27:51.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.db_log_dir: 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.wal_dir: 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.write_buffer_manager: 0x55ffb80f81e0 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T01:27:51.146 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.unordered_write: 0 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.row_cache: None 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.wal_filter: None 2024-03-18T01:27:51.147 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.two_write_queues: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.wal_compression: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.atomic_flush: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.log_readahead_size: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T01:27:51.148 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_background_jobs: 2 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_background_compactions: -1 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_subcompactions: 1 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T01:27:51.149 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_open_files: -1 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_background_flushes: -1 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Compression algorithms supported: 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: kZSTD supported: 0 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: kXpressCompression supported: 0 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: kZlibCompression supported: 1 2024-03-18T01:27:51.150 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: kSnappyCompression supported: 1 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: kLZ4Compression supported: 1 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: kBZip2Compression supported: 0 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.merge_operator: 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_filter: None 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T01:27:51.151 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ffb8125bc0) 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: cache_index_and_filter_blocks: 1 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: pin_top_level_index_and_filter: 1 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: index_type: 0 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: data_block_index_type: 0 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: index_shortening: 1 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: checksum: 4 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: no_block_cache: 0 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_cache: 0x55ffb812b090 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_cache_name: BinnedLRUCache 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_cache_options: 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: capacity : 536870912 2024-03-18T01:27:51.152 INFO:journalctl@ceph.mon.c.smithi203.stdout: num_shard_bits : 4 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: strict_capacity_limit : 0 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: high_pri_pool_ratio: 0.000 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_cache_compressed: (nil) 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: persistent_cache: (nil) 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_size: 4096 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_size_deviation: 10 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_restart_interval: 16 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: index_block_restart_interval: 1 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: metadata_block_size: 4096 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: partition_filters: 0 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: use_delta_encoding: 1 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: filter_policy: bloomfilter 2024-03-18T01:27:51.153 INFO:journalctl@ceph.mon.c.smithi203.stdout: whole_key_filtering: 1 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: verify_compression: 0 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: read_amp_bytes_per_bit: 0 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: format_version: 5 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: enable_index_compression: 1 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: block_align: 0 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: max_auto_readahead_size: 262144 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: prepopulate_block_cache: 0 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: initial_auto_readahead_size: 8192 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression: NoCompression 2024-03-18T01:27:51.154 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.num_levels: 7 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T01:27:51.155 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T01:27:51.156 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T01:27:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T01:27:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T01:27:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T01:27:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T01:27:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T01:27:51.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T01:27:51.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T01:27:51.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T01:27:51.158 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T01:27:51.159 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.table_properties_collectors: 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.inplace_update_support: 0 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T01:27:51.160 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.bloom_locality: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.max_successive_merges: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.ttl: 2592000 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.enable_blob_files: false 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.min_blob_size: 0 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T01:27:51.161 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f37c507b-4616-4c9b-8ce6-53dcc852b5af 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710725270796317, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710725270796756, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710725270, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f37c507b-4616-4c9b-8ce6-53dcc852b5af", "db_session_id": "OVNZ1OKNEO0AQ5W7QSCJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710725270796837, "job": 1, "event": "recovery_finished"} 2024-03-18T01:27:51.162 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ffb821e000 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: DB pointer 0x55ffb8208000 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: ** DB Stats ** 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T01:27:51.163 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: ** Compaction Stats [default] ** 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: ** Compaction Stats [default] ** 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-18T01:27:51.164 INFO:journalctl@ceph.mon.c.smithi203.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: Block cache BinnedLRUCache@0x55ffb812b090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T01:27:51.165 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: using public_addr v2:172.21.15.203:0/0 -> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: starting mon.c rank -1 at public addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] at bind addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(???) e0 preinit fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).mds e1 new map 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout: e1 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout: legacy client fscid: -1 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout: 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout: No filesystems configured 2024-03-18T01:27:51.166 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Added host smithi046 2024-03-18T01:27:51.167 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:27:51.168 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Updating smithi046:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Updating smithi046:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Deploying cephadm binary to smithi203 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.169 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Added host smithi203 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.170 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4161717077' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Updating smithi203:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Updating smithi203:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4161717077' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi046:172.21.15.46=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi046:172.21.15.46=b;smithi203:172.21.15.203=c;count:3 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: Deploying daemon mon.c on smithi203 2024-03-18T01:27:51.172 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:51.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 ceph-mon[36345]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T01:27:51.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 bash[36297]: 459e01704c44f6672e945efba134ee3847269d0c96721f8194c861dc0c8d6ee9 2024-03-18T01:27:51.173 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:50 smithi203 systemd[1]: Started Ceph mon.c for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:27:55.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.867+0000 7fedbb7fe700 1 -- 172.21.15.203:0/1148459305 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7fedb4013070 con 0x7fedc4100e40 2024-03-18T01:27:55.868 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:27:55.868 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","modified":"2024-03-18T01:27:50.830204Z","created":"2024-03-18T01:25:36.021301Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T01:27:55.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fedb0038fc0 msgr2=0x7fedb003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 --2- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fedb0038fc0 0x7fedb003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fedc0006fd0 tx=0x7fedc0006e40 comp rx=0 tx=0).stop 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 msgr2=0x7fedc419d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 --2- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 0x7fedc419d410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fedb4005850 tx=0x7fedb4005030 comp rx=0 tx=0).stop 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 shutdown_connections 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 --2- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fedb0038fc0 0x7fedb003b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 --2- 172.21.15.203:0/1148459305 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fedc4100e40 0x7fedc419d410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 >> 172.21.15.203:0/1148459305 conn(0x7fedc40fc710 msgr2=0x7fedc406a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 shutdown_connections 2024-03-18T01:27:55.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:55.870+0000 7fedcc454700 1 -- 172.21.15.203:0/1148459305 wait complete. 2024-03-18T01:27:55.872 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-03-18T01:27:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: mon.a calling monitor election 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/1148459305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: mon.c calling monitor election 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: monmap epoch 2 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: last_changed 2024-03-18T01:27:50.830204+0000 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: min_mon_release 19 (squid) 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: election_strategy: 1 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: fsmap 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: mgrmap e13: x(active, since 58s) 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: overall HEALTH_OK 2024-03-18T01:27:56.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:56.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:56.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:27:56.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: mon.a calling monitor election 2024-03-18T01:27:56.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/1148459305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T01:27:56.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:56.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: mon.c calling monitor election 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: monmap epoch 2 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: last_changed 2024-03-18T01:27:50.830204+0000 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: min_mon_release 19 (squid) 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: election_strategy: 1 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-03-18T01:27:56.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: fsmap 2024-03-18T01:27:56.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:27:56.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: mgrmap e13: x(active, since 58s) 2024-03-18T01:27:56.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: overall HEALTH_OK 2024-03-18T01:27:56.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:56.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:27:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:56 smithi007 ceph-mon[31996]: Deploying daemon mon.b on smithi046 2024-03-18T01:27:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:27:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:27:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:27:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:56 smithi203 ceph-mon[36345]: Deploying daemon mon.b on smithi046 2024-03-18T01:27:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:27:57.529 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T01:27:57.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:27:57.774 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:27:58.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:27:57 smithi007 ceph-mon[31996]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:58.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:27:57 smithi203 ceph-mon[36345]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:27:59.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.091+0000 7f84ed263700 1 -- 172.21.15.203:0/2468842320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e8102000 msgr2=0x7f84e81023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:59.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.091+0000 7f84ed263700 1 --2- 172.21.15.203:0/2468842320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e8102000 0x7f84e81023e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009910 tx=0x7f84d8009c20 comp rx=0 tx=0).stop 2024-03-18T01:27:59.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.091+0000 7f84ed263700 1 -- 172.21.15.203:0/2468842320 shutdown_connections 2024-03-18T01:27:59.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.091+0000 7f84ed263700 1 --2- 172.21.15.203:0/2468842320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e8102000 0x7f84e81023e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:59.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.091+0000 7f84ed263700 1 -- 172.21.15.203:0/2468842320 >> 172.21.15.203:0/2468842320 conn(0x7f84e80fd870 msgr2=0x7f84e80ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.091+0000 7f84ed263700 1 -- 172.21.15.203:0/2468842320 shutdown_connections 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.091+0000 7f84ed263700 1 -- 172.21.15.203:0/2468842320 wait complete. 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.091+0000 7f84ed263700 1 Processor -- start 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84ed263700 1 -- start start 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84ed263700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 0x7f84e819d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84ed263700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e819dc10 0x7f84e81978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84ed263700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f84e810c3d0 con 0x7f84e819dc10 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84ed263700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f84e810c250 con 0x7f84e8102000 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84e6ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 0x7f84e819d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84e6ffd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 0x7f84e819d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:42622/0 (socket says 172.21.15.203:42622) 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84e6ffd700 1 -- 172.21.15.203:0/2091454372 learned_addr learned my addr 172.21.15.203:0/2091454372 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84e6ffd700 1 -- 172.21.15.203:0/2091454372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 msgr2=0x7f84e819d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T01:27:59.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84e6ffd700 1 -- 172.21.15.203:0/2091454372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 msgr2=0x7f84e819d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T01:27:59.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84e6ffd700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 0x7f84e819d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T01:27:59.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84e6ffd700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 0x7f84e819d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:27:59.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.092+0000 7f84e67fc700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e819dc10 0x7f84e81978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:59.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.093+0000 7f84e67fc700 1 -- 172.21.15.203:0/2091454372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 msgr2=0x7f84e819d6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:27:59.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.093+0000 7f84e67fc700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 0x7f84e819d6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:59.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.093+0000 7f84e67fc700 1 -- 172.21.15.203:0/2091454372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e819e250 con 0x7f84e819dc10 2024-03-18T01:27:59.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.093+0000 7f84e67fc700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e819dc10 0x7f84e81978e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00b4d0 tx=0x7f84dc00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:59.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.094+0000 7f84cffff700 1 -- 172.21.15.203:0/2091454372 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f84dc011630 con 0x7f84e819dc10 2024-03-18T01:27:59.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.094+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e8197e20 con 0x7f84e819dc10 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.094+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e81983b0 con 0x7f84e819dc10 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.094+0000 7f84cffff700 1 -- 172.21.15.203:0/2091454372 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84dc011cb0 con 0x7f84e819dc10 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.094+0000 7f84cffff700 1 -- 172.21.15.203:0/2091454372 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f84dc0193b0 con 0x7f84e819dc10 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.094+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e81034e0 con 0x7f84e819dc10 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.095+0000 7f84cffff700 1 -- 172.21.15.203:0/2091454372 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f84dc018020 con 0x7f84e819dc10 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.095+0000 7f84cffff700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84d0038d60 0x7f84d003b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.095+0000 7f84cffff700 1 -- 172.21.15.203:0/2091454372 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f84dc04bdf0 con 0x7f84e819dc10 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.096+0000 7f84e6ffd700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84d0038d60 0x7f84d003b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:27:59.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.097+0000 7f84e6ffd700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84d0038d60 0x7f84d003b220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f84d800b5c0 tx=0x7f84d8009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:27:59.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.111+0000 7f84cffff700 1 -- 172.21.15.203:0/2091454372 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f84dc01f020 con 0x7f84e819dc10 2024-03-18T01:27:59.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.388+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f84e8198dc0 con 0x7f84e819dc10 2024-03-18T01:27:59.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.389+0000 7f84cffff700 1 -- 172.21.15.203:0/2091454372 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7f84dc010380 con 0x7f84e819dc10 2024-03-18T01:27:59.390 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:27:59.390 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","modified":"2024-03-18T01:27:50.830204Z","created":"2024-03-18T01:25:36.021301Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84d0038d60 msgr2=0x7f84d003b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84d0038d60 0x7f84d003b220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f84d800b5c0 tx=0x7f84d8009650 comp rx=0 tx=0).stop 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e819dc10 msgr2=0x7f84e81978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e819dc10 0x7f84e81978e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00b4d0 tx=0x7f84dc00b890 comp rx=0 tx=0).stop 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 shutdown_connections 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84d0038d60 0x7f84d003b220 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84e8102000 0x7f84e819d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 --2- 172.21.15.203:0/2091454372 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84e819dc10 0x7f84e81978e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 >> 172.21.15.203:0/2091454372 conn(0x7f84e80fd870 msgr2=0x7f84e81039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:27:59.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 shutdown_connections 2024-03-18T01:27:59.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:27:59.391+0000 7f84ed263700 1 -- 172.21.15.203:0/2091454372 wait complete. 2024-03-18T01:27:59.393 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-03-18T01:28:00.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:27:59 smithi046 systemd[1]: Started Ceph mon.b for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:28:00.946 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T01:28:00.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T01:28:01.190 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: mon.c calling monitor election 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: mon.a calling monitor election 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: monmap epoch 3 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: last_changed 2024-03-18T01:27:59.854559+0000 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: min_mon_release 19 (squid) 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: election_strategy: 1 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: fsmap 2024-03-18T01:28:05.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:28:05.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: mgrmap e13: x(active, since 67s) 2024-03-18T01:28:05.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T01:28:05.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T01:28:05.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T01:28:05.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: mon.b (rank 2) addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] is down (out of quorum) 2024-03-18T01:28:05.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:05.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: mon.c calling monitor election 2024-03-18T01:28:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: mon.a calling monitor election 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: monmap epoch 3 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:28:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: last_changed 2024-03-18T01:27:59.854559+0000 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: min_mon_release 19 (squid) 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: election_strategy: 1 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: fsmap 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: mgrmap e13: x(active, since 67s) 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: mon.b (rank 2) addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] is down (out of quorum) 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:05.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:05.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.621+0000 7f5bd41c4700 1 -- 172.21.15.203:0/185150366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bb80035c0 msgr2=0x7f5bb8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:05.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.621+0000 7f5bd41c4700 1 --2- 172.21.15.203:0/185150366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bb80035c0 0x7f5bb8005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc100c00 tx=0x7f5bc800feb0 comp rx=0 tx=0).stop 2024-03-18T01:28:05.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.621+0000 7f5bd41c4700 1 -- 172.21.15.203:0/185150366 shutdown_connections 2024-03-18T01:28:05.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.621+0000 7f5bd41c4700 1 --2- 172.21.15.203:0/185150366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bb80035c0 0x7f5bb8005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:05.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.621+0000 7f5bd41c4700 1 -- 172.21.15.203:0/185150366 >> 172.21.15.203:0/185150366 conn(0x7f5bcc0fc7e0 msgr2=0x7f5bcc0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:05.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.621+0000 7f5bd41c4700 1 -- 172.21.15.203:0/185150366 shutdown_connections 2024-03-18T01:28:05.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.621+0000 7f5bd41c4700 1 -- 172.21.15.203:0/185150366 wait complete. 2024-03-18T01:28:05.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.622+0000 7f5bd41c4700 1 Processor -- start 2024-03-18T01:28:05.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.622+0000 7f5bd41c4700 1 -- start start 2024-03-18T01:28:05.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd41c4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 0x7f5bcc19d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd41c4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 0x7f5bcc197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd41c4700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc197d80 0x7f5bcc198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd41c4700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc108d10 con 0x7f5bcc197d80 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd41c4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc108b90 con 0x7f5bb80035c0 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd41c4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc108e90 con 0x7f5bcc19dc00 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd175f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 0x7f5bcc197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd175f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 0x7f5bcc197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:42626/0 (socket says 172.21.15.203:42626) 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd175f700 1 -- 172.21.15.203:0/955088382 learned_addr learned my addr 172.21.15.203:0/955088382 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd175f700 1 -- 172.21.15.203:0/955088382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 msgr2=0x7f5bcc197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd175f700 1 -- 172.21.15.203:0/955088382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 msgr2=0x7f5bcc197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd175f700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 0x7f5bcc197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd1f60700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 0x7f5bcc19d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd175f700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 0x7f5bcc197840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:28:05.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd1f60700 1 -- 172.21.15.203:0/955088382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 msgr2=0x7f5bcc19d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd1f60700 1 -- 172.21.15.203:0/955088382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 msgr2=0x7f5bcc19d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd1f60700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 0x7f5bcc19d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd1f60700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 0x7f5bcc19d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.623+0000 7f5bd2761700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc197d80 0x7f5bcc198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bd2761700 1 -- 172.21.15.203:0/955088382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 msgr2=0x7f5bcc197840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bd2761700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 0x7f5bcc197840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bd2761700 1 -- 172.21.15.203:0/955088382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 msgr2=0x7f5bcc19d6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bd2761700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 0x7f5bcc19d6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bd2761700 1 -- 172.21.15.203:0/955088382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bcc1a4520 con 0x7f5bcc197d80 2024-03-18T01:28:05.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bd2761700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc197d80 0x7f5bcc198230 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400bd50 tx=0x7f5bc400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:05.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bc2ffd700 1 -- 172.21.15.203:0/955088382 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4019070 con 0x7f5bcc197d80 2024-03-18T01:28:05.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bcc1a47b0 con 0x7f5bcc197d80 2024-03-18T01:28:05.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bc2ffd700 1 -- 172.21.15.203:0/955088382 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc400cd10 con 0x7f5bcc197d80 2024-03-18T01:28:05.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bc2ffd700 1 -- 172.21.15.203:0/955088382 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc40127e0 con 0x7f5bcc197d80 2024-03-18T01:28:05.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.624+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bcc1a4cc0 con 0x7f5bcc197d80 2024-03-18T01:28:05.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.626+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bcc102410 con 0x7f5bcc197d80 2024-03-18T01:28:05.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.626+0000 7f5bc2ffd700 1 -- 172.21.15.203:0/955088382 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f5bc401c070 con 0x7f5bcc197d80 2024-03-18T01:28:05.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.626+0000 7f5bc2ffd700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb0039450 0x7f5bb003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:05.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.626+0000 7f5bc2ffd700 1 -- 172.21.15.203:0/955088382 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5bc404b900 con 0x7f5bcc197d80 2024-03-18T01:28:05.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.627+0000 7f5bd1f60700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb0039450 0x7f5bb003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:05.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.627+0000 7f5bd1f60700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb0039450 0x7f5bb003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8005b40 tx=0x7f5bc8005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:05.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.637+0000 7f5bc2ffd700 1 -- 172.21.15.203:0/955088382 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5bc4010ca0 con 0x7f5bcc197d80 2024-03-18T01:28:05.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.952+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5bcc02db70 con 0x7f5bcc197d80 2024-03-18T01:28:05.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.955+0000 7f5bc2ffd700 1 -- 172.21.15.203:0/955088382 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1275 (secure 0 0 0) 0x7f5bc4014440 con 0x7f5bcc197d80 2024-03-18T01:28:05.956 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-18T01:28:05.956 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","modified":"2024-03-18T01:27:59.854559Z","created":"2024-03-18T01:25:36.021301Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T01:28:05.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.957+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb0039450 msgr2=0x7f5bb003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:05.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.957+0000 7f5bd41c4700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb0039450 0x7f5bb003b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8005b40 tx=0x7f5bc8005a90 comp rx=0 tx=0).stop 2024-03-18T01:28:05.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.957+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc197d80 msgr2=0x7f5bcc198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:05.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.957+0000 7f5bd41c4700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc197d80 0x7f5bcc198230 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5bc400bd50 tx=0x7f5bc400be30 comp rx=0 tx=0).stop 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.958+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 shutdown_connections 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.958+0000 7f5bd41c4700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb0039450 0x7f5bb003b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.958+0000 7f5bd41c4700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc197d80 0x7f5bcc198230 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.958+0000 7f5bd41c4700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bb80035c0 0x7f5bcc19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.958+0000 7f5bd41c4700 1 --2- 172.21.15.203:0/955088382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc19dc00 0x7f5bcc197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.958+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 >> 172.21.15.203:0/955088382 conn(0x7f5bcc0fc7e0 msgr2=0x7f5bcc18ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.958+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 shutdown_connections 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:05.958+0000 7f5bd41c4700 1 -- 172.21.15.203:0/955088382 wait complete. 2024-03-18T01:28:05.959 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-03-18T01:28:06.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:06.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:06.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:05 smithi007 ceph-mon[31996]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:06.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:06.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:06.327 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:06.328 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:06.328 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:05 smithi203 ceph-mon[36345]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:06.328 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:06.328 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:07.055 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T01:28:07.055 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: mon.c calling monitor election 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: mon.a calling monitor election 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: monmap epoch 3 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: last_changed 2024-03-18T01:27:59.854559+0000 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: min_mon_release 19 (squid) 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: election_strategy: 1 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: fsmap 2024-03-18T01:28:07.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: mgrmap e13: x(active, since 67s) 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: mon.b (rank 2) addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] is down (out of quorum) 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:07.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:28:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: mon.b calling monitor election 2024-03-18T01:28:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-18T01:28:07.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: mon.b calling monitor election 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: mon.c calling monitor election 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: mon.a calling monitor election 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: monmap epoch 3 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: last_changed 2024-03-18T01:27:59.854559+0000 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: min_mon_release 19 (squid) 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: election_strategy: 1 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-03-18T01:28:07.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: fsmap 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: mgrmap e13: x(active, since 69s) 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: Cluster is now healthy 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: overall HEALTH_OK 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:07.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: mon.b calling monitor election 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: mon.b calling monitor election 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: mon.c calling monitor election 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: mon.a calling monitor election 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: monmap epoch 3 2024-03-18T01:28:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: last_changed 2024-03-18T01:27:59.854559+0000 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: min_mon_release 19 (squid) 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: election_strategy: 1 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: fsmap 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: mgrmap e13: x(active, since 69s) 2024-03-18T01:28:08.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T01:28:08.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: Cluster is now healthy 2024-03-18T01:28:08.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: overall HEALTH_OK 2024-03-18T01:28:08.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: mon.b calling monitor election 2024-03-18T01:28:08.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: Updating smithi007:/etc/ceph/ceph.conf 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: Updating smithi203:/etc/ceph/ceph.conf 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: mon.b calling monitor election 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: mon.c calling monitor election 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: mon.a calling monitor election 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: monmap epoch 3 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: last_changed 2024-03-18T01:27:59.854559+0000 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: created 2024-03-18T01:25:36.021301+0000 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: min_mon_release 19 (squid) 2024-03-18T01:28:08.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: election_strategy: 1 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: 2: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.b 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: fsmap 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: mgrmap e13: x(active, since 69s) 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: Cluster is now healthy 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: overall HEALTH_OK 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.980 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:28:08.855+0000 7faa09c00700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-18T01:28:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: Updating smithi046:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: Updating smithi203:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:08.980 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: Updating smithi007:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:08.981 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: Updating smithi046:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: Updating smithi203:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: Updating smithi007:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:09.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:09.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:09.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: Updating smithi046:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:09.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: Updating smithi203:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:09.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:09.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: Updating smithi007:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T01:28:09.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:09.046 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:09.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.866+0000 7f832d200700 1 -- 172.21.15.7:0/385729199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8328104100 msgr2=0x7f83281044e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.866+0000 7f832d200700 1 --2- 172.21.15.7:0/385729199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8328104100 0x7f83281044e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8318009910 tx=0x7f8318009c20 comp rx=0 tx=0).stop 2024-03-18T01:28:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.866+0000 7f832d200700 1 -- 172.21.15.7:0/385729199 shutdown_connections 2024-03-18T01:28:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.866+0000 7f832d200700 1 --2- 172.21.15.7:0/385729199 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8328104100 0x7f83281044e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.866+0000 7f832d200700 1 -- 172.21.15.7:0/385729199 >> 172.21.15.7:0/385729199 conn(0x7f83280ff970 msgr2=0x7f8328101d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.866+0000 7f832d200700 1 -- 172.21.15.7:0/385729199 shutdown_connections 2024-03-18T01:28:09.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.867+0000 7f832d200700 1 -- 172.21.15.7:0/385729199 wait complete. 2024-03-18T01:28:09.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:09 smithi007 ceph-mon[31996]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T01:28:09.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:09 smithi007 ceph-mon[31996]: Reconfiguring daemon mon.a on smithi007 2024-03-18T01:28:09.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f832d200700 1 Processor -- start 2024-03-18T01:28:09.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f832d200700 1 -- start start 2024-03-18T01:28:09.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f832d200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8328104100 0x7f832819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:09.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f832d200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f832819fdf0 0x7f8328199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f832d200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f832819a050 0x7f832819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f832d200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f832810e4d0 con 0x7f832819a050 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f832d200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f832810e350 con 0x7f8328104100 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f832d200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f832810e650 con 0x7f832819fdf0 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.868+0000 7f8325400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f832819fdf0 0x7f8328199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8325400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f832819fdf0 0x7f8328199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38606/0 (socket says 172.21.15.7:38606) 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8325400700 1 -- 172.21.15.7:0/336715042 learned_addr learned my addr 172.21.15.7:0/336715042 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8325e00700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8328104100 0x7f832819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8326800700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f832819a050 0x7f832819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8325400700 1 -- 172.21.15.7:0/336715042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8328104100 msgr2=0x7f832819f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8325400700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8328104100 0x7f832819f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:09.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8325400700 1 -- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f832819a050 msgr2=0x7f832819a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:09.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8325400700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f832819a050 0x7f832819a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:09.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.869+0000 7f8325400700 1 -- 172.21.15.7:0/336715042 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83281a6780 con 0x7f832819fdf0 2024-03-18T01:28:09.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.870+0000 7f8325400700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f832819fdf0 0x7f8328199b10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f831400b780 tx=0x7f831400ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:09.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.870+0000 7f831ea00700 1 -- 172.21.15.7:0/336715042 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8314016070 con 0x7f832819fdf0 2024-03-18T01:28:09.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.870+0000 7f831ea00700 1 -- 172.21.15.7:0/336715042 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83140036a0 con 0x7f832819fdf0 2024-03-18T01:28:09.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.870+0000 7f831ea00700 1 -- 172.21.15.7:0/336715042 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8314011af0 con 0x7f832819fdf0 2024-03-18T01:28:09.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.870+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83281a69b0 con 0x7f832819fdf0 2024-03-18T01:28:09.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.870+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83281a6f40 con 0x7f832819fdf0 2024-03-18T01:28:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.871+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8328063900 con 0x7f832819fdf0 2024-03-18T01:28:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.872+0000 7f831ea00700 1 -- 172.21.15.7:0/336715042 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f8314003850 con 0x7f832819fdf0 2024-03-18T01:28:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.872+0000 7f831ea00700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f830c039450 0x7f830c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.872+0000 7f831ea00700 1 -- 172.21.15.7:0/336715042 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f831404aee0 con 0x7f832819fdf0 2024-03-18T01:28:09.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.873+0000 7f8325e00700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f830c039450 0x7f830c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:09.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.873+0000 7f8325e00700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f830c039450 0x7f830c03b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8318005e00 tx=0x7f83180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:09.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:09.877+0000 7f831ea00700 1 -- 172.21.15.7:0/336715042 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f831404e030 con 0x7f832819fdf0 2024-03-18T01:28:10.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:09 smithi046 ceph-mon[36108]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T01:28:10.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:09 smithi046 ceph-mon[36108]: Reconfiguring daemon mon.a on smithi007 2024-03-18T01:28:10.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.115+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f83281a7260 con 0x7f832819fdf0 2024-03-18T01:28:10.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.115+0000 7f831ea00700 1 -- 172.21.15.7:0/336715042 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7f831400e260 con 0x7f832819fdf0 2024-03-18T01:28:10.116 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:28:10.116 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-03-18T01:28:10.116 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:28:10.116 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-03-18T01:28:10.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f830c039450 msgr2=0x7f830c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f830c039450 0x7f830c03b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8318005e00 tx=0x7f83180058e0 comp rx=0 tx=0).stop 2024-03-18T01:28:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f832819fdf0 msgr2=0x7f8328199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f832819fdf0 0x7f8328199b10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f831400b780 tx=0x7f831400ba90 comp rx=0 tx=0).stop 2024-03-18T01:28:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 shutdown_connections 2024-03-18T01:28:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f830c039450 0x7f830c03b910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f832819a050 0x7f832819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8328104100 0x7f832819f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:10.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 --2- 172.21.15.7:0/336715042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f832819fdf0 0x7f8328199b10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 >> 172.21.15.7:0/336715042 conn(0x7f83280ff970 msgr2=0x7f8328105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.118+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 shutdown_connections 2024-03-18T01:28:10.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:10.119+0000 7f832d200700 1 -- 172.21.15.7:0/336715042 wait complete. 2024-03-18T01:28:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:09 smithi203 ceph-mon[36345]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T01:28:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:09 smithi203 ceph-mon[36345]: Reconfiguring daemon mon.a on smithi007 2024-03-18T01:28:10.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:10 smithi007 ceph-mon[31996]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:10.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/336715042' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:10.977 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T01:28:10.978 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:28:10.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T01:28:11.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:10 smithi046 ceph-mon[36108]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:11.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:10 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/336715042' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:11.078 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:28:11.078 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:28:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:10 smithi203 ceph-mon[36345]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:10 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/336715042' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:11.173 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:28:11.173 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T01:28:11.212 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:28:11.212 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:28:11.288 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:28:11.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T01:28:11.323 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:28:11.324 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T01:28:11.393 INFO:tasks.cephadm:Adding mgr.x on smithi007 2024-03-18T01:28:11.394 INFO:tasks.cephadm:Adding mgr.y on smithi046 2024-03-18T01:28:11.394 INFO:tasks.cephadm:Adding mgr.z on smithi203 2024-03-18T01:28:11.394 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi007=x;smithi046=y;smithi203=z' 2024-03-18T01:28:11.669 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:28:12.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:12 smithi046 ceph-mon[36108]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:12 smithi046 ceph-mon[36108]: Reconfiguring mon.b (monmap changed)... 2024-03-18T01:28:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:12 smithi046 ceph-mon[36108]: Reconfiguring daemon mon.b on smithi046 2024-03-18T01:28:12.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:12 smithi007 ceph-mon[31996]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:12.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:12.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:12.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:12 smithi007 ceph-mon[31996]: Reconfiguring mon.b (monmap changed)... 2024-03-18T01:28:12.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:12.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:12.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:12.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:12 smithi007 ceph-mon[31996]: Reconfiguring daemon mon.b on smithi046 2024-03-18T01:28:12.815 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:12 smithi203 ceph-mon[36345]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:12.815 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:12.815 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:12.815 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:12 smithi203 ceph-mon[36345]: Reconfiguring mon.b (monmap changed)... 2024-03-18T01:28:12.815 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:12.815 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:12.815 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:12.815 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:12 smithi203 ceph-mon[36345]: Reconfiguring daemon mon.b on smithi046 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.176+0000 7f29b779d700 1 -- 172.21.15.203:0/4209202113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0100d30 msgr2=0x7f29b0101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.176+0000 7f29b779d700 1 --2- 172.21.15.203:0/4209202113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0100d30 0x7f29b0101150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009910 tx=0x7f29ac009c20 comp rx=0 tx=0).stop 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.176+0000 7f29b779d700 1 -- 172.21.15.203:0/4209202113 shutdown_connections 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.176+0000 7f29b779d700 1 --2- 172.21.15.203:0/4209202113 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0100d30 0x7f29b0101150 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.176+0000 7f29b779d700 1 -- 172.21.15.203:0/4209202113 >> 172.21.15.203:0/4209202113 conn(0x7f29b00fc2b0 msgr2=0x7f29b00fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.176+0000 7f29b779d700 1 -- 172.21.15.203:0/4209202113 shutdown_connections 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.176+0000 7f29b779d700 1 -- 172.21.15.203:0/4209202113 wait complete. 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b779d700 1 Processor -- start 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b779d700 1 -- start start 2024-03-18T01:28:13.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b779d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29b0100d30 0x7f29b0197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b779d700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0197b80 0x7f29b019cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b779d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0198000 0x7f29b019f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b779d700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f29b0106d70 con 0x7f29b0197b80 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b779d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f29b0106bf0 con 0x7f29b0198000 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b779d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f29b0106ef0 con 0x7f29b0100d30 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b5539700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29b0100d30 0x7f29b0197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b5539700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29b0100d30 0x7f29b0197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:58444/0 (socket says 172.21.15.203:58444) 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b5539700 1 -- 172.21.15.203:0/3195761485 learned_addr learned my addr 172.21.15.203:0/3195761485 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b5d3a700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0198000 0x7f29b019f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b4d38700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0197b80 0x7f29b019cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.177+0000 7f29b5539700 1 -- 172.21.15.203:0/3195761485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0198000 msgr2=0x7f29b019f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29b5539700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0198000 0x7f29b019f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29b5539700 1 -- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0197b80 msgr2=0x7f29b019cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:13.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29b5539700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0197b80 0x7f29b019cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:13.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29b5539700 1 -- 172.21.15.203:0/3195761485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b019fa50 con 0x7f29b0100d30 2024-03-18T01:28:13.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29b4d38700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0197b80 0x7f29b019cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:28:13.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29b5539700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29b0100d30 0x7f29b0197640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f29ac000c00 tx=0x7f29ac005000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:13.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29a67fc700 1 -- 172.21.15.203:0/3195761485 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac027070 con 0x7f29b0100d30 2024-03-18T01:28:13.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b019fce0 con 0x7f29b0100d30 2024-03-18T01:28:13.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29a67fc700 1 -- 172.21.15.203:0/3195761485 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29ac022470 con 0x7f29b0100d30 2024-03-18T01:28:13.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29a67fc700 1 -- 172.21.15.203:0/3195761485 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac00f680 con 0x7f29b0100d30 2024-03-18T01:28:13.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.178+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b01a01c0 con 0x7f29b0100d30 2024-03-18T01:28:13.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.179+0000 7f29a67fc700 1 -- 172.21.15.203:0/3195761485 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f29ac02a070 con 0x7f29b0100d30 2024-03-18T01:28:13.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.179+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b0063900 con 0x7f29b0100d30 2024-03-18T01:28:13.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.179+0000 7f29a67fc700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f299c039450 0x7f299c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:13.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.179+0000 7f29a67fc700 1 -- 172.21.15.203:0/3195761485 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f29ac04d4e0 con 0x7f29b0100d30 2024-03-18T01:28:13.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.180+0000 7f29b4d38700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f299c039450 0x7f299c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:13.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.181+0000 7f29b4d38700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f299c039450 0x7f299c03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009910 tx=0x7f29a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:13.184 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.184+0000 7f29a67fc700 1 -- 172.21.15.203:0/3195761485 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f29ac013070 con 0x7f29b0100d30 2024-03-18T01:28:13.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.397+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi046=y;smithi203=z", "target": ["mon-mgr", ""]}) -- 0x7f29b00627a0 con 0x7f299c039450 2024-03-18T01:28:13.410 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.411+0000 7f29a67fc700 1 -- 172.21.15.203:0/3195761485 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f29b00627a0 con 0x7f299c039450 2024-03-18T01:28:13.410 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-03-18T01:28:13.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f299c039450 msgr2=0x7f299c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:13.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f299c039450 0x7f299c03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009910 tx=0x7f29a0008040 comp rx=0 tx=0).stop 2024-03-18T01:28:13.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29b0100d30 msgr2=0x7f29b0197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:13.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29b0100d30 0x7f29b0197640 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f29ac000c00 tx=0x7f29ac005000 comp rx=0 tx=0).stop 2024-03-18T01:28:13.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 shutdown_connections 2024-03-18T01:28:13.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f299c039450 0x7f299c03b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:13.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29b0198000 0x7f29b019f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:13.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29b0100d30 0x7f29b0197640 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:13.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 --2- 172.21.15.203:0/3195761485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f29b0197b80 0x7f29b019cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:13.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 >> 172.21.15.203:0/3195761485 conn(0x7f29b00fc2b0 msgr2=0x7f29b006a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:13.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 shutdown_connections 2024-03-18T01:28:13.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:28:13.413+0000 7f29b779d700 1 -- 172.21.15.203:0/3195761485 wait complete. 2024-03-18T01:28:14.057 DEBUG:teuthology.orchestra.run.smithi046:mgr.y> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.y.service 2024-03-18T01:28:14.060 DEBUG:teuthology.orchestra.run.smithi203:mgr.z> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.z.service 2024-03-18T01:28:14.062 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T01:28:14.062 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:28:14.063 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T01:28:14.122 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T01:28:14.123 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-03-18T01:28:14.176 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T01:28:14.176 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:14.176 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-18T01:28:14.177 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:14.177 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:14.177 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-18 01:27:13.048354953 +0000 2024-03-18T01:28:14.177 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-18 01:20:42.563637326 +0000 2024-03-18T01:28:14.177 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-18 01:20:42.563637326 +0000 2024-03-18T01:28:14.177 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-18 01:20:42.563637326 +0000 2024-03-18T01:28:14.177 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T01:28:14.263 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-18T01:28:14.264 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-18T01:28:14.264 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000272686 s, 1.9 MB/s 2024-03-18T01:28:14.268 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T01:28:14.326 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-03-18T01:28:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T01:28:14.377 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:14.377 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-18T01:28:14.377 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:14.378 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:14.378 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-18 01:27:13.142353551 +0000 2024-03-18T01:28:14.378 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-18 01:20:43.150628677 +0000 2024-03-18T01:28:14.378 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-18 01:20:43.150628677 +0000 2024-03-18T01:28:14.378 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-18 01:20:43.150628677 +0000 2024-03-18T01:28:14.378 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T01:28:14.384 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:14.384 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.384 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.384 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: Reconfiguring mon.c (monmap changed)... 2024-03-18T01:28:14.384 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:14.384 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:14.384 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:14.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: Reconfiguring daemon mon.c on smithi203 2024-03-18T01:28:14.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi046=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:14.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: Saving service mgr spec with placement smithi007=x;smithi046=y;smithi203=z;count:3 2024-03-18T01:28:14.385 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:14 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.457 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:14.457 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.457 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.457 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: Reconfiguring mon.c (monmap changed)... 2024-03-18T01:28:14.457 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:14.457 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:14.457 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:14.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: Reconfiguring daemon mon.c on smithi203 2024-03-18T01:28:14.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi046=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:14.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: Saving service mgr spec with placement smithi007=x;smithi046=y;smithi203=z;count:3 2024-03-18T01:28:14.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:14 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-18T01:28:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-18T01:28:14.466 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000236878 s, 2.2 MB/s 2024-03-18T01:28:14.471 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T01:28:14.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: Reconfiguring mon.c (monmap changed)... 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: Reconfiguring daemon mon.c on smithi203 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi007=x;smithi046=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: Saving service mgr spec with placement smithi007=x;smithi046=y;smithi203=z;count:3 2024-03-18T01:28:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:14 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:14.526 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-03-18T01:28:14.572 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T01:28:14.573 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:14.573 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-03-18T01:28:14.573 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:14.573 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:14.573 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-18 01:27:13.218352416 +0000 2024-03-18T01:28:14.573 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-18 01:20:43.695620647 +0000 2024-03-18T01:28:14.573 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-18 01:20:43.695620647 +0000 2024-03-18T01:28:14.573 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-18 01:20:43.695620647 +0000 2024-03-18T01:28:14.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T01:28:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-18T01:28:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-18T01:28:14.715 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000281382 s, 1.8 MB/s 2024-03-18T01:28:14.719 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T01:28:14.781 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-03-18T01:28:14.834 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T01:28:14.834 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:14.834 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-03-18T01:28:14.835 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:14.835 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:14.835 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-18 01:27:13.296351252 +0000 2024-03-18T01:28:14.835 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-18 01:20:44.261612308 +0000 2024-03-18T01:28:14.835 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-18 01:20:44.261612308 +0000 2024-03-18T01:28:14.835 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-18 01:20:44.261612308 +0000 2024-03-18T01:28:14.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T01:28:14.921 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-18T01:28:14.921 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-18T01:28:14.921 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000278387 s, 1.8 MB/s 2024-03-18T01:28:14.924 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T01:28:14.982 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-18T01:28:14.982 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T01:28:15.002 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T01:28:15.002 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 794 Links: 1 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-18 01:27:59.318807792 +0000 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-18 01:19:55.013183226 +0000 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-18 01:19:55.013183226 +0000 2024-03-18T01:28:15.060 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-18 01:19:55.013183226 +0000 2024-03-18T01:28:15.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T01:28:15.135 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-18T01:28:15.135 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-18T01:28:15.135 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000125806 s, 4.1 MB/s 2024-03-18T01:28:15.136 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T01:28:15.195 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-03-18T01:28:15.252 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T01:28:15.252 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:15.253 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 815 Links: 1 2024-03-18T01:28:15.253 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:15.253 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:15.253 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-18 01:27:59.318807792 +0000 2024-03-18T01:28:15.253 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-18 01:19:55.361180135 +0000 2024-03-18T01:28:15.253 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-18 01:19:55.361180135 +0000 2024-03-18T01:28:15.253 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-18 01:19:55.361180135 +0000 2024-03-18T01:28:15.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T01:28:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-18T01:28:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-18T01:28:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.00011536 s, 4.4 MB/s 2024-03-18T01:28:15.327 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T01:28:15.385 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-03-18T01:28:15.442 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T01:28:15.442 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:15.442 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 831 Links: 1 2024-03-18T01:28:15.442 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:15.442 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:15.442 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-18 01:27:59.319807783 +0000 2024-03-18T01:28:15.443 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-18 01:19:55.721176937 +0000 2024-03-18T01:28:15.443 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-18 01:19:55.721176937 +0000 2024-03-18T01:28:15.443 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-18 01:19:55.721176937 +0000 2024-03-18T01:28:15.443 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T01:28:15.514 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-18T01:28:15.514 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-18T01:28:15.514 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000124126 s, 4.1 MB/s 2024-03-18T01:28:15.515 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T01:28:15.573 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-03-18T01:28:15.629 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T01:28:15.629 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:15.629 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 853 Links: 1 2024-03-18T01:28:15.629 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:15.629 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:15.630 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-18 01:27:59.319807783 +0000 2024-03-18T01:28:15.630 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-18 01:19:56.068173854 +0000 2024-03-18T01:28:15.630 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-18 01:19:56.068173854 +0000 2024-03-18T01:28:15.630 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-18 01:19:56.068173854 +0000 2024-03-18T01:28:15.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T01:28:15.699 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-18T01:28:15.699 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-18T01:28:15.699 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000113247 s, 4.5 MB/s 2024-03-18T01:28:15.701 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T01:28:15.758 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-18T01:28:15.758 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T01:28:15.773 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T01:28:15.773 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-18 01:27:49.815655351 +0000 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-18 01:19:44.376734800 +0000 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-18 01:19:44.376734800 +0000 2024-03-18T01:28:15.828 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-18 01:19:44.376734800 +0000 2024-03-18T01:28:15.828 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T01:28:15.894 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-18T01:28:15.894 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-18T01:28:15.894 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.00019068 s, 2.7 MB/s 2024-03-18T01:28:15.896 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T01:28:15.952 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-18 01:27:49.815655351 +0000 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-18 01:19:44.715726005 +0000 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-18 01:19:44.715726005 +0000 2024-03-18T01:28:16.008 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-18 01:19:44.715726005 +0000 2024-03-18T01:28:16.009 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T01:28:16.074 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-18T01:28:16.074 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-18T01:28:16.074 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000213192 s, 2.4 MB/s 2024-03-18T01:28:16.075 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T01:28:16.131 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-03-18T01:28:16.185 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T01:28:16.185 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:16.185 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-18T01:28:16.186 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:16.186 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:16.186 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-18 01:27:49.816655325 +0000 2024-03-18T01:28:16.186 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-18 01:19:45.046717417 +0000 2024-03-18T01:28:16.186 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-18 01:19:45.046717417 +0000 2024-03-18T01:28:16.186 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-18 01:19:45.046717417 +0000 2024-03-18T01:28:16.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T01:28:16.251 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-18T01:28:16.252 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-18T01:28:16.252 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000241713 s, 2.1 MB/s 2024-03-18T01:28:16.253 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T01:28:16.308 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-18 01:27:49.816655325 +0000 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-18 01:19:45.386708597 +0000 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-18 01:19:45.386708597 +0000 2024-03-18T01:28:16.363 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-18 01:19:45.386708597 +0000 2024-03-18T01:28:16.364 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T01:28:16.426 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:16.426 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.426 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.426 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:16.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:16.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:16.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:16.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T01:28:16.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:16.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:16.427 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:16 smithi203 ceph-mon[36345]: Deploying daemon mgr.z on smithi203 2024-03-18T01:28:16.429 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-18T01:28:16.429 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-18T01:28:16.429 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000189266 s, 2.7 MB/s 2024-03-18T01:28:16.430 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T01:28:16.485 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/vg_nvme/lv_4... 2024-03-18T01:28:16.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T01:28:16.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:16.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:16 smithi046 ceph-mon[36108]: Deploying daemon mgr.z on smithi203 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T01:28:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:16.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:16.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:16 smithi007 ceph-mon[31996]: Deploying daemon mgr.z on smithi203 2024-03-18T01:28:16.945 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:28:17.391 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:17 smithi203 systemd[1]: Starting Ceph mgr.z for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:28:17.691 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:17 smithi203 podman[38195]: 2024-03-18 01:28:17.422002129 +0000 UTC m=+0.113642993 container create 94ad95bb8a37c6f6356fa41f052828780b4b61637ca2cd8cc4791eec03128b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T01:28:17.691 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:17 smithi203 podman[38195]: 2024-03-18 01:28:17.325668283 +0000 UTC m=+0.017309115 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:28:18.141 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:17 smithi203 podman[38195]: 2024-03-18 01:28:17.775569059 +0000 UTC m=+0.467209884 container init 94ad95bb8a37c6f6356fa41f052828780b4b61637ca2cd8cc4791eec03128b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T01:28:18.141 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:17 smithi203 podman[38195]: 2024-03-18 01:28:17.778204533 +0000 UTC m=+0.469845360 container start 94ad95bb8a37c6f6356fa41f052828780b4b61637ca2cd8cc4791eec03128b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T01:28:18.141 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:17 smithi203 ceph-mgr[38214]: -- 172.21.15.203:0/2289519449 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5555616e25a0 con 0x5555616b6c00 2024-03-18T01:28:18.141 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:17 smithi203 bash[38195]: 94ad95bb8a37c6f6356fa41f052828780b4b61637ca2cd8cc4791eec03128b88 2024-03-18T01:28:18.142 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:17 smithi203 systemd[1]: Started Ceph mgr.z for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:28:18.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:18 smithi007 ceph-mon[31996]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:18.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.436 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.436 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:18.436 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T01:28:18.436 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:18.436 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:18.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:18 smithi046 ceph-mon[36108]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T01:28:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-mon[36345]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:18.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T01:28:18.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:18.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:18.642 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:18.321+0000 7fc868eb1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T01:28:18.642 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:18 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:18.413+0000 7fc868eb1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T01:28:19.380 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-mon[36345]: Deploying daemon mgr.y on smithi046 2024-03-18T01:28:19.380 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:19.052+0000 7fc868eb1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T01:28:19.380 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:19.143+0000 7fc868eb1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T01:28:19.380 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:19.285+0000 7fc868eb1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T01:28:19.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:19 smithi046 ceph-mon[36108]: Deploying daemon mgr.y on smithi046 2024-03-18T01:28:19.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:19 smithi007 ceph-mon[31996]: Deploying daemon mgr.y on smithi046 2024-03-18T01:28:19.641 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:19.379+0000 7fc868eb1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T01:28:19.641 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:19.461+0000 7fc868eb1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T01:28:19.754 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T01:28:19.754 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T01:28:19.755 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-03-18T01:28:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-03-18T01:28:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0378078 s, 277 MB/s 2024-03-18T01:28:19.755 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-03-18T01:28:20.026 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:19.725+0000 7fc868eb1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T01:28:20.026 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:19.840+0000 7fc868eb1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T01:28:20.026 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:19.928+0000 7fc868eb1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T01:28:20.027 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:20 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:20.025+0000 7fc868eb1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T01:28:20.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:20 smithi203 ceph-mon[36345]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:20.391 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:20 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:20.204+0000 7fc868eb1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T01:28:20.467 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:20 smithi046 ceph-mon[36108]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:20.468 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:20 smithi046 podman[37489]: 2024-03-18 01:28:20.44312084 +0000 UTC m=+0.332513696 container start 532e3a82d13e75bc91874cc0cee488c04c6e431e7b1862e3b9a49bb35fa77815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:28:20.468 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:20 smithi046 ceph-mgr[37508]: -- 172.21.15.46:0/3799945625 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b99cbc5a0 con 0x559b99c90c00 2024-03-18T01:28:20.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:20 smithi007 ceph-mon[31996]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:20.701 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_4 2024-03-18T01:28:20.768 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:20 smithi046 bash[37489]: 532e3a82d13e75bc91874cc0cee488c04c6e431e7b1862e3b9a49bb35fa77815 2024-03-18T01:28:20.768 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:20 smithi046 systemd[1]: Started Ceph mgr.y for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:28:21.129 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:28:21.268 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:21.012+0000 7f30f60af200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T01:28:21.268 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:21.111+0000 7f30f60af200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T01:28:21.573 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:21.203+0000 7fc868eb1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T01:28:21.573 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:21.338+0000 7fc868eb1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T01:28:21.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:21 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:21 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:21 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:21 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:21 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:21.828 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.828 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.829 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.829 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.829 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:21.838 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.838 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.838 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.839 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:21.839 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:21.839 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:21.700+0000 7fc868eb1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T01:28:21.839 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:21.792+0000 7fc868eb1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T01:28:22.141 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:21 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:21.991+0000 7fc868eb1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:28:22.193 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:21.827+0000 7f30f60af200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T01:28:22.193 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:21 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:21.926+0000 7f30f60af200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T01:28:22.193 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:22.073+0000 7f30f60af200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T01:28:22.193 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:22.192+0000 7f30f60af200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T01:28:22.464 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:22.175+0000 7fc868eb1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T01:28:22.518 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:22.283+0000 7f30f60af200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:22.463+0000 7fc868eb1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:22.729+0000 7fc868eb1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:22.731 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:22.732 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.732 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:22.732 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:22.732 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:22.785 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:22.785 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.785 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.785 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.785 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:22.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:22.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:22.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:22.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:22 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:22.557+0000 7f30f60af200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:22.670+0000 7f30f60af200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:22.758+0000 7f30f60af200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:22.856+0000 7f30f60af200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.857 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T01:28:22.858 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:22 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:23.141 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:22 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:22.964+0000 7fc868eb1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T01:28:23.268 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:23 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:23.038+0000 7f30f60af200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 -- 172.21.15.7:0/1765168192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc103520 msgr2=0x7f5abc103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1765168192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc103520 0x7f5abc103980 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80099c0 tx=0x7f5aa8009cd0 comp rx=0 tx=0).stop 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 -- 172.21.15.7:0/1765168192 shutdown_connections 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1765168192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5abc103ec0 0x7f5abc10a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1765168192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5abc102320 0x7f5abc102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1765168192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc103520 0x7f5abc103980 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 -- 172.21.15.7:0/1765168192 >> 172.21.15.7:0/1765168192 conn(0x7f5abc0fd880 msgr2=0x7f5abc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 -- 172.21.15.7:0/1765168192 shutdown_connections 2024-03-18T01:28:23.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.394+0000 7f5ac2600700 1 -- 172.21.15.7:0/1765168192 wait complete. 2024-03-18T01:28:23.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.395+0000 7f5ac2600700 1 Processor -- start 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5ac2600700 1 -- start start 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5ac2600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc102320 0x7f5abc076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5ac2600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5abc103520 0x7f5abc0746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5ac2600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5abc103ec0 0x7f5abc074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5ac2600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5abc10df30 con 0x7f5abc102320 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5ac2600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5abc10ddb0 con 0x7f5abc103ec0 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5ac2600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5abc10e0b0 con 0x7f5abc103520 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5abb400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc102320 0x7f5abc076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5abb400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc102320 0x7f5abc076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:38522/0 (socket says 172.21.15.7:38522) 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5abaa00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5abc103520 0x7f5abc0746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.396+0000 7f5abb400700 1 -- 172.21.15.7:0/1334385771 learned_addr learned my addr 172.21.15.7:0/1334385771 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:28:23.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abbe00700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5abc103ec0 0x7f5abc074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:23.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abb400700 1 -- 172.21.15.7:0/1334385771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5abc103520 msgr2=0x7f5abc0746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:23.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abb400700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5abc103520 0x7f5abc0746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:23.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abb400700 1 -- 172.21.15.7:0/1334385771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5abc103ec0 msgr2=0x7f5abc074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:23.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abb400700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5abc103ec0 0x7f5abc074c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abb400700 1 -- 172.21.15.7:0/1334385771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5abc0754b0 con 0x7f5abc102320 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abaa00700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5abc103520 0x7f5abc0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abbe00700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5abc103ec0 0x7f5abc074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5abb400700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc102320 0x7f5abc076070 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00ea50 tx=0x7f5aac00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.397+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac00cbd0 con 0x7f5abc102320 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.398+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5aac00cd70 con 0x7f5abc102320 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.398+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5abc1a2700 con 0x7f5abc102320 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.398+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac0187c0 con 0x7f5abc102320 2024-03-18T01:28:23.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.398+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5abc1a2c10 con 0x7f5abc102320 2024-03-18T01:28:23.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.399+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5abc068f90 con 0x7f5abc102320 2024-03-18T01:28:23.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.399+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 13) ==== 46161+0+0 (secure 0 0 0) 0x7f5aac01f020 con 0x7f5abc102320 2024-03-18T01:28:23.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.400+0000 7f5ab3e00700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aa0039200 0x7f5aa003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:23.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.400+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5aac04c2a0 con 0x7f5abc102320 2024-03-18T01:28:23.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.400+0000 7f5abaa00700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aa0039200 0x7f5aa003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:23.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.401+0000 7f5abaa00700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aa0039200 0x7f5aa003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8000c00 tx=0x7f5aa8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:23.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.406+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5aac01e2c0 con 0x7f5abc102320 2024-03-18T01:28:23.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:23.659+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5abc062710 con 0x7f5aa0039200 2024-03-18T01:28:23.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:23 smithi007 ceph-mon[31996]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T01:28:23.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:23 smithi007 ceph-mon[31996]: Reconfiguring daemon mgr.x on smithi007 2024-03-18T01:28:23.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:23 smithi203 ceph-mon[36345]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T01:28:23.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:23 smithi203 ceph-mon[36345]: Reconfiguring daemon mgr.x on smithi007 2024-03-18T01:28:23.891 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:23 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:23.764+0000 7fc868eb1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T01:28:24.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:23 smithi046 ceph-mon[36108]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T01:28:24.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:23 smithi046 ceph-mon[36108]: Reconfiguring daemon mgr.x on smithi007 2024-03-18T01:28:24.518 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:24.061+0000 7f30f60af200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T01:28:24.518 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:24.196+0000 7f30f60af200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T01:28:24.636 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:24 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:24.333+0000 7fc868eb1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T01:28:24.636 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:24 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:24.430+0000 7fc868eb1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T01:28:24.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:24 smithi007 ceph-mon[31996]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:24.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:24 smithi007 ceph-mon[31996]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:24.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:28:24.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:28:24.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:24.833 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-mon[36108]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:24.833 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-mon[36108]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:24.833 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:28:24.833 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:28:24.833 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:24.833 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:24.539+0000 7f30f60af200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T01:28:24.833 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:24.630+0000 7f30f60af200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T01:28:24.833 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:24 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:24.832+0000 7f30f60af200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:28:24.891 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:28:24 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z[38210]: 2024-03-18T01:28:24.645+0000 7fc868eb1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T01:28:24.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:24 smithi203 ceph-mon[36345]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:24.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:24 smithi203 ceph-mon[36345]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:24.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:28:24.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:28:24.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:25.268 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:25.022+0000 7f30f60af200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T01:28:25.583 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:25.314+0000 7f30f60af200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T01:28:25.583 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:25.583+0000 7f30f60af200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T01:28:25.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:25.675+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 14) ==== 91697+0+0 (secure 0 0 0) 0x7f5aac013190 con 0x7f5abc102320 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: Standby manager daemon z started 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:25.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:25 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:26.018 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:25.827+0000 7f30f60af200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: Standby manager daemon z started 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:26.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:25 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:26.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:26.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:26.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: Standby manager daemon z started 2024-03-18T01:28:26.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T01:28:26.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T01:28:26.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T01:28:26.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.? 172.21.15.203:0/858581169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T01:28:26.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:26.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:26.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:26.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:25 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:27.018 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:26 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:26.659+0000 7f30f60af200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T01:28:27.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:26 smithi046 ceph-mon[36108]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:27.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:26 smithi046 ceph-mon[36108]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-18T01:28:27.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T01:28:27.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:26 smithi007 ceph-mon[31996]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:27.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:26 smithi007 ceph-mon[31996]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-18T01:28:27.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T01:28:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:26 smithi203 ceph-mon[36345]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:26 smithi203 ceph-mon[36345]: mgrmap e14: x(active, since 88s), standbys: z 2024-03-18T01:28:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T01:28:27.518 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:27.249+0000 7f30f60af200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T01:28:27.518 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:27.349+0000 7f30f60af200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T01:28:27.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:27.685+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mon.0 v2:172.21.15.7:3300/0 8 ==== mgrmap(e 15) ==== 137233+0+0 (secure 0 0 0) 0x7f5aac00f3c0 con 0x7f5abc102320 2024-03-18T01:28:28.018 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y[37504]: 2024-03-18T01:28:27.571+0000 7f30f60af200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: Standby manager daemon y started 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3476220059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]: dispatch 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]: dispatch 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]': finished 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T01:28:28.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:28.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: Standby manager daemon y started 2024-03-18T01:28:28.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T01:28:28.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T01:28:28.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T01:28:28.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T01:28:28.026 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3476220059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]: dispatch 2024-03-18T01:28:28.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]: dispatch 2024-03-18T01:28:28.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]': finished 2024-03-18T01:28:28.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T01:28:28.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:28.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: Standby manager daemon y started 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: from='mgr.? 172.21.15.46:0/2747298645' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3476220059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]: dispatch 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]: dispatch 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae73eb8b-e60e-49d5-9fb1-9240be07f1d7"}]': finished 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T01:28:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:28.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:28 smithi007 ceph-mon[31996]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:28.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:28 smithi007 ceph-mon[31996]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-18T01:28:28.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T01:28:28.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:28 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/488877116' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:28:29.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:28 smithi046 ceph-mon[36108]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:29.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:28 smithi046 ceph-mon[36108]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-18T01:28:29.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T01:28:29.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:28 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/488877116' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:28:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:28 smithi203 ceph-mon[36345]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:28 smithi203 ceph-mon[36345]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-18T01:28:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T01:28:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:28 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/488877116' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:28:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:30 smithi203 ceph-mon[36345]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:31.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:30 smithi046 ceph-mon[36108]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:31.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:30 smithi007 ceph-mon[31996]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:33.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:32 smithi046 ceph-mon[36108]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:33.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:32 smithi007 ceph-mon[31996]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:32 smithi203 ceph-mon[36345]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:34.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:34 smithi007 ceph-mon[31996]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:35.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:34 smithi046 ceph-mon[36108]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:34 smithi203 ceph-mon[36345]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:36.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:36 smithi007 ceph-mon[31996]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:36.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T01:28:36.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:37.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:36 smithi046 ceph-mon[36108]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T01:28:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:36 smithi203 ceph-mon[36345]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T01:28:37.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:37.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:37 smithi007 ceph-mon[31996]: Deploying daemon osd.0 on smithi007 2024-03-18T01:28:38.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:37 smithi046 ceph-mon[36108]: Deploying daemon osd.0 on smithi007 2024-03-18T01:28:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:37 smithi203 ceph-mon[36345]: Deploying daemon osd.0 on smithi007 2024-03-18T01:28:39.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:38 smithi046 ceph-mon[36108]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:39.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:38 smithi007 ceph-mon[31996]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:39.041 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:38 smithi203 ceph-mon[36345]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:41.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:40 smithi046 ceph-mon[36108]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:41.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:40 smithi007 ceph-mon[31996]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:40 smithi203 ceph-mon[36345]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:43.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:42 smithi046 ceph-mon[36108]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:42 smithi007 ceph-mon[31996]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:42 smithi203 ceph-mon[36345]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:45.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:44 smithi046 ceph-mon[36108]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:45.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:45.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:45.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:45.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:44 smithi007 ceph-mon[31996]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:45.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:44 smithi203 ceph-mon[36345]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:47.022 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:46 smithi046 ceph-mon[36108]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:47.050 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:46 smithi007 ceph-mon[31996]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:46 smithi203 ceph-mon[36345]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:48.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:48 smithi007 ceph-mon[31996]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:48.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:48.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:48.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:48.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:48.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:48 smithi203 ceph-mon[36345]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:48.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:48 smithi046 ceph-mon[36108]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.254+0000 7f5ab3e00700 1 -- 172.21.15.7:0/1334385771 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5abc062710 con 0x7f5aa0039200 2024-03-18T01:28:49.255 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2024-03-18T01:28:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.257+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aa0039200 msgr2=0x7f5aa003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.257+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aa0039200 0x7f5aa003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8000c00 tx=0x7f5aa8005fb0 comp rx=0 tx=0).stop 2024-03-18T01:28:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.257+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc102320 msgr2=0x7f5abc076070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.257+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc102320 0x7f5abc076070 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00ea50 tx=0x7f5aac00ed60 comp rx=0 tx=0).stop 2024-03-18T01:28:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.258+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 shutdown_connections 2024-03-18T01:28:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.258+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aa0039200 0x7f5aa003b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.258+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5abc103ec0 0x7f5abc074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.258+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5abc102320 0x7f5abc076070 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.258+0000 7f5ac2600700 1 --2- 172.21.15.7:0/1334385771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5abc103520 0x7f5abc0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.258+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 >> 172.21.15.7:0/1334385771 conn(0x7f5abc0fd880 msgr2=0x7f5abc106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.258+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 shutdown_connections 2024-03-18T01:28:49.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:28:49.258+0000 7f5ac2600700 1 -- 172.21.15.7:0/1334385771 wait complete. 2024-03-18T01:28:49.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:49 smithi046 ceph-mon[36108]: from='osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T01:28:49.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:49 smithi046 ceph-mon[36108]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T01:28:49.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:49 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:49.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:49 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:49.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:49 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:49.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:49 smithi007 ceph-mon[31996]: from='osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T01:28:49.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:49 smithi007 ceph-mon[31996]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T01:28:49.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:49 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:49.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:49 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:49.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:49 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:49.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:49 smithi203 ceph-mon[36345]: from='osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T01:28:49.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:49 smithi203 ceph-mon[36345]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T01:28:49.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:49 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:49.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:49 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:49.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:49 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:28:50.092 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T01:28:50.097 INFO:tasks.cephadm:Deploying osd.1 on smithi046 with /dev/vg_nvme/lv_4... 2024-03-18T01:28:50.098 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T01:28:50.341 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:28:50.572 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:50 smithi046 ceph-mon[36108]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:50.572 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:50 smithi046 ceph-mon[36108]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T01:28:50.572 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:50 smithi046 ceph-mon[36108]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T01:28:50.573 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:50 smithi046 ceph-mon[36108]: from='osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:28:50.573 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:50.573 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:50 smithi046 ceph-mon[36108]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:28:50.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:50 smithi007 ceph-mon[31996]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:50.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:50 smithi007 ceph-mon[31996]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T01:28:50.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:50 smithi007 ceph-mon[31996]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T01:28:50.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:50 smithi007 ceph-mon[31996]: from='osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:28:50.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:50.633 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:50 smithi007 ceph-mon[31996]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:28:50.633 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:28:50 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0[46887]: 2024-03-18T01:28:50.332+0000 7f4a31a00700 -1 osd.0 0 waiting for initial osdmap 2024-03-18T01:28:50.634 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:28:50 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0[46887]: 2024-03-18T01:28:50.344+0000 7f4a2c800700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:28:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:50 smithi203 ceph-mon[36345]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:50 smithi203 ceph-mon[36345]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T01:28:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:50 smithi203 ceph-mon[36345]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T01:28:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:50 smithi203 ceph-mon[36345]: from='osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:28:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:50 smithi203 ceph-mon[36345]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:28:51.363 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:51 smithi046 ceph-mon[36108]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:28:51.363 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:51 smithi046 ceph-mon[36108]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T01:28:51.363 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:51 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:51.363 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:51 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:51.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:51 smithi007 ceph-mon[31996]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:28:51.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:51 smithi007 ceph-mon[31996]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T01:28:51.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:51 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:51.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:51 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:51.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:51 smithi203 ceph-mon[36345]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:28:51.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:51 smithi203 ceph-mon[36345]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T01:28:51.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:51 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:51.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:51 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:52.201 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T01:28:52.202 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T01:28:52.202 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-18T01:28:52.202 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-18T01:28:52.202 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00852828 s, 1.2 GB/s 2024-03-18T01:28:52.202 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-18T01:28:52.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:52 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:28:52.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:52 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:28:52.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:52 smithi203 ceph-mon[36345]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:52.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:52.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:52 smithi203 ceph-mon[36345]: osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] boot 2024-03-18T01:28:52.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:52 smithi203 ceph-mon[36345]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T01:28:52.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:52.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:52 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:28:52.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:52 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:28:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:52 smithi046 ceph-mon[36108]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:52 smithi046 ceph-mon[36108]: osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] boot 2024-03-18T01:28:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:52 smithi046 ceph-mon[36108]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T01:28:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:52.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_4 2024-03-18T01:28:52.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:52 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:28:52.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:52 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:28:52.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:52 smithi007 ceph-mon[31996]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T01:28:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:52 smithi007 ceph-mon[31996]: osd.0 [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] boot 2024-03-18T01:28:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:52 smithi007 ceph-mon[31996]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T01:28:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T01:28:53.045 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:28:53.447 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:53 smithi046 ceph-mon[36108]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T01:28:53.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:53 smithi203 ceph-mon[36345]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T01:28:53.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:53 smithi007 ceph-mon[31996]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T01:28:54.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:54 smithi007 ceph-mon[31996]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:54.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 -- 172.21.15.46:0/828797154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144101200 msgr2=0x7f7144101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:54.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 --2- 172.21.15.46:0/828797154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144101200 0x7f7144101620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71340099c0 tx=0x7f7134009cd0 comp rx=0 tx=0).stop 2024-03-18T01:28:54.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 -- 172.21.15.46:0/828797154 shutdown_connections 2024-03-18T01:28:54.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 --2- 172.21.15.46:0/828797154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7144102da0 0x7f7144109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:54.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 --2- 172.21.15.46:0/828797154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144101200 0x7f7144101620 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 --2- 172.21.15.46:0/828797154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7144102400 0x7f7144102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 -- 172.21.15.46:0/828797154 >> 172.21.15.46:0/828797154 conn(0x7f71440fc7a0 msgr2=0x7f71440febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:28:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 -- 172.21.15.46:0/828797154 shutdown_connections 2024-03-18T01:28:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.572+0000 7f714a1ae700 1 -- 172.21.15.46:0/828797154 wait complete. 2024-03-18T01:28:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f714a1ae700 1 Processor -- start 2024-03-18T01:28:54.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f714a1ae700 1 -- start start 2024-03-18T01:28:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f714a1ae700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7144101200 0x7f7144198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f714a1ae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144102400 0x7f7144198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f714a1ae700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7144102da0 0x7f714419fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f714a1ae700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f714410ce70 con 0x7f7144101200 2024-03-18T01:28:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f714a1ae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f714410ccf0 con 0x7f7144102400 2024-03-18T01:28:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f714a1ae700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f714410cff0 con 0x7f7144102da0 2024-03-18T01:28:54.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f7142ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144102400 0x7f7144198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f7142ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144102400 0x7f7144198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57442/0 (socket says 172.21.15.46:57442) 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f7142ffd700 1 -- 172.21.15.46:0/2109996613 learned_addr learned my addr 172.21.15.46:0/2109996613 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.573+0000 7f7143fff700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7144102da0 0x7f714419fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f71437fe700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7144101200 0x7f7144198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7142ffd700 1 -- 172.21.15.46:0/2109996613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7144102da0 msgr2=0x7f714419fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7142ffd700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7144102da0 0x7f714419fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7142ffd700 1 -- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7144101200 msgr2=0x7f7144198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7142ffd700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7144101200 0x7f7144198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7142ffd700 1 -- 172.21.15.46:0/2109996613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71441a0260 con 0x7f7144102400 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7143fff700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7144102da0 0x7f714419fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f71437fe700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7144101200 0x7f7144198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7142ffd700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144102400 0x7f7144198b40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f712c00c970 tx=0x7f712c00cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:54.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7140ff9700 1 -- 172.21.15.46:0/2109996613 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c007aa0 con 0x7f7144102400 2024-03-18T01:28:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7140ff9700 1 -- 172.21.15.46:0/2109996613 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f712c004d10 con 0x7f7144102400 2024-03-18T01:28:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f7140ff9700 1 -- 172.21.15.46:0/2109996613 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c00f450 con 0x7f7144102400 2024-03-18T01:28:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.574+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71441a0550 con 0x7f7144102400 2024-03-18T01:28:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.575+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71441a0ac0 con 0x7f7144102400 2024-03-18T01:28:54.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.575+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7144068f90 con 0x7f7144102400 2024-03-18T01:28:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.576+0000 7f7140ff9700 1 -- 172.21.15.46:0/2109996613 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 15) ==== 137233+0+0 (secure 0 0 0) 0x7f712c004750 con 0x7f7144102400 2024-03-18T01:28:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.577+0000 7f7140ff9700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71300a2880 0x7f71300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:28:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.577+0000 7f7140ff9700 1 -- 172.21.15.46:0/2109996613 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f712c0ccd90 con 0x7f7144102400 2024-03-18T01:28:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.578+0000 7f71437fe700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71300a2880 0x7f71300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:28:54.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.579+0000 7f71437fe700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71300a2880 0x7f71300a4d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7134006010 tx=0x7f7134005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:28:54.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.582+0000 7f7140ff9700 1 -- 172.21.15.46:0/2109996613 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f712c0d0030 con 0x7f7144102400 2024-03-18T01:28:54.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:54 smithi203 ceph-mon[36345]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:54.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:54 smithi046 ceph-mon[36108]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:54.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:28:54.799+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f71441a0d80 con 0x7f71300a2880 2024-03-18T01:28:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi007 2024-03-18T01:28:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:28:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi007 to 17095M 2024-03-18T01:28:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:28:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:28:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi007 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi007 to 17095M 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:28:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:28:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi007 2024-03-18T01:28:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:28:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi007 to 17095M 2024-03-18T01:28:56.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:56.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:56.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:28:56.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:28:56.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:28:56.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:28:56.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:28:56.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:28:56.679 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:56 smithi046 ceph-mon[36108]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:56.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:56 smithi203 ceph-mon[36345]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:56.953 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:56 smithi007 ceph-mon[31996]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:57 smithi203 ceph-mon[36345]: from='client.? 172.21.15.46:0/936569326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]: dispatch 2024-03-18T01:28:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:57 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]: dispatch 2024-03-18T01:28:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:57 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]': finished 2024-03-18T01:28:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:57 smithi203 ceph-mon[36345]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T01:28:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:28:57.914 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:57 smithi046 ceph-mon[36108]: from='client.? 172.21.15.46:0/936569326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]: dispatch 2024-03-18T01:28:57.914 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:57 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]: dispatch 2024-03-18T01:28:57.914 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:57 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]': finished 2024-03-18T01:28:57.914 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:57 smithi046 ceph-mon[36108]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T01:28:57.914 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:28:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:57 smithi007 ceph-mon[31996]: from='client.? 172.21.15.46:0/936569326' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]: dispatch 2024-03-18T01:28:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:57 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]: dispatch 2024-03-18T01:28:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:57 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c"}]': finished 2024-03-18T01:28:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:57 smithi007 ceph-mon[31996]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T01:28:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:28:58.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:58 smithi203 ceph-mon[36345]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:58.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:28:58 smithi203 ceph-mon[36345]: from='client.? 172.21.15.46:0/3492306932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:28:59.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:58 smithi046 ceph-mon[36108]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:59.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:28:58 smithi046 ceph-mon[36108]: from='client.? 172.21.15.46:0/3492306932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:28:59.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:58 smithi007 ceph-mon[31996]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:28:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:28:58 smithi007 ceph-mon[31996]: from='client.? 172.21.15.46:0/3492306932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:29:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:00 smithi203 ceph-mon[36345]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:01.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:00 smithi046 ceph-mon[36108]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:01.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:00 smithi007 ceph-mon[31996]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:02.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:02 smithi203 ceph-mon[36345]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:03.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:02 smithi046 ceph-mon[36108]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:03.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:02 smithi007 ceph-mon[31996]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:04.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:04 smithi046 ceph-mon[36108]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:04.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T01:29:04.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:05.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:04 smithi007 ceph-mon[31996]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:05.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T01:29:05.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:04 smithi203 ceph-mon[36345]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T01:29:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:05.892 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:05 smithi046 ceph-mon[36108]: Deploying daemon osd.1 on smithi046 2024-03-18T01:29:06.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:05 smithi007 ceph-mon[31996]: Deploying daemon osd.1 on smithi046 2024-03-18T01:29:06.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:05 smithi203 ceph-mon[36345]: Deploying daemon osd.1 on smithi046 2024-03-18T01:29:06.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:06 smithi046 ceph-mon[36108]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:07.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:06 smithi007 ceph-mon[31996]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:07.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:06 smithi203 ceph-mon[36345]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:08.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:08 smithi046 ceph-mon[36108]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:09.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:08 smithi007 ceph-mon[31996]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:08 smithi203 ceph-mon[36345]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:10.827 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:10 smithi046 ceph-mon[36108]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:10.828 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:10.828 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:10.828 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:10.943 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:10 smithi203 ceph-mon[36345]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:10.944 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:10.944 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:10.944 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:11.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:10 smithi007 ceph-mon[31996]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:11.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:11.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:11.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:13.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:12 smithi046 ceph-mon[36108]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:13.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:12 smithi007 ceph-mon[31996]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:12 smithi203 ceph-mon[36345]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:13.695 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 1 on host 'smithi046' 2024-03-18T01:29:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.694+0000 7f7140ff9700 1 -- 172.21.15.46:0/2109996613 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f71441a0d80 con 0x7f71300a2880 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71300a2880 msgr2=0x7f71300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71300a2880 0x7f71300a4d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7134006010 tx=0x7f7134005c40 comp rx=0 tx=0).stop 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144102400 msgr2=0x7f7144198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144102400 0x7f7144198b40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f712c00c970 tx=0x7f712c00cd30 comp rx=0 tx=0).stop 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 shutdown_connections 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71300a2880 0x7f71300a4d40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7144102da0 0x7f714419fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7144101200 0x7f7144198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 --2- 172.21.15.46:0/2109996613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7144102400 0x7f7144198b40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 >> 172.21.15.46:0/2109996613 conn(0x7f71440fc7a0 msgr2=0x7f71440fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:29:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 shutdown_connections 2024-03-18T01:29:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:29:13.696+0000 7f714a1ae700 1 -- 172.21.15.46:0/2109996613 wait complete. 2024-03-18T01:29:13.986 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:13.987 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:13.987 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:13.987 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:13.987 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:13.987 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:13.987 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:13.987 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:14.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:14.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:14.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:14.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:14.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:14.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:14.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:14.294 DEBUG:teuthology.orchestra.run.smithi046:osd.1> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T01:29:14.296 INFO:tasks.cephadm:Deploying osd.2 on smithi203 with /dev/vg_nvme/lv_4... 2024-03-18T01:29:14.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T01:29:14.542 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:29:14.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:14 smithi203 ceph-mon[36345]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:14.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:14 smithi203 ceph-mon[36345]: from='osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T01:29:14.917 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:14 smithi203 ceph-mon[36345]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T01:29:15.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:14 smithi046 ceph-mon[36108]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:15.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:14 smithi046 ceph-mon[36108]: from='osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T01:29:15.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:14 smithi046 ceph-mon[36108]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T01:29:15.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:14 smithi007 ceph-mon[31996]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:15.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:14 smithi007 ceph-mon[31996]: from='osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T01:29:15.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:14 smithi007 ceph-mon[31996]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T01:29:15.921 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:15 smithi046 ceph-mon[36108]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T01:29:15.921 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:15 smithi046 ceph-mon[36108]: from='osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:29:15.921 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:15 smithi046 ceph-mon[36108]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T01:29:15.921 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:15 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:15.921 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:15 smithi046 ceph-mon[36108]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:29:15.921 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:15 smithi046 ceph-mon[36108]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:29:16.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:15 smithi203 ceph-mon[36345]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T01:29:16.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:15 smithi203 ceph-mon[36345]: from='osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:29:16.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:15 smithi203 ceph-mon[36345]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T01:29:16.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:15 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:16.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:15 smithi203 ceph-mon[36345]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:29:16.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:15 smithi203 ceph-mon[36345]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:29:16.235 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T01:29:16.235 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T01:29:16.235 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-18T01:29:16.235 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-18T01:29:16.235 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0165843 s, 632 MB/s 2024-03-18T01:29:16.235 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-18T01:29:16.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:15 smithi007 ceph-mon[31996]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T01:29:16.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:15 smithi007 ceph-mon[31996]: from='osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:29:16.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:15 smithi007 ceph-mon[31996]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T01:29:16.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:15 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:16.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:15 smithi007 ceph-mon[31996]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:29:16.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:15 smithi007 ceph-mon[31996]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:29:16.759 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:16 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1[40708]: 2024-03-18T01:29:16.383+0000 7faa9d110700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T01:29:16.759 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:16 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1[40708]: 2024-03-18T01:29:16.387+0000 7faa94efb700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:29:16.873 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-03-18T01:29:17.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:16 smithi046 ceph-mon[36108]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:17.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:16 smithi046 ceph-mon[36108]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T01:29:17.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:17.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:17.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:16 smithi046 ceph-mon[36108]: from='osd.1 ' entity='osd.1' 2024-03-18T01:29:17.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:17.109 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:29:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:16 smithi203 ceph-mon[36345]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:16 smithi203 ceph-mon[36345]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T01:29:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:16 smithi203 ceph-mon[36345]: from='osd.1 ' entity='osd.1' 2024-03-18T01:29:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:17.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:16 smithi007 ceph-mon[31996]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:17.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:16 smithi007 ceph-mon[31996]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T01:29:17.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:17.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:17.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:16 smithi007 ceph-mon[31996]: from='osd.1 ' entity='osd.1' 2024-03-18T01:29:17.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:18.116 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:29:18.116 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:29:18.116 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] boot 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:18.117 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:17 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] boot 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:18.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:17 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: osd.1 [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] boot 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:29:18.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:18.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:18.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:17 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:18.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 -- 172.21.15.203:0/720667873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d28103e70 msgr2=0x7f2d2810a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:18.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/720667873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d28103e70 0x7f2d2810a720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2d2000b0a0 tx=0x7f2d2000b3b0 comp rx=0 tx=0).stop 2024-03-18T01:29:18.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 -- 172.21.15.203:0/720667873 shutdown_connections 2024-03-18T01:29:18.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/720667873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d28103e70 0x7f2d2810a720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:18.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/720667873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d281022d0 0x7f2d281026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:18.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/720667873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d281034d0 0x7f2d28103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:18.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 -- 172.21.15.203:0/720667873 >> 172.21.15.203:0/720667873 conn(0x7f2d280fd8b0 msgr2=0x7f2d280ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:29:18.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 -- 172.21.15.203:0/720667873 shutdown_connections 2024-03-18T01:29:18.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.615+0000 7f2d2cb83700 1 -- 172.21.15.203:0/720667873 wait complete. 2024-03-18T01:29:18.618 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2cb83700 1 Processor -- start 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2cb83700 1 -- start start 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2cb83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d281022d0 0x7f2d281984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2cb83700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d281034d0 0x7f2d28198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2cb83700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d28103e70 0x7f2d2819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2cb83700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2810df30 con 0x7f2d28103e70 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2cb83700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2810ddb0 con 0x7f2d281022d0 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2cb83700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2810e0b0 con 0x7f2d281034d0 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d1ffff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d281034d0 0x7f2d28198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d1ffff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d281034d0 0x7f2d28198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:47940/0 (socket says 172.21.15.203:47940) 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d1ffff700 1 -- 172.21.15.203:0/2329025550 learned_addr learned my addr 172.21.15.203:0/2329025550 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d2659c700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d281022d0 0x7f2d281984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:29:18.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d1ffff700 1 -- 172.21.15.203:0/2329025550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d281022d0 msgr2=0x7f2d281984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d1ffff700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d281022d0 0x7f2d281984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d1ffff700 1 -- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d28103e70 msgr2=0x7f2d2819fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d1ffff700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d28103e70 0x7f2d2819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.616+0000 7f2d1ffff700 1 -- 172.21.15.203:0/2329025550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d281a0120 con 0x7f2d281034d0 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.617+0000 7f2d1ffff700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d281034d0 0x7f2d28198a00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000b560 tx=0x7f2d1000b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.617+0000 7f2d1f7fe700 1 -- 172.21.15.203:0/2329025550 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10015020 con 0x7f2d281034d0 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.617+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d281a0410 con 0x7f2d281034d0 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.617+0000 7f2d1f7fe700 1 -- 172.21.15.203:0/2329025550 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d10011d10 con 0x7f2d281034d0 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.617+0000 7f2d1f7fe700 1 -- 172.21.15.203:0/2329025550 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d10010e50 con 0x7f2d281034d0 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.617+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d281a09d0 con 0x7f2d281034d0 2024-03-18T01:29:18.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.618+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d28068f90 con 0x7f2d281034d0 2024-03-18T01:29:18.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.619+0000 7f2d1f7fe700 1 -- 172.21.15.203:0/2329025550 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) ==== 137233+0+0 (secure 0 0 0) 0x7f2d10018020 con 0x7f2d281034d0 2024-03-18T01:29:18.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.619+0000 7f2d1f7fe700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2d140a29a0 0x7f2d140a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:29:18.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.620+0000 7f2d1f7fe700 1 -- 172.21.15.203:0/2329025550 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f2d100cc980 con 0x7f2d281034d0 2024-03-18T01:29:18.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.620+0000 7f2d2659c700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2d140a29a0 0x7f2d140a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:29:18.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.621+0000 7f2d2659c700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2d140a29a0 0x7f2d140a4e60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d28103330 tx=0x7f2d18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:29:18.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.624+0000 7f2d1f7fe700 1 -- 172.21.15.203:0/2329025550 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d1001f020 con 0x7f2d281034d0 2024-03-18T01:29:18.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:18.829+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2d28062710 con 0x7f2d140a29a0 2024-03-18T01:29:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:18 smithi203 ceph-mon[36345]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:18 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi046 2024-03-18T01:29:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:18 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi046 to 17084M 2024-03-18T01:29:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:29:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:29:19.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:19.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:18 smithi046 ceph-mon[36108]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:19.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:18 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi046 2024-03-18T01:29:19.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:18 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi046 to 17084M 2024-03-18T01:29:19.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:29:19.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:29:19.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:19.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:18 smithi007 ceph-mon[31996]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T01:29:19.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:18 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi046 2024-03-18T01:29:19.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:18 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi046 to 17084M 2024-03-18T01:29:19.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:29:19.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:29:19.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:20.066 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:19 smithi203 ceph-mon[36345]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:29:20.067 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:19 smithi203 ceph-mon[36345]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T01:29:20.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:19 smithi046 ceph-mon[36108]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:29:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:19 smithi046 ceph-mon[36108]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T01:29:20.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:19 smithi007 ceph-mon[31996]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:29:20.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:19 smithi007 ceph-mon[31996]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T01:29:21.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:20 smithi203 ceph-mon[36345]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:21.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:20 smithi046 ceph-mon[36108]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:21.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:20 smithi007 ceph-mon[31996]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:22.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:21 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/2540140888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]: dispatch 2024-03-18T01:29:22.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:21 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]: dispatch 2024-03-18T01:29:22.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:21 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]': finished 2024-03-18T01:29:22.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:21 smithi203 ceph-mon[36345]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T01:29:22.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:21 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:22.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:21 smithi046 ceph-mon[36108]: from='client.? 172.21.15.203:0/2540140888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]: dispatch 2024-03-18T01:29:22.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:21 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]: dispatch 2024-03-18T01:29:22.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:21 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]': finished 2024-03-18T01:29:22.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:21 smithi046 ceph-mon[36108]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T01:29:22.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:21 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:22.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:21 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/2540140888' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]: dispatch 2024-03-18T01:29:22.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:21 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]: dispatch 2024-03-18T01:29:22.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:21 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32b52cb8-4fcc-4a3a-9e2a-69a8205f0844"}]': finished 2024-03-18T01:29:22.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:21 smithi007 ceph-mon[31996]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T01:29:22.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:21 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:23.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:22 smithi203 ceph-mon[36345]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:23.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:22 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/2088186665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:29:23.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:22 smithi046 ceph-mon[36108]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:23.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:22 smithi046 ceph-mon[36108]: from='client.? 172.21.15.203:0/2088186665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:29:23.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:22 smithi007 ceph-mon[31996]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:23.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:22 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/2088186665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:29:25.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:24 smithi046 ceph-mon[36108]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:25.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:24 smithi007 ceph-mon[31996]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:25.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:24 smithi203 ceph-mon[36345]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:27.153 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:26 smithi203 ceph-mon[36345]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:27.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:26 smithi046 ceph-mon[36108]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:27.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:26 smithi007 ceph-mon[31996]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:29.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:28 smithi007 ceph-mon[31996]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:29.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T01:29:29.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:29.178 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:28 smithi203 ceph-mon[36345]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:29.178 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T01:29:29.178 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:29.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:28 smithi046 ceph-mon[36108]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:29.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T01:29:29.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:30.070 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:29 smithi203 ceph-mon[36345]: Deploying daemon osd.2 on smithi203 2024-03-18T01:29:30.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:29 smithi046 ceph-mon[36108]: Deploying daemon osd.2 on smithi203 2024-03-18T01:29:30.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:29 smithi007 ceph-mon[31996]: Deploying daemon osd.2 on smithi203 2024-03-18T01:29:31.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:30 smithi046 ceph-mon[36108]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:31.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:30 smithi007 ceph-mon[31996]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:31.343 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:30 smithi203 ceph-mon[36345]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:31.994 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:31 smithi203 ceph-mon[36345]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:32.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:31 smithi046 ceph-mon[36108]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:32.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:31 smithi007 ceph-mon[31996]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:34.765 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:34 smithi203 ceph-mon[36345]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:34.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:34 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:34.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:34 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:34.766 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:34 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:34 smithi046 ceph-mon[36108]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:34 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:34 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:34 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:35.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:34 smithi007 ceph-mon[31996]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:34 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:34 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:34 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:36.777 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:36 smithi203 ceph-mon[36345]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:36.777 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:36.777 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:36.777 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:36.777 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:36.777 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:36.860 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 2 on host 'smithi203' 2024-03-18T01:29:36.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.859+0000 7f2d1f7fe700 1 -- 172.21.15.203:0/2329025550 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2d28062710 con 0x7f2d140a29a0 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2d140a29a0 msgr2=0x7f2d140a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2d140a29a0 0x7f2d140a4e60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d28103330 tx=0x7f2d18009260 comp rx=0 tx=0).stop 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d281034d0 msgr2=0x7f2d28198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d281034d0 0x7f2d28198a00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2d1000b560 tx=0x7f2d1000b870 comp rx=0 tx=0).stop 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 shutdown_connections 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2d140a29a0 0x7f2d140a4e60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d28103e70 0x7f2d2819fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d281022d0 0x7f2d281984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 --2- 172.21.15.203:0/2329025550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d281034d0 0x7f2d28198a00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:36.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 >> 172.21.15.203:0/2329025550 conn(0x7f2d280fd8b0 msgr2=0x7f2d28106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:29:36.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 shutdown_connections 2024-03-18T01:29:36.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:29:36.862+0000 7f2d2cb83700 1 -- 172.21.15.203:0/2329025550 wait complete. 2024-03-18T01:29:37.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:36 smithi046 ceph-mon[36108]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:37.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:37.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:36 smithi007 ceph-mon[31996]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:37.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:37.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:37.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:37.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:37.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:37.451 DEBUG:teuthology.orchestra.run.smithi203:osd.2> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T01:29:37.453 INFO:tasks.cephadm:Deploying osd.3 on smithi007 with /dev/vg_nvme/lv_3... 2024-03-18T01:29:37.453 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T01:29:37.742 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:37 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2[41472]: 2024-03-18T01:29:37.505+0000 7ff86b08a7c0 -1 osd.2 0 log_to_monitors true 2024-03-18T01:29:37.922 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:29:38.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:37 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:38.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:37 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:38.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:37 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:38.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:37 smithi007 ceph-mon[31996]: from='osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T01:29:38.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:37 smithi007 ceph-mon[31996]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T01:29:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:37 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:37 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:37 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:37 smithi203 ceph-mon[36345]: from='osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T01:29:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:37 smithi203 ceph-mon[36345]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T01:29:38.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:37 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:38.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:37 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:38.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:37 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:29:38.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:37 smithi046 ceph-mon[36108]: from='osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T01:29:38.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:37 smithi046 ceph-mon[36108]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T01:29:39.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:38 smithi007 ceph-mon[31996]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:39.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:38 smithi007 ceph-mon[31996]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T01:29:39.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:38 smithi007 ceph-mon[31996]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T01:29:39.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:39.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:38 smithi007 ceph-mon[31996]: from='osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:29:39.128 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:38 smithi007 ceph-mon[31996]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:29:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:38 smithi203 ceph-mon[36345]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:38 smithi203 ceph-mon[36345]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T01:29:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:38 smithi203 ceph-mon[36345]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T01:29:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:38 smithi203 ceph-mon[36345]: from='osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:29:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:38 smithi203 ceph-mon[36345]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:29:39.142 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:38 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2[41472]: 2024-03-18T01:29:38.857+0000 7ff8606ef700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T01:29:39.142 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:38 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2[41472]: 2024-03-18T01:29:38.861+0000 7ff85c4e2700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:29:39.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:38 smithi046 ceph-mon[36108]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:39.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:38 smithi046 ceph-mon[36108]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T01:29:39.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:38 smithi046 ceph-mon[36108]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T01:29:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:38 smithi046 ceph-mon[36108]: from='osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:29:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:38 smithi046 ceph-mon[36108]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:29:39.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:39 smithi203 ceph-mon[36345]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:29:39.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:39 smithi203 ceph-mon[36345]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T01:29:39.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:39 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:39.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:39 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:40.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:39 smithi007 ceph-mon[31996]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:29:40.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:39 smithi007 ceph-mon[31996]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T01:29:40.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:39 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:40.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:39 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:40.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:39 smithi046 ceph-mon[36108]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:29:40.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:39 smithi046 ceph-mon[36108]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T01:29:40.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:39 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:40.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:39 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T01:29:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T01:29:40.877 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-03-18T01:29:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-03-18T01:29:40.877 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0388684 s, 270 MB/s 2024-03-18T01:29:40.877 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-03-18T01:29:41.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] boot 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:41.024 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:41.024 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:29:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:29:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] boot 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:29:41.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: osd.2 [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] boot 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:29:41.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:41.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_3 2024-03-18T01:29:42.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:41 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi203 2024-03-18T01:29:42.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:41 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi203 to 17080M 2024-03-18T01:29:42.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:41 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:29:42.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:41 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T01:29:42.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:41 smithi007 ceph-mon[31996]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T01:29:42.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:41 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:29:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:41 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi203 2024-03-18T01:29:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:41 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi203 to 17080M 2024-03-18T01:29:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:41 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:29:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:41 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T01:29:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:41 smithi203 ceph-mon[36345]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T01:29:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:41 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:29:42.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:41 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi203 2024-03-18T01:29:42.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:41 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi203 to 17080M 2024-03-18T01:29:42.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:41 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:29:42.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:41 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T01:29:42.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:41 smithi046 ceph-mon[36108]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T01:29:42.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:41 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:29:42.284 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:29:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:42 smithi007 ceph-mon[31996]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:29:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:42 smithi007 ceph-mon[31996]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T01:29:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:42 smithi203 ceph-mon[36345]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:29:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:42 smithi203 ceph-mon[36345]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T01:29:43.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:42 smithi046 ceph-mon[36108]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:43.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:29:43.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:42 smithi046 ceph-mon[36108]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T01:29:44.052 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:29:43 smithi007 sudo[51126]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:29:44.052 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:29:43 smithi007 sudo[51126]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:44.052 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:29:43 smithi007 sudo[51126]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:44.052 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:29:43 smithi007 sudo[51126]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.283+0000 7f6877200700 1 -- 172.21.15.7:0/2696495088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6878105b30 msgr2=0x7f6878105f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.283+0000 7f6877200700 1 --2- 172.21.15.7:0/2696495088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6878105b30 0x7f6878105f50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6868009920 tx=0x7f6868009c30 comp rx=0 tx=0).stop 2024-03-18T01:29:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.284+0000 7f6877200700 1 -- 172.21.15.7:0/2696495088 shutdown_connections 2024-03-18T01:29:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.284+0000 7f6877200700 1 --2- 172.21.15.7:0/2696495088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6878104a10 0x7f687810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:44.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.284+0000 7f6877200700 1 --2- 172.21.15.7:0/2696495088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6878105b30 0x7f6878105f50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.284+0000 7f6877200700 1 --2- 172.21.15.7:0/2696495088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6878104070 0x7f68781044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.284+0000 7f6877200700 1 -- 172.21.15.7:0/2696495088 >> 172.21.15.7:0/2696495088 conn(0x7f68780ff9f0 msgr2=0x7f6878101e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:29:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.284+0000 7f6877200700 1 -- 172.21.15.7:0/2696495088 shutdown_connections 2024-03-18T01:29:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.284+0000 7f6877200700 1 -- 172.21.15.7:0/2696495088 wait complete. 2024-03-18T01:29:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.285+0000 7f6877200700 1 Processor -- start 2024-03-18T01:29:44.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.285+0000 7f6877200700 1 -- start start 2024-03-18T01:29:44.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6877200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6878104070 0x7f687819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:29:44.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6877200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6878104a10 0x7f687819ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:29:44.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6877200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6878105b30 0x7f68781a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:29:44.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6877200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6878110160 con 0x7f6878104070 2024-03-18T01:29:44.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6877200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f687810ffe0 con 0x7f6878105b30 2024-03-18T01:29:44.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6877200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f68781102e0 con 0x7f6878104a10 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6875e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6878104070 0x7f687819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6875e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6878104070 0x7f687819a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51252/0 (socket says 172.21.15.7:51252) 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6875400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6878104a10 0x7f687819ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6875e00700 1 -- 172.21.15.7:0/2693704342 learned_addr learned my addr 172.21.15.7:0/2693704342 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.286+0000 7f6876800700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6878105b30 0x7f68781a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f6875e00700 1 -- 172.21.15.7:0/2693704342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6878104a10 msgr2=0x7f687819ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f6875e00700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6878104a10 0x7f687819ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f6875e00700 1 -- 172.21.15.7:0/2693704342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6878105b30 msgr2=0x7f68781a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f6875e00700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6878105b30 0x7f68781a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f6875e00700 1 -- 172.21.15.7:0/2693704342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68781a2450 con 0x7f6878104070 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f6875e00700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6878104070 0x7f687819a840 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6868005250 tx=0x7f6868005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:29:44.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f686ea00700 1 -- 172.21.15.7:0/2693704342 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686800f930 con 0x7f6878104070 2024-03-18T01:29:44.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f686ea00700 1 -- 172.21.15.7:0/2693704342 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f686800ba40 con 0x7f6878104070 2024-03-18T01:29:44.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.287+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68781a26e0 con 0x7f6878104070 2024-03-18T01:29:44.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.288+0000 7f686ea00700 1 -- 172.21.15.7:0/2693704342 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868003b90 con 0x7f6878104070 2024-03-18T01:29:44.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.288+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68781a2ae0 con 0x7f6878104070 2024-03-18T01:29:44.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.289+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6878068f90 con 0x7f6878104070 2024-03-18T01:29:44.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.291+0000 7f686ea00700 1 -- 172.21.15.7:0/2693704342 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 15) ==== 137233+0+0 (secure 0 0 0) 0x7f68680054a0 con 0x7f6878104070 2024-03-18T01:29:44.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.293+0000 7f686ea00700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f685c0a2800 0x7f685c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:29:44.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.293+0000 7f686ea00700 1 -- 172.21.15.7:0/2693704342 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f68680ce6d0 con 0x7f6878104070 2024-03-18T01:29:44.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.293+0000 7f6875400700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f685c0a2800 0x7f685c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:29:44.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.294+0000 7f6875400700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f685c0a2800 0x7f685c0a4cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6864009c60 tx=0x7f6864009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:29:44.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.297+0000 7f686ea00700 1 -- 172.21.15.7:0/2693704342 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6868013030 con 0x7f6878104070 2024-03-18T01:29:44.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:44.539+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6878062710 con 0x7f685c0a2800 2024-03-18T01:29:44.802 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:29:44 smithi007 sudo[51155]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:29:44.802 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:29:44 smithi007 sudo[51155]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:44.802 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:29:44 smithi007 sudo[51155]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:44.802 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:29:44 smithi007 sudo[51155]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:44 smithi203 ceph-mon[36345]: pgmap v93: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:44 smithi203 ceph-mon[36345]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T01:29:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:29:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:29:45.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:45.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:44 smithi046 sudo[44429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:29:45.268 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:44 smithi046 sudo[44429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:45.269 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:44 smithi046 sudo[44429]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:45.269 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:44 smithi046 sudo[44429]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:44 smithi046 ceph-mon[36108]: pgmap v93: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:44 smithi046 ceph-mon[36108]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T01:29:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:29:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:29:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:45.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:44 smithi007 ceph-mon[31996]: pgmap v93: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:45.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:44 smithi007 ceph-mon[31996]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T01:29:45.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:45.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:29:45.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:29:45.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:45.768 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:45 smithi046 sudo[44432]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:29:45.769 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:45 smithi046 sudo[44432]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:45.769 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:45 smithi046 sudo[44432]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:45.769 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:29:45 smithi046 sudo[44432]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:45.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:29:45.898+0000 7f686ea00700 1 -- 172.21.15.7:0/2693704342 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f6868098b80 con 0x7f6878104070 2024-03-18T01:29:45.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:45 smithi203 ceph-mon[36345]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:29:45.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:45 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:45.941 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:45 smithi203 sudo[45540]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:29:45.942 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:45 smithi203 sudo[45540]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:45.942 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:45 smithi203 sudo[45540]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:45.942 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:45 smithi203 sudo[45540]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:46.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:45 smithi046 ceph-mon[36108]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:29:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:45 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:46.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:45 smithi007 ceph-mon[31996]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:29:46.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:45 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:46.391 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:45 smithi203 sudo[45543]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:29:46.391 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:45 smithi203 sudo[45543]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:46.391 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:45 smithi203 sudo[45543]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:46.391 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:29:45 smithi203 sudo[45543]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:46.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:46 smithi007 sudo[51302]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:29:46.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:46 smithi007 sudo[51302]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:46.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:46 smithi007 sudo[51302]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:46.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:46 smithi007 sudo[51302]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:47 smithi007 sudo[51326]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:29:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:47 smithi007 sudo[51326]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:47 smithi007 sudo[51326]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:47 smithi007 sudo[51326]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:47 smithi007 ceph-mon[31996]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:47 smithi007 ceph-mon[31996]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T01:29:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:47 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:47 smithi007 ceph-mon[31996]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 ceph-mon[36345]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 ceph-mon[36345]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T01:29:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 sudo[45546]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:29:47.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 sudo[45546]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:47.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 sudo[45546]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:47.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 sudo[45546]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:47.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:47 smithi046 ceph-mon[36108]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:47.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:47 smithi046 ceph-mon[36108]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T01:29:47.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:47 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:29:47.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:47 smithi046 ceph-mon[36108]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:47.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 sudo[45549]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:29:47.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 sudo[45549]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:47.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 sudo[45549]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:47.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:47 smithi203 sudo[45549]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:48.419 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 sudo[44435]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T01:29:48.420 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 sudo[44435]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:48.420 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 sudo[44435]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:48.420 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 sudo[44435]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:48.695 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 sudo[44439]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T01:29:48.695 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 sudo[44439]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T01:29:48.695 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 sudo[44439]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T01:29:48.695 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 sudo[44439]: pam_unix(sudo:session): session closed for user root 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.696 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:48.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:48.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:48.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:48.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T01:29:48.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T01:29:49.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:49 smithi007 ceph-mon[31996]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:49.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:49 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1573232859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748"}]: dispatch 2024-03-18T01:29:49.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:49 smithi007 ceph-mon[31996]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:49.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:49 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1573232859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748"}]': finished 2024-03-18T01:29:49.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:49 smithi007 ceph-mon[31996]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T01:29:49.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:49 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:29:49.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:49 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1703743698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:29:49.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:49 smithi203 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:49.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:49 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1573232859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748"}]: dispatch 2024-03-18T01:29:49.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:49 smithi203 ceph-mon[36345]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:49.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:49 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1573232859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748"}]': finished 2024-03-18T01:29:49.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:49 smithi203 ceph-mon[36345]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T01:29:49.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:49 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:29:49.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:49 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1703743698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:29:50.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:49 smithi046 ceph-mon[36108]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T01:29:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:49 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1573232859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748"}]: dispatch 2024-03-18T01:29:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:49 smithi046 ceph-mon[36108]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T01:29:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:49 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1573232859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748"}]': finished 2024-03-18T01:29:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:49 smithi046 ceph-mon[36108]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T01:29:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:49 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:29:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:49 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1703743698' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:29:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:50 smithi203 ceph-mon[36345]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:51.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:50 smithi046 ceph-mon[36108]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:51.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:50 smithi007 ceph-mon[31996]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:52.825 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:52 smithi203 ceph-mon[36345]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:52.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:52 smithi007 ceph-mon[31996]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:53.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:52 smithi046 ceph-mon[36108]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:29:54.858 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:54 smithi007 ceph-mon[31996]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 58 KiB/s wr, 5 op/s 2024-03-18T01:29:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:54 smithi203 ceph-mon[36345]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 58 KiB/s wr, 5 op/s 2024-03-18T01:29:55.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:54 smithi046 ceph-mon[36108]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 58 KiB/s wr, 5 op/s 2024-03-18T01:29:56.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:56 smithi203 ceph-mon[36345]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-03-18T01:29:57.007 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:56 smithi007 ceph-mon[31996]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-03-18T01:29:57.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:56 smithi046 ceph-mon[36108]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 40 KiB/s wr, 3 op/s 2024-03-18T01:29:57.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T01:29:57.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T01:29:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:58.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T01:29:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:29:58.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:58 smithi007 ceph-mon[31996]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-18T01:29:58.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:29:58 smithi007 ceph-mon[31996]: Deploying daemon osd.3 on smithi007 2024-03-18T01:29:58.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:58 smithi203 ceph-mon[36345]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-18T01:29:58.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:29:58 smithi203 ceph-mon[36345]: Deploying daemon osd.3 on smithi007 2024-03-18T01:29:59.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:58 smithi046 ceph-mon[36108]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-18T01:29:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:29:58 smithi046 ceph-mon[36108]: Deploying daemon osd.3 on smithi007 2024-03-18T01:30:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:00 smithi203 ceph-mon[36345]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 876 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-18T01:30:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:00 smithi203 ceph-mon[36345]: overall HEALTH_OK 2024-03-18T01:30:00.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:00 smithi007 ceph-mon[31996]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 876 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-18T01:30:00.977 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:00 smithi007 ceph-mon[31996]: overall HEALTH_OK 2024-03-18T01:30:01.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:00 smithi046 ceph-mon[36108]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 876 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-18T01:30:01.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:00 smithi046 ceph-mon[36108]: overall HEALTH_OK 2024-03-18T01:30:02.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:02 smithi007 ceph-mon[31996]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T01:30:02.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:02 smithi203 ceph-mon[36345]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T01:30:03.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:02 smithi046 ceph-mon[36108]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T01:30:04.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:04 smithi203 ceph-mon[36345]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T01:30:04.961 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:04 smithi007 ceph-mon[31996]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T01:30:05.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:04 smithi046 ceph-mon[36108]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T01:30:06.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:06 smithi007 ceph-mon[31996]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:06.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:06.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:06.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:06 smithi203 ceph-mon[36345]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:07.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:06 smithi046 ceph-mon[36108]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:07.035 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:07.035 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:07.035 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:08.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:08 smithi007 ceph-mon[31996]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:08.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:08 smithi203 ceph-mon[36345]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:09.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:08 smithi046 ceph-mon[36108]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:10.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.380+0000 7f686ea00700 1 -- 172.21.15.7:0/2693704342 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6878062710 con 0x7f685c0a2800 2024-03-18T01:30:10.382 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 3 on host 'smithi007' 2024-03-18T01:30:10.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:10 smithi007 ceph-mon[31996]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:10.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:10.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:10.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:10 smithi007 ceph-mon[31996]: from='osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:30:10.383 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:10 smithi007 ceph-mon[31996]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f685c0a2800 msgr2=0x7f685c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f685c0a2800 0x7f685c0a4cc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6864009c60 tx=0x7f6864009260 comp rx=0 tx=0).stop 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6878104070 msgr2=0x7f687819a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6878104070 0x7f687819a840 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6868005250 tx=0x7f6868005280 comp rx=0 tx=0).stop 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 shutdown_connections 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f685c0a2800 0x7f685c0a4cc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6878105b30 0x7f68781a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6878104070 0x7f687819a840 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:10.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 --2- 172.21.15.7:0/2693704342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6878104a10 0x7f687819ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.384+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 >> 172.21.15.7:0/2693704342 conn(0x7f68780ff9f0 msgr2=0x7f6878108ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:30:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.385+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 shutdown_connections 2024-03-18T01:30:10.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:30:10.385+0000 7f6877200700 1 -- 172.21.15.7:0/2693704342 wait complete. 2024-03-18T01:30:10.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:10 smithi046 ceph-mon[36108]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:10 smithi046 ceph-mon[36108]: from='osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:30:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:10 smithi046 ceph-mon[36108]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:30:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:10 smithi203 ceph-mon[36345]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:10.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:10.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:10.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:10 smithi203 ceph-mon[36345]: from='osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:30:10.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:10 smithi203 ceph-mon[36345]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T01:30:11.115 DEBUG:teuthology.orchestra.run.smithi007:osd.3> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T01:30:11.120 INFO:tasks.cephadm:Deploying osd.4 on smithi046 with /dev/vg_nvme/lv_3... 2024-03-18T01:30:11.121 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T01:30:11.351 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-mon[31996]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T01:30:11.351 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-mon[31996]: from='osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:30:11.351 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-mon[31996]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T01:30:11.352 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:11.352 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-mon[31996]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:30:11.352 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:11.352 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:11.352 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:11.391 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:30:11.514 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:11 smithi046 ceph-mon[36108]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T01:30:11.514 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:11 smithi046 ceph-mon[36108]: from='osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:30:11.514 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:11 smithi046 ceph-mon[36108]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T01:30:11.514 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:11 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:11.514 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:11 smithi046 ceph-mon[36108]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:30:11.514 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:11 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:11.514 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:11 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:11.514 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:11 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:11.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:11 smithi203 ceph-mon[36345]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T01:30:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:11 smithi203 ceph-mon[36345]: from='osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:30:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:11 smithi203 ceph-mon[36345]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T01:30:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:11 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:11 smithi203 ceph-mon[36345]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:30:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:11 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:11 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:11 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:12.052 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3[53436]: 2024-03-18T01:30:11.878+0000 7fd270600700 -1 osd.3 0 waiting for initial osdmap 2024-03-18T01:30:12.053 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:30:11 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3[53436]: 2024-03-18T01:30:11.902+0000 7fd265a00700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:30:12.448 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:12 smithi007 ceph-mon[31996]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:12.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:12 smithi007 ceph-mon[31996]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:30:12.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:12 smithi007 ceph-mon[31996]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T01:30:12.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:12.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:12.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:12 smithi007 ceph-mon[31996]: from='osd.3 ' entity='osd.3' 2024-03-18T01:30:12.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:12 smithi046 ceph-mon[36108]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:12.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:12 smithi046 ceph-mon[36108]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:30:12.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:12 smithi046 ceph-mon[36108]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T01:30:12.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:12.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:12.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:12 smithi046 ceph-mon[36108]: from='osd.3 ' entity='osd.3' 2024-03-18T01:30:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:12 smithi203 ceph-mon[36345]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-03-18T01:30:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:12 smithi203 ceph-mon[36345]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:30:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:12 smithi203 ceph-mon[36345]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T01:30:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:12.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:12.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:12 smithi203 ceph-mon[36345]: from='osd.3 ' entity='osd.3' 2024-03-18T01:30:13.384 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:13 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:30:13.385 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:13 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:30:13.385 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:13 smithi046 ceph-mon[36108]: osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] boot 2024-03-18T01:30:13.385 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:13 smithi046 ceph-mon[36108]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T01:30:13.385 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:13 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:30:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:13 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:30:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:13 smithi007 ceph-mon[31996]: osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] boot 2024-03-18T01:30:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:13 smithi007 ceph-mon[31996]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T01:30:13.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:13.566 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T01:30:13.566 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T01:30:13.566 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-18T01:30:13.566 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-18T01:30:13.566 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0085393 s, 1.2 GB/s 2024-03-18T01:30:13.566 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-18T01:30:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:13 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:30:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:13 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:30:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:13 smithi203 ceph-mon[36345]: osd.3 [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] boot 2024-03-18T01:30:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:13 smithi203 ceph-mon[36345]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T01:30:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T01:30:14.364 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:14 smithi046 ceph-mon[36108]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:14.364 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:14 smithi046 ceph-mon[36108]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T01:30:14.430 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_3 2024-03-18T01:30:14.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:14 smithi007 ceph-mon[31996]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:14 smithi007 ceph-mon[31996]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T01:30:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:14 smithi203 ceph-mon[36345]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:14 smithi203 ceph-mon[36345]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T01:30:14.677 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:30:15.476 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:15 smithi046 ceph-mon[36108]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T01:30:15.477 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:15 smithi046 ceph-mon[36108]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T01:30:15.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:15 smithi007 ceph-mon[31996]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T01:30:15.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:15 smithi007 ceph-mon[31996]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T01:30:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:15 smithi203 ceph-mon[36345]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T01:30:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:15 smithi203 ceph-mon[36345]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T01:30:16.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 -- 172.21.15.46:0/1727725012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102470 msgr2=0x7f0a701028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1727725012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102470 0x7f0a701028d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0a600099c0 tx=0x7f0a60009cd0 comp rx=0 tx=0).stop 2024-03-18T01:30:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 -- 172.21.15.46:0/1727725012 shutdown_connections 2024-03-18T01:30:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1727725012 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a70102e10 0x7f0a701096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1727725012 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a70101270 0x7f0a70101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1727725012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102470 0x7f0a701028d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 -- 172.21.15.46:0/1727725012 >> 172.21.15.46:0/1727725012 conn(0x7f0a700fc830 msgr2=0x7f0a700fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:30:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 -- 172.21.15.46:0/1727725012 shutdown_connections 2024-03-18T01:30:16.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 -- 172.21.15.46:0/1727725012 wait complete. 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 Processor -- start 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.286+0000 7f0a785e9700 1 -- start start 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a785e9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a70101270 0x7f0a70198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a785e9700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a70102470 0x7f0a70198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a785e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102e10 0x7f0a7019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a785e9700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010ce90 con 0x7f0a70102470 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a785e9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010cd10 con 0x7f0a70102e10 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a785e9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7010d010 con 0x7f0a70101270 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102e10 0x7f0a7019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:30:16.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102e10 0x7f0a7019fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45056/0 (socket says 172.21.15.46:45056) 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 -- 172.21.15.46:0/1290793002 learned_addr learned my addr 172.21.15.46:0/1290793002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76385700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a70101270 0x7f0a70198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 -- 172.21.15.46:0/1290793002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a70101270 msgr2=0x7f0a70198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a70101270 0x7f0a70198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 -- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a70102470 msgr2=0x7f0a70198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a70102470 0x7f0a70198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 -- 172.21.15.46:0/1290793002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a701a0250 con 0x7f0a70102e10 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a76b86700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102e10 0x7f0a7019fbb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800ef30 tx=0x7f0a6800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a677fe700 1 -- 172.21.15.46:0/1290793002 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a680092e0 con 0x7f0a70102e10 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a701a04e0 con 0x7f0a70102e10 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a677fe700 1 -- 172.21.15.46:0/1290793002 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a68004bb0 con 0x7f0a70102e10 2024-03-18T01:30:16.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a677fe700 1 -- 172.21.15.46:0/1290793002 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68004280 con 0x7f0a70102e10 2024-03-18T01:30:16.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.287+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a701a0a50 con 0x7f0a70102e10 2024-03-18T01:30:16.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.288+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a70068f90 con 0x7f0a70102e10 2024-03-18T01:30:16.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.289+0000 7f0a677fe700 1 -- 172.21.15.46:0/1290793002 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f0a6801b030 con 0x7f0a70102e10 2024-03-18T01:30:16.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.290+0000 7f0a677fe700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a5c0a2890 0x7f0a5c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:30:16.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.290+0000 7f0a677fe700 1 -- 172.21.15.46:0/1290793002 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7f0a680cd3d0 con 0x7f0a70102e10 2024-03-18T01:30:16.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.291+0000 7f0a76385700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a5c0a2890 0x7f0a5c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:30:16.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.292+0000 7f0a76385700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a5c0a2890 0x7f0a5c0a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0a701022d0 tx=0x7f0a6c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:30:16.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.295+0000 7f0a677fe700 1 -- 172.21.15.46:0/1290793002 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a68022020 con 0x7f0a70102e10 2024-03-18T01:30:16.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:16.510+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0a700fe9d0 con 0x7f0a5c0a2890 2024-03-18T01:30:17.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:17.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi007 2024-03-18T01:30:17.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:17.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi007 to 8547M 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:30:17.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi007 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi007 to 8547M 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:30:17.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:30:17.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:17.164 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:17.164 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi007 2024-03-18T01:30:17.164 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi007 to 8547M 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T01:30:17.165 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:30:17.166 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:18.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:17 smithi007 ceph-mon[31996]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:30:18.086 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:17 smithi046 ceph-mon[36108]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:30:18.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:17 smithi203 ceph-mon[36345]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:30:19.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:18 smithi007 ceph-mon[31996]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:18 smithi203 ceph-mon[36345]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:19.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:18 smithi046 ceph-mon[36108]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 440 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:20.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:19 smithi203 ceph-mon[36345]: from='client.? 172.21.15.46:0/3590762258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]: dispatch 2024-03-18T01:30:20.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:19 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]: dispatch 2024-03-18T01:30:20.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:19 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]': finished 2024-03-18T01:30:20.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:19 smithi203 ceph-mon[36345]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T01:30:20.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:19 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:20.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:19 smithi046 ceph-mon[36108]: from='client.? 172.21.15.46:0/3590762258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]: dispatch 2024-03-18T01:30:20.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:19 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]: dispatch 2024-03-18T01:30:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:19 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]': finished 2024-03-18T01:30:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:19 smithi046 ceph-mon[36108]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T01:30:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:19 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:19 smithi007 ceph-mon[31996]: from='client.? 172.21.15.46:0/3590762258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]: dispatch 2024-03-18T01:30:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:19 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]: dispatch 2024-03-18T01:30:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:19 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "175f4003-26aa-4f4c-b0f9-cb83d4d8ab87"}]': finished 2024-03-18T01:30:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:19 smithi007 ceph-mon[31996]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T01:30:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:19 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:21.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:20 smithi203 ceph-mon[36345]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-18T01:30:21.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:20 smithi203 ceph-mon[36345]: from='client.? 172.21.15.46:0/1307627907' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:30:21.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:20 smithi046 ceph-mon[36108]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-18T01:30:21.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:20 smithi046 ceph-mon[36108]: from='client.? 172.21.15.46:0/1307627907' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:30:21.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:20 smithi007 ceph-mon[31996]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-18T01:30:21.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:20 smithi007 ceph-mon[31996]: from='client.? 172.21.15.46:0/1307627907' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:30:23.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:22 smithi203 ceph-mon[36345]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:30:23.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:22 smithi046 ceph-mon[36108]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:30:23.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:22 smithi007 ceph-mon[31996]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:30:25.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:24 smithi203 ceph-mon[36345]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:30:25.151 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:24 smithi046 ceph-mon[36108]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:30:25.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:24 smithi007 ceph-mon[31996]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:30:27.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:26 smithi007 ceph-mon[31996]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T01:30:27.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T01:30:27.105 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:26 smithi203 ceph-mon[36345]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T01:30:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T01:30:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:27.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:26 smithi046 ceph-mon[36108]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T01:30:27.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T01:30:27.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:27.915 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:27 smithi046 ceph-mon[36108]: Deploying daemon osd.4 on smithi046 2024-03-18T01:30:28.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:27 smithi203 ceph-mon[36345]: Deploying daemon osd.4 on smithi046 2024-03-18T01:30:28.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:27 smithi007 ceph-mon[31996]: Deploying daemon osd.4 on smithi046 2024-03-18T01:30:28.933 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:28 smithi046 ceph-mon[36108]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T01:30:29.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:28 smithi007 ceph-mon[31996]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T01:30:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:28 smithi203 ceph-mon[36345]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T01:30:31.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:30 smithi203 ceph-mon[36345]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:31.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:30 smithi046 ceph-mon[36108]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:31.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:30 smithi007 ceph-mon[31996]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:32.948 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:32 smithi046 ceph-mon[36108]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:32.948 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:32.948 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:32.948 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:32 smithi203 ceph-mon[36345]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:33.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:32 smithi007 ceph-mon[31996]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:33.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:33.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:33.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:34 smithi203 ceph-mon[36345]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:35.162 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:34 smithi046 ceph-mon[36108]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:35.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:34 smithi007 ceph-mon[31996]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:36.149 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 4 on host 'smithi046' 2024-03-18T01:30:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.147+0000 7f0a677fe700 1 -- 172.21.15.46:0/1290793002 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0a700fe9d0 con 0x7f0a5c0a2890 2024-03-18T01:30:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.149+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a5c0a2890 msgr2=0x7f0a5c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.149+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a5c0a2890 0x7f0a5c0a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0a701022d0 tx=0x7f0a6c009190 comp rx=0 tx=0).stop 2024-03-18T01:30:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.149+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102e10 msgr2=0x7f0a7019fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.149+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102e10 0x7f0a7019fbb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800ef30 tx=0x7f0a6800c3a0 comp rx=0 tx=0).stop 2024-03-18T01:30:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.149+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 shutdown_connections 2024-03-18T01:30:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.149+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a5c0a2890 0x7f0a5c0a4d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.149+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a70102e10 0x7f0a7019fbb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.150+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a70101270 0x7f0a70198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.150+0000 7f0a785e9700 1 --2- 172.21.15.46:0/1290793002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a70102470 0x7f0a70198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.150+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 >> 172.21.15.46:0/1290793002 conn(0x7f0a700fc830 msgr2=0x7f0a700fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:30:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.150+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 shutdown_connections 2024-03-18T01:30:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:30:36.150+0000 7f0a785e9700 1 -- 172.21.15.46:0/1290793002 wait complete. 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:30:36.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:36 smithi046 ceph-mon[36108]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:30:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:30:36.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:36 smithi007 ceph-mon[31996]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:30:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:36.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:36.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:36.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:36.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:30:36.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:36 smithi203 ceph-mon[36345]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T01:30:36.753 DEBUG:teuthology.orchestra.run.smithi046:osd.4> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T01:30:36.756 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-03-18T01:30:36.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T01:30:37.004 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:30:38.262 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:38 smithi046 ceph-mon[36108]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:38.262 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:38 smithi046 ceph-mon[36108]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T01:30:38.262 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:38 smithi046 ceph-mon[36108]: from='osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:30:38.263 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:38 smithi046 ceph-mon[36108]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T01:30:38.263 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:38.263 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:38 smithi046 ceph-mon[36108]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:30:38.265 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:30:38 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4[47240]: 2024-03-18T01:30:38.107+0000 7f4acd401700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T01:30:38.265 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:30:38 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4[47240]: 2024-03-18T01:30:38.114+0000 7f4ac91f4700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:30:38.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:38 smithi203 ceph-mon[36345]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:38.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:38 smithi203 ceph-mon[36345]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T01:30:38.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:38 smithi203 ceph-mon[36345]: from='osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:30:38.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:38 smithi203 ceph-mon[36345]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T01:30:38.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:38.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:38 smithi203 ceph-mon[36345]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:30:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:38 smithi007 ceph-mon[31996]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:38 smithi007 ceph-mon[31996]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T01:30:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:38 smithi007 ceph-mon[31996]: from='osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:30:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:38 smithi007 ceph-mon[31996]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T01:30:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:38 smithi007 ceph-mon[31996]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:30:39.009 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T01:30:39.009 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T01:30:39.009 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-18T01:30:39.009 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-18T01:30:39.009 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129231 s, 811 MB/s 2024-03-18T01:30:39.009 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-18T01:30:39.312 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:39 smithi203 ceph-mon[36345]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:30:39.312 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:39 smithi203 ceph-mon[36345]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T01:30:39.312 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:39 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:39.312 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:39 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:39.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:39 smithi046 ceph-mon[36108]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:30:39.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:39 smithi046 ceph-mon[36108]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T01:30:39.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:39 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:39.460 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:39 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:39.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:39 smithi007 ceph-mon[31996]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:30:39.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:39 smithi007 ceph-mon[31996]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T01:30:39.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:39 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:39.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:39 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:39.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-03-18T01:30:39.849 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:30:40.270 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] boot 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi046 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:40.271 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:40.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi046 to 8542M 2024-03-18T01:30:40.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:40.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:40.272 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] boot 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi046 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi046 to 8542M 2024-03-18T01:30:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:40.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:40.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:30:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:30:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: osd.4 [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] boot 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi046 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:30:40.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi046 to 8542M 2024-03-18T01:30:40.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:40.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:40.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:30:40.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:41.361 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:41 smithi203 ceph-mon[36345]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 -- 172.21.15.203:0/775036037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102e10 msgr2=0x7f93181096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 --2- 172.21.15.203:0/775036037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102e10 0x7f93181096c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9308009990 tx=0x7f9308009ca0 comp rx=0 tx=0).stop 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 -- 172.21.15.203:0/775036037 shutdown_connections 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 --2- 172.21.15.203:0/775036037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102e10 0x7f93181096c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 --2- 172.21.15.203:0/775036037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9318101270 0x7f9318101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 --2- 172.21.15.203:0/775036037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9318102470 0x7f93181028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 -- 172.21.15.203:0/775036037 >> 172.21.15.203:0/775036037 conn(0x7f93180fc830 msgr2=0x7f93180fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 -- 172.21.15.203:0/775036037 shutdown_connections 2024-03-18T01:30:41.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 -- 172.21.15.203:0/775036037 wait complete. 2024-03-18T01:30:41.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.364+0000 7f931d8c8700 1 Processor -- start 2024-03-18T01:30:41.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f931d8c8700 1 -- start start 2024-03-18T01:30:41.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f931d8c8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9318101270 0x7f93181984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:30:41.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f931d8c8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102470 0x7f9318198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f931d8c8700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9318102e10 0x7f931819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f931d8c8700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f931810ce50 con 0x7f9318102e10 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f931d8c8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f931810ccd0 con 0x7f9318101270 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f931d8c8700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f931810cfd0 con 0x7f9318102470 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102470 0x7f9318198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102470 0x7f9318198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:50660/0 (socket says 172.21.15.203:50660) 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 -- 172.21.15.203:0/1347068193 learned_addr learned my addr 172.21.15.203:0/1347068193 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f9316ffd700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9318101270 0x7f93181984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 -- 172.21.15.203:0/1347068193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9318101270 msgr2=0x7f93181984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9318101270 0x7f93181984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:41.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 -- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9318102e10 msgr2=0x7f931819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9318102e10 0x7f931819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 -- 172.21.15.203:0/1347068193 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93181a0140 con 0x7f9318102470 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f9316ffd700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9318101270 0x7f93181984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.365+0000 7f93167fc700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102470 0x7f9318198a20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f930c00b560 tx=0x7f930c00b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.366+0000 7f931c8c6700 1 -- 172.21.15.203:0/1347068193 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c015020 con 0x7f9318102470 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.366+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93181a0430 con 0x7f9318102470 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.366+0000 7f931c8c6700 1 -- 172.21.15.203:0/1347068193 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f930c011d10 con 0x7f9318102470 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.366+0000 7f931c8c6700 1 -- 172.21.15.203:0/1347068193 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c00f2e0 con 0x7f9318102470 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.366+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93181a09f0 con 0x7f9318102470 2024-03-18T01:30:41.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.367+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9318068f90 con 0x7f9318102470 2024-03-18T01:30:41.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.368+0000 7f931c8c6700 1 -- 172.21.15.203:0/1347068193 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f930c018020 con 0x7f9318102470 2024-03-18T01:30:41.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.369+0000 7f931c8c6700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f93000a2890 0x7f93000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:30:41.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.369+0000 7f931c8c6700 1 -- 172.21.15.203:0/1347068193 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f930c0cd770 con 0x7f9318102470 2024-03-18T01:30:41.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.369+0000 7f9316ffd700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f93000a2890 0x7f93000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:30:41.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.370+0000 7f9316ffd700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f93000a2890 0x7f93000a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93181022d0 tx=0x7f9310009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:30:41.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.373+0000 7f931c8c6700 1 -- 172.21.15.203:0/1347068193 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f930c01f020 con 0x7f9318102470 2024-03-18T01:30:41.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:41 smithi046 ceph-mon[36108]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T01:30:41.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:41 smithi007 ceph-mon[31996]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T01:30:41.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:30:41.577+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f931806b0e0 con 0x7f93000a2890 2024-03-18T01:30:42.489 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:42 smithi203 ceph-mon[36345]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:42.489 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:42 smithi203 ceph-mon[36345]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:30:42.489 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:30:42.490 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:30:42.490 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:42.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:42 smithi046 ceph-mon[36108]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:42.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:42 smithi046 ceph-mon[36108]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:30:42.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:30:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:30:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:42.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:42 smithi007 ceph-mon[31996]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:42.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:42 smithi007 ceph-mon[31996]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:30:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:30:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:30:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:44.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:44 smithi046 ceph-mon[36108]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:44.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:44 smithi007 ceph-mon[31996]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:44.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:44 smithi203 ceph-mon[36345]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:45.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:45 smithi046 ceph-mon[36108]: from='client.? 172.21.15.203:0/1363805313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]: dispatch 2024-03-18T01:30:45.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:45 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]: dispatch 2024-03-18T01:30:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:45 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]': finished 2024-03-18T01:30:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:45 smithi046 ceph-mon[36108]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T01:30:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:45 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:30:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:45 smithi046 ceph-mon[36108]: from='client.? 172.21.15.203:0/826791711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:30:45.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:45 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/1363805313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]: dispatch 2024-03-18T01:30:45.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:45 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]: dispatch 2024-03-18T01:30:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:45 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]': finished 2024-03-18T01:30:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:45 smithi007 ceph-mon[31996]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T01:30:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:45 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:30:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:45 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/826791711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:30:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:45 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/1363805313' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]: dispatch 2024-03-18T01:30:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:45 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]: dispatch 2024-03-18T01:30:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:45 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30797882-136d-4d56-8f01-3e02850e5c9c"}]': finished 2024-03-18T01:30:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:45 smithi203 ceph-mon[36345]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T01:30:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:45 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:30:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:45 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/826791711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:30:46.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:46 smithi203 ceph-mon[36345]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:46.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:46 smithi046 ceph-mon[36108]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:46.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:46 smithi007 ceph-mon[31996]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:48.582 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:48 smithi046 ceph-mon[36108]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:48 smithi203 ceph-mon[36345]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:48.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:48 smithi007 ceph-mon[31996]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:50.585 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:50 smithi203 ceph-mon[36345]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:50.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:50 smithi046 ceph-mon[36108]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:50.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:50 smithi007 ceph-mon[31996]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:52.564 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:52 smithi203 ceph-mon[36345]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:52.564 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T01:30:52.564 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:52.564 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:52 smithi203 ceph-mon[36345]: Deploying daemon osd.5 on smithi203 2024-03-18T01:30:52.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:52 smithi046 ceph-mon[36108]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T01:30:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:52 smithi046 ceph-mon[36108]: Deploying daemon osd.5 on smithi203 2024-03-18T01:30:52.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:52 smithi007 ceph-mon[31996]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T01:30:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:30:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:52 smithi007 ceph-mon[31996]: Deploying daemon osd.5 on smithi203 2024-03-18T01:30:54.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:54 smithi203 ceph-mon[36345]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:54.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:54 smithi046 ceph-mon[36108]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:54.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:54 smithi007 ceph-mon[31996]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:56.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:56 smithi203 ceph-mon[36345]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:56.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:56 smithi046 ceph-mon[36108]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:56.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:56 smithi007 ceph-mon[31996]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:58.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:58 smithi203 ceph-mon[36345]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:58.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:58.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:58.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:30:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:58.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:58 smithi046 ceph-mon[36108]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:58.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:58.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:58.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:30:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:58.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:58 smithi007 ceph-mon[31996]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:30:58.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:30:58.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:30:58.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:30:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:00 smithi046 ceph-mon[36108]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:00 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:00 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:00 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:00 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:00 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:00 smithi203 ceph-mon[36345]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:00 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:00 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:00 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:00.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:00 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:00.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:00 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:00 smithi007 ceph-mon[31996]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:00.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:00 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:00 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:00 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:00.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:00 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:00.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:00 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:00.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.915+0000 7f931c8c6700 1 -- 172.21.15.203:0/1347068193 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f931806b0e0 con 0x7f93000a2890 2024-03-18T01:31:00.916 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-03-18T01:31:00.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f93000a2890 msgr2=0x7f93000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:00.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f93000a2890 0x7f93000a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93181022d0 tx=0x7f9310009040 comp rx=0 tx=0).stop 2024-03-18T01:31:00.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102470 msgr2=0x7f9318198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:00.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102470 0x7f9318198a20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f930c00b560 tx=0x7f930c00b920 comp rx=0 tx=0).stop 2024-03-18T01:31:00.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 shutdown_connections 2024-03-18T01:31:00.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f93000a2890 0x7f93000a4d50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:00.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9318102e10 0x7f931819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:00.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9318101270 0x7f93181984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:00.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 --2- 172.21.15.203:0/1347068193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9318102470 0x7f9318198a20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:00.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.917+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 >> 172.21.15.203:0/1347068193 conn(0x7f93180fc830 msgr2=0x7f931806a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:31:00.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.918+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 shutdown_connections 2024-03-18T01:31:00.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:31:00.918+0000 7f931d8c8700 1 -- 172.21.15.203:0/1347068193 wait complete. 2024-03-18T01:31:01.620 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T01:31:01.623 INFO:tasks.cephadm:Deploying osd.6 on smithi007 with /dev/vg_nvme/lv_2... 2024-03-18T01:31:01.623 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T01:31:02.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:01 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:02.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:01 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:02.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:01 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:02.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:01 smithi007 ceph-mon[31996]: from='osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:31:02.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:01 smithi007 ceph-mon[31996]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:31:02.080 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:31:02.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:01 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:02.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:01 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:02.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:01 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:02.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:01 smithi203 ceph-mon[36345]: from='osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:31:02.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:01 smithi203 ceph-mon[36345]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:31:02.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:01 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:01 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:01 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:01 smithi046 ceph-mon[36108]: from='osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:31:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:01 smithi046 ceph-mon[36108]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T01:31:03.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:02 smithi203 ceph-mon[36345]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:03.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:02 smithi203 ceph-mon[36345]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T01:31:03.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:02 smithi203 ceph-mon[36345]: from='osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:31:03.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:02 smithi203 ceph-mon[36345]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T01:31:03.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:03.124 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:02 smithi203 ceph-mon[36345]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:31:03.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:02 smithi007 ceph-mon[31996]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:03.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:02 smithi007 ceph-mon[31996]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T01:31:03.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:02 smithi007 ceph-mon[31996]: from='osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:31:03.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:02 smithi007 ceph-mon[31996]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T01:31:03.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:03.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:02 smithi007 ceph-mon[31996]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:31:03.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:02 smithi046 ceph-mon[36108]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:02 smithi046 ceph-mon[36108]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T01:31:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:02 smithi046 ceph-mon[36108]: from='osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:31:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:02 smithi046 ceph-mon[36108]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T01:31:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:02 smithi046 ceph-mon[36108]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:31:03.891 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:31:03 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5[48455]: 2024-03-18T01:31:03.561+0000 7f1877f19700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T01:31:03.892 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:31:03 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5[48455]: 2024-03-18T01:31:03.568+0000 7f186fd04700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:31:04.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:03 smithi007 ceph-mon[31996]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:31:04.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:03 smithi007 ceph-mon[31996]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T01:31:04.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:03 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:04.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:03 smithi007 ceph-mon[31996]: from='osd.5 ' entity='osd.5' 2024-03-18T01:31:04.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:03 smithi046 ceph-mon[36108]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:31:04.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:03 smithi046 ceph-mon[36108]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T01:31:04.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:03 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:04.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:03 smithi046 ceph-mon[36108]: from='osd.5 ' entity='osd.5' 2024-03-18T01:31:04.285 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:03 smithi203 ceph-mon[36345]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:31:04.285 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:03 smithi203 ceph-mon[36345]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T01:31:04.285 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:03 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:04.285 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:03 smithi203 ceph-mon[36345]: from='osd.5 ' entity='osd.5' 2024-03-18T01:31:05.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] boot 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:04 smithi007 ceph-mon[31996]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T01:31:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T01:31:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T01:31:05.112 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-03-18T01:31:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-03-18T01:31:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0359253 s, 292 MB/s 2024-03-18T01:31:05.112 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-03-18T01:31:05.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] boot 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:04 smithi046 ceph-mon[36108]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T01:31:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:31:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: osd.5 [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] boot 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:04 smithi203 ceph-mon[36345]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T01:31:06.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:05 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi203 2024-03-18T01:31:06.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:05 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi203 to 8540M 2024-03-18T01:31:06.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:05 smithi007 ceph-mon[31996]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T01:31:06.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_2 2024-03-18T01:31:06.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:05 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi203 2024-03-18T01:31:06.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:05 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi203 to 8540M 2024-03-18T01:31:06.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:05 smithi046 ceph-mon[36108]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T01:31:06.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:05 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi203 2024-03-18T01:31:06.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:05 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi203 to 8540M 2024-03-18T01:31:06.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:05 smithi203 ceph-mon[36345]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T01:31:06.610 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:31:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:06 smithi007 ceph-mon[31996]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:07.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:06 smithi046 ceph-mon[36108]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:07.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:06 smithi203 ceph-mon[36345]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:08.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:07 smithi007 ceph-mon[31996]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T01:31:08.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:07 smithi046 ceph-mon[36108]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T01:31:08.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:07 smithi203 ceph-mon[36345]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T01:31:08.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 -- 172.21.15.7:0/2820549324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24e0105750 msgr2=0x7f24e0105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 --2- 172.21.15.7:0/2820549324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24e0105750 0x7f24e0105bb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009990 tx=0x7f24cc009ca0 comp rx=0 tx=0).stop 2024-03-18T01:31:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 -- 172.21.15.7:0/2820549324 shutdown_connections 2024-03-18T01:31:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 --2- 172.21.15.7:0/2820549324 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24e01060f0 0x7f24e010c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 --2- 172.21.15.7:0/2820549324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24e0104550 0x7f24e0104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 --2- 172.21.15.7:0/2820549324 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24e0105750 0x7f24e0105bb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 -- 172.21.15.7:0/2820549324 >> 172.21.15.7:0/2820549324 conn(0x7f24e00ffaf0 msgr2=0x7f24e0101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:31:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 -- 172.21.15.7:0/2820549324 shutdown_connections 2024-03-18T01:31:08.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.852+0000 7f24e5c00700 1 -- 172.21.15.7:0/2820549324 wait complete. 2024-03-18T01:31:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.854+0000 7f24e5c00700 1 Processor -- start 2024-03-18T01:31:08.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.854+0000 7f24e5c00700 1 -- start start 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.854+0000 7f24e5c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24e0104550 0x7f24e019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.854+0000 7f24e5c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24e0105750 0x7f24e019ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.854+0000 7f24e5c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24e01060f0 0x7f24e01a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.854+0000 7f24e5c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f24e0110110 con 0x7f24e01060f0 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24e5c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24e010ff90 con 0x7f24e0104550 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24e5c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24e0110290 con 0x7f24e0105750 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24dea00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24e0104550 0x7f24e019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24dea00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24e0104550 0x7f24e019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:58962/0 (socket says 172.21.15.7:58962) 2024-03-18T01:31:08.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24dea00700 1 -- 172.21.15.7:0/894093266 learned_addr learned my addr 172.21.15.7:0/894093266 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:31:08.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24de000700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24e0105750 0x7f24e019ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:31:08.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24df400700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24e01060f0 0x7f24e01a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24dea00700 1 -- 172.21.15.7:0/894093266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24e0105750 msgr2=0x7f24e019ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24dea00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24e0105750 0x7f24e019ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24dea00700 1 -- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24e01060f0 msgr2=0x7f24e01a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24dea00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24e01060f0 0x7f24e01a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24dea00700 1 -- 172.21.15.7:0/894093266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e01a2320 con 0x7f24e0104550 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.855+0000 7f24df400700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24e01060f0 0x7f24e01a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.856+0000 7f24dea00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24e0104550 0x7f24e019a6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f24d000e970 tx=0x7f24d000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.856+0000 7f24d7400700 1 -- 172.21.15.7:0/894093266 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d000cbd0 con 0x7f24e0104550 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.856+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e01a2610 con 0x7f24e0104550 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.856+0000 7f24d7400700 1 -- 172.21.15.7:0/894093266 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24d0004500 con 0x7f24e0104550 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.856+0000 7f24d7400700 1 -- 172.21.15.7:0/894093266 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d0003e30 con 0x7f24e0104550 2024-03-18T01:31:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.856+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e01a2ba0 con 0x7f24e0104550 2024-03-18T01:31:08.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.857+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e0068f90 con 0x7f24e0104550 2024-03-18T01:31:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.859+0000 7f24d7400700 1 -- 172.21.15.7:0/894093266 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f24d0004000 con 0x7f24e0104550 2024-03-18T01:31:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.861+0000 7f24d7400700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24c40a2890 0x7f24c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:31:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.861+0000 7f24d7400700 1 -- 172.21.15.7:0/894093266 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f24d00cdf90 con 0x7f24e0104550 2024-03-18T01:31:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.861+0000 7f24de000700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24c40a2890 0x7f24c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:31:08.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.862+0000 7f24de000700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24c40a2890 0x7f24c40a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f24cc00b2e0 tx=0x7f24cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:31:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:08.865+0000 7f24d7400700 1 -- 172.21.15.7:0/894093266 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f24d000b2c0 con 0x7f24e0104550 2024-03-18T01:31:09.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:09.109+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f24e0062710 con 0x7f24c40a2890 2024-03-18T01:31:09.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:08 smithi046 ceph-mon[36108]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:09.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:08 smithi046 ceph-mon[36108]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T01:31:09.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:08 smithi007 ceph-mon[31996]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:09.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:08 smithi007 ceph-mon[31996]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T01:31:09.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:08 smithi203 ceph-mon[36345]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:09.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:08 smithi203 ceph-mon[36345]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T01:31:10.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:09 smithi007 ceph-mon[31996]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-18T01:31:10.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:09 smithi007 ceph-mon[31996]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:31:10.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:31:10.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:31:10.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:10.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:09 smithi046 ceph-mon[36108]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-18T01:31:10.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:09 smithi046 ceph-mon[36108]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:31:10.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:31:10.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:31:10.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:10.292 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:09 smithi203 ceph-mon[36345]: pgmap v158: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-18T01:31:10.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:09 smithi203 ceph-mon[36345]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:31:10.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:31:10.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:31:10.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:12.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:12 smithi046 ceph-mon[36108]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:31:12.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:12 smithi007 ceph-mon[31996]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:31:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:12 smithi203 ceph-mon[36345]: pgmap v159: 1 pgs: 1 peering; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:31:14.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:14 smithi046 ceph-mon[36108]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T01:31:14.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:14 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3537737380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646180e1-c71d-4288-88f0-20ded1848288"}]: dispatch 2024-03-18T01:31:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:14 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3537737380' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "646180e1-c71d-4288-88f0-20ded1848288"}]': finished 2024-03-18T01:31:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:14 smithi046 ceph-mon[36108]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T01:31:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:14 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:14 smithi007 ceph-mon[31996]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T01:31:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:14 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3537737380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646180e1-c71d-4288-88f0-20ded1848288"}]: dispatch 2024-03-18T01:31:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:14 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3537737380' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "646180e1-c71d-4288-88f0-20ded1848288"}]': finished 2024-03-18T01:31:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:14 smithi007 ceph-mon[31996]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T01:31:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:14 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:14 smithi203 ceph-mon[36345]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T01:31:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:14 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3537737380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "646180e1-c71d-4288-88f0-20ded1848288"}]: dispatch 2024-03-18T01:31:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:14 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3537737380' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "646180e1-c71d-4288-88f0-20ded1848288"}]': finished 2024-03-18T01:31:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:14 smithi203 ceph-mon[36345]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T01:31:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:14 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:15.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:15 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/502602642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:31:15.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:15 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/502602642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:31:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:15 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/502602642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:31:16.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:16 smithi046 ceph-mon[36108]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:31:16.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:16 smithi007 ceph-mon[31996]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:31:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:16 smithi203 ceph-mon[36345]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:31:18.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:18 smithi046 ceph-mon[36108]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:18.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:18 smithi007 ceph-mon[31996]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:18 smithi203 ceph-mon[36345]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:20.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:20 smithi046 ceph-mon[36108]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:20.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:20 smithi007 ceph-mon[31996]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:20 smithi203 ceph-mon[36345]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:22.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:22 smithi046 ceph-mon[36108]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:22.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:22 smithi007 ceph-mon[31996]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:22.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:22 smithi203 ceph-mon[36345]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:23.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:23 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T01:31:23.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:23 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:23.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:23 smithi046 ceph-mon[36108]: Deploying daemon osd.6 on smithi007 2024-03-18T01:31:23.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:23 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T01:31:23.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:23 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:23.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:23 smithi007 ceph-mon[31996]: Deploying daemon osd.6 on smithi007 2024-03-18T01:31:23.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:23 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T01:31:23.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:23 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:23.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:23 smithi203 ceph-mon[36345]: Deploying daemon osd.6 on smithi007 2024-03-18T01:31:24.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:24 smithi046 ceph-mon[36108]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:24.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:24 smithi007 ceph-mon[31996]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:24.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:24 smithi203 ceph-mon[36345]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:26.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:26 smithi046 ceph-mon[36108]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:26 smithi007 ceph-mon[31996]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:26 smithi203 ceph-mon[36345]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:28.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:28 smithi007 ceph-mon[31996]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:28.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:28 smithi046 ceph-mon[36108]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:28 smithi203 ceph-mon[36345]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:30.470 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:30 smithi007 ceph-mon[31996]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:30.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:30 smithi046 ceph-mon[36108]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:30 smithi203 ceph-mon[36345]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:33.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:32 smithi046 ceph-mon[36108]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:33.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:32 smithi007 ceph-mon[31996]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:33.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:33.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:33.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:32 smithi203 ceph-mon[36345]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:34.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:34 smithi007 ceph-mon[31996]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:35.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:34 smithi046 ceph-mon[36108]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:34 smithi203 ceph-mon[36345]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:36.438 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:36 smithi007 ceph-mon[31996]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:36.439 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:36.439 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:36.439 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:36 smithi007 ceph-mon[31996]: from='osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T01:31:36.439 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:36 smithi007 ceph-mon[31996]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T01:31:36.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.572+0000 7f24d7400700 1 -- 172.21.15.7:0/894093266 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f24e0062710 con 0x7f24c40a2890 2024-03-18T01:31:36.573 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 6 on host 'smithi007' 2024-03-18T01:31:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24c40a2890 msgr2=0x7f24c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24c40a2890 0x7f24c40a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f24cc00b2e0 tx=0x7f24cc005fd0 comp rx=0 tx=0).stop 2024-03-18T01:31:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24e0104550 msgr2=0x7f24e019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24e0104550 0x7f24e019a6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f24d000e970 tx=0x7f24d000ec80 comp rx=0 tx=0).stop 2024-03-18T01:31:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 shutdown_connections 2024-03-18T01:31:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24c40a2890 0x7f24c40a4d50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24e01060f0 0x7f24e01a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24e0104550 0x7f24e019a6c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 --2- 172.21.15.7:0/894093266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24e0105750 0x7f24e019ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.576+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 >> 172.21.15.7:0/894093266 conn(0x7f24e00ffaf0 msgr2=0x7f24e0108a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:31:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.577+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 shutdown_connections 2024-03-18T01:31:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:31:36.577+0000 7f24e5c00700 1 -- 172.21.15.7:0/894093266 wait complete. 2024-03-18T01:31:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:36 smithi203 ceph-mon[36345]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:36 smithi203 ceph-mon[36345]: from='osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T01:31:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:36 smithi203 ceph-mon[36345]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T01:31:36.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:36 smithi046 ceph-mon[36108]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:36.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:36.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:36.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:36 smithi046 ceph-mon[36108]: from='osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T01:31:36.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:36 smithi046 ceph-mon[36108]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T01:31:37.499 DEBUG:teuthology.orchestra.run.smithi007:osd.6> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.6.service 2024-03-18T01:31:37.503 INFO:tasks.cephadm:Deploying osd.7 on smithi046 with /dev/vg_nvme/lv_2... 2024-03-18T01:31:37.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T01:31:37.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:37 smithi007 ceph-mon[31996]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T01:31:37.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:37 smithi007 ceph-mon[31996]: from='osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:31:37.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:37 smithi007 ceph-mon[31996]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T01:31:37.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:37 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:37.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:37 smithi007 ceph-mon[31996]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:31:37.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:37 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:37.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:37 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:37 smithi203 ceph-mon[36345]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T01:31:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:37 smithi203 ceph-mon[36345]: from='osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:31:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:37 smithi203 ceph-mon[36345]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T01:31:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:37 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:37 smithi203 ceph-mon[36345]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:31:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:37 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:37 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:37.761 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:31:37.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:37 smithi046 ceph-mon[36108]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T01:31:37.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:37 smithi046 ceph-mon[36108]: from='osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:31:37.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:37 smithi046 ceph-mon[36108]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T01:31:37.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:37 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:37.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:37 smithi046 ceph-mon[36108]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:31:37.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:37 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:37.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:37 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:38.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:38 smithi046 ceph-mon[36108]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:38.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:38 smithi046 ceph-mon[36108]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:31:38.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:38 smithi046 ceph-mon[36108]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T01:31:38.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:38.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:38.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:38 smithi007 ceph-mon[31996]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:38 smithi007 ceph-mon[31996]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:31:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:38 smithi007 ceph-mon[31996]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T01:31:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:38.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:38.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:38 smithi203 ceph-mon[36345]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:38.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:38 smithi203 ceph-mon[36345]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:31:38.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:38 smithi203 ceph-mon[36345]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T01:31:38.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:38.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:39.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:39 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:31:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:39 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:31:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:39 smithi046 ceph-mon[36108]: osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] boot 2024-03-18T01:31:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:39 smithi046 ceph-mon[36108]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T01:31:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:39 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:39 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:31:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:39 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:31:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:39 smithi007 ceph-mon[31996]: osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] boot 2024-03-18T01:31:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:39 smithi007 ceph-mon[31996]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T01:31:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:39 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:39.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:39 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:31:39.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:39 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:31:39.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:39 smithi203 ceph-mon[36345]: osd.6 [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] boot 2024-03-18T01:31:39.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:39 smithi203 ceph-mon[36345]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T01:31:39.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:39 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T01:31:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T01:31:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T01:31:39.759 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-18T01:31:39.759 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-18T01:31:39.759 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00808926 s, 1.3 GB/s 2024-03-18T01:31:39.760 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-18T01:31:40.393 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:40 smithi046 ceph-mon[36108]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:40.393 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:40 smithi046 ceph-mon[36108]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T01:31:40.394 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_2 2024-03-18T01:31:40.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:40 smithi203 ceph-mon[36345]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:40.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:40 smithi203 ceph-mon[36345]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T01:31:40.646 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:31:40.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:40 smithi007 ceph-mon[31996]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-03-18T01:31:40.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:40 smithi007 ceph-mon[31996]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 -- 172.21.15.46:0/803863195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 msgr2=0x7fd4fc10a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 --2- 172.21.15.46:0/803863195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 0x7fd4fc10a450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00b0a0 tx=0x7fd4ec00b3b0 comp rx=0 tx=0).stop 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 -- 172.21.15.46:0/803863195 shutdown_connections 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 --2- 172.21.15.46:0/803863195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 0x7fd4fc10a450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 --2- 172.21.15.46:0/803863195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4fc102de0 0x7fd4fc1051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 --2- 172.21.15.46:0/803863195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4fc105710 0x7fd4fc107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 -- 172.21.15.46:0/803863195 >> 172.21.15.46:0/803863195 conn(0x7fd4fc0fc790 msgr2=0x7fd4fc0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 -- 172.21.15.46:0/803863195 shutdown_connections 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.331+0000 7fd500c44700 1 -- 172.21.15.46:0/803863195 wait complete. 2024-03-18T01:31:42.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd500c44700 1 Processor -- start 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd500c44700 1 -- start start 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd500c44700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4fc102de0 0x7fd4fc1026d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd500c44700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4fc105710 0x7fd4fc100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd500c44700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 0x7fd4fc1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd500c44700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc10df20 con 0x7fd4fc105710 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd500c44700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc10dda0 con 0x7fd4fc108040 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd500c44700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd4fc10e0a0 con 0x7fd4fc102de0 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 0x7fd4fc1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 0x7fd4fc1012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39348/0 (socket says 172.21.15.46:39348) 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fad9d700 1 -- 172.21.15.46:0/1700928131 learned_addr learned my addr 172.21.15.46:0/1700928131 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fa59c700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4fc102de0 0x7fd4fc1026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:31:42.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fad9d700 1 -- 172.21.15.46:0/1700928131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4fc102de0 msgr2=0x7fd4fc1026d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fad9d700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4fc102de0 0x7fd4fc1026d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fad9d700 1 -- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4fc105710 msgr2=0x7fd4fc100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fad9d700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4fc105710 0x7fd4fc100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fad9d700 1 -- 172.21.15.46:0/1700928131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4fc101b10 con 0x7fd4fc108040 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.332+0000 7fd4fa59c700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4fc102de0 0x7fd4fc1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.333+0000 7fd4fad9d700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 0x7fd4fc1012d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec007f00 tx=0x7fd4ec004e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.333+0000 7fd4f37fe700 1 -- 172.21.15.46:0/1700928131 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec021070 con 0x7fd4fc108040 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.333+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4fc1a25d0 con 0x7fd4fc108040 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.333+0000 7fd4f37fe700 1 -- 172.21.15.46:0/1700928131 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4ec01d930 con 0x7fd4fc108040 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.333+0000 7fd4f37fe700 1 -- 172.21.15.46:0/1700928131 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec005570 con 0x7fd4fc108040 2024-03-18T01:31:42.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.333+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4fc1a2a80 con 0x7fd4fc108040 2024-03-18T01:31:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.335+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4fc068f90 con 0x7fd4fc108040 2024-03-18T01:31:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.335+0000 7fd4f37fe700 1 -- 172.21.15.46:0/1700928131 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fd4ec024070 con 0x7fd4fc108040 2024-03-18T01:31:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.336+0000 7fd4f37fe700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4e80a2890 0x7fd4e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:31:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.336+0000 7fd4f37fe700 1 -- 172.21.15.46:0/1700928131 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fd4ec00f020 con 0x7fd4fc108040 2024-03-18T01:31:42.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.337+0000 7fd4fa59c700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4e80a2890 0x7fd4e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:31:42.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.338+0000 7fd4fa59c700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4e80a2890 0x7fd4e80a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4005d90 tx=0x7fd4f4005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:31:42.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.341+0000 7fd4f37fe700 1 -- 172.21.15.46:0/1700928131 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd4ec02b020 con 0x7fd4fc108040 2024-03-18T01:31:42.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:42 smithi046 ceph-mon[36108]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:42.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:42.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:42 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:42 smithi007 ceph-mon[31996]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:42 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:31:42.556+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fd4fc062710 con 0x7fd4e80a2890 2024-03-18T01:31:42.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:42 smithi203 ceph-mon[36345]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:42.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:42.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:42.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:42.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:42.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:42 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:31:43.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi007 2024-03-18T01:31:43.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi007 to 5698M 2024-03-18T01:31:43.486 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:43.487 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:43 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi007 2024-03-18T01:31:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi007 to 5698M 2024-03-18T01:31:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:31:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:31:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:31:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:43 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:43.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi007 2024-03-18T01:31:43.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi007 to 5698M 2024-03-18T01:31:43.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:43.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:43.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:43.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='client.24290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:31:43.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:31:43.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:31:43.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:31:43.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:43.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:31:43.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:43 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:31:44.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:44 smithi046 ceph-mon[36108]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:45.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:44 smithi007 ceph-mon[31996]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:44 smithi203 ceph-mon[36345]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:46.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:45 smithi046 ceph-mon[36108]: from='client.? 172.21.15.46:0/3043738064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]: dispatch 2024-03-18T01:31:46.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:45 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]: dispatch 2024-03-18T01:31:46.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:45 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]': finished 2024-03-18T01:31:46.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:45 smithi046 ceph-mon[36108]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T01:31:46.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:45 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:31:46.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:45 smithi007 ceph-mon[31996]: from='client.? 172.21.15.46:0/3043738064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]: dispatch 2024-03-18T01:31:46.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:45 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]: dispatch 2024-03-18T01:31:46.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:45 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]': finished 2024-03-18T01:31:46.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:45 smithi007 ceph-mon[31996]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T01:31:46.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:45 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:31:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:45 smithi203 ceph-mon[36345]: from='client.? 172.21.15.46:0/3043738064' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]: dispatch 2024-03-18T01:31:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:45 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]: dispatch 2024-03-18T01:31:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:45 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec"}]': finished 2024-03-18T01:31:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:45 smithi203 ceph-mon[36345]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T01:31:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:45 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:31:47.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:46 smithi046 ceph-mon[36108]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:47.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:46 smithi046 ceph-mon[36108]: from='client.? 172.21.15.46:0/921513900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:31:47.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:46 smithi007 ceph-mon[31996]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:46 smithi007 ceph-mon[31996]: from='client.? 172.21.15.46:0/921513900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:31:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:46 smithi203 ceph-mon[36345]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:46 smithi203 ceph-mon[36345]: from='client.? 172.21.15.46:0/921513900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:31:49.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:48 smithi046 ceph-mon[36108]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:49.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:48 smithi007 ceph-mon[31996]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:49.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:48 smithi203 ceph-mon[36345]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:50.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:50 smithi046 ceph-mon[36108]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:51.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:50 smithi007 ceph-mon[31996]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:50 smithi203 ceph-mon[36345]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:52.847 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:52 smithi046 ceph-mon[36108]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:53.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:52 smithi007 ceph-mon[31996]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:53.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:52 smithi203 ceph-mon[36345]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:53.883 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:53 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T01:31:53.884 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:53 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:53.884 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:53 smithi046 ceph-mon[36108]: Deploying daemon osd.7 on smithi046 2024-03-18T01:31:54.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:53 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T01:31:54.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:53 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:54.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:53 smithi007 ceph-mon[31996]: Deploying daemon osd.7 on smithi046 2024-03-18T01:31:54.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:53 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T01:31:54.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:53 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:31:54.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:53 smithi203 ceph-mon[36345]: Deploying daemon osd.7 on smithi046 2024-03-18T01:31:54.949 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:54 smithi046 ceph-mon[36108]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:55.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:54 smithi007 ceph-mon[31996]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:55.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:54 smithi203 ceph-mon[36345]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:56.973 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:56 smithi046 ceph-mon[36108]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:57.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:56 smithi007 ceph-mon[31996]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:57.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:56 smithi203 ceph-mon[36345]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:58.866 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:58 smithi046 ceph-mon[36108]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:58 smithi007 ceph-mon[31996]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:31:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:58 smithi203 ceph-mon[36345]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:00.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:59 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:00.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:59 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:00.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:31:59 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:00.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:59 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:00.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:59 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:00.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:31:59 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:59 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:59 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:31:59 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:00.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:00 smithi007 ceph-mon[31996]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:01.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:00 smithi046 ceph-mon[36108]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:00 smithi203 ceph-mon[36345]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:02.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.602+0000 7fd4f37fe700 1 -- 172.21.15.46:0/1700928131 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd4fc062710 con 0x7fd4e80a2890 2024-03-18T01:32:02.603 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 7 on host 'smithi046' 2024-03-18T01:32:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4e80a2890 msgr2=0x7fd4e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4e80a2890 0x7fd4e80a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd4f4005d90 tx=0x7fd4f4005d20 comp rx=0 tx=0).stop 2024-03-18T01:32:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 msgr2=0x7fd4fc1012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 0x7fd4fc1012d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec007f00 tx=0x7fd4ec004e20 comp rx=0 tx=0).stop 2024-03-18T01:32:02.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 shutdown_connections 2024-03-18T01:32:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4e80a2890 0x7fd4e80a4d50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4fc108040 0x7fd4fc1012d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4fc102de0 0x7fd4fc1026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 --2- 172.21.15.46:0/1700928131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd4fc105710 0x7fd4fc100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 >> 172.21.15.46:0/1700928131 conn(0x7fd4fc0fc790 msgr2=0x7fd4fc103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:32:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 shutdown_connections 2024-03-18T01:32:02.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:32:02.605+0000 7fd500c44700 1 -- 172.21.15.46:0/1700928131 wait complete. 2024-03-18T01:32:03.236 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T01:32:03.237 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:02 smithi046 ceph-mon[36108]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T01:32:03.271 DEBUG:teuthology.orchestra.run.smithi046:osd.7> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.7.service 2024-03-18T01:32:03.274 INFO:tasks.cephadm:Deploying osd.8 on smithi203 with /dev/vg_nvme/lv_2... 2024-03-18T01:32:03.274 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T01:32:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:02 smithi007 ceph-mon[31996]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T01:32:03.303 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T01:32:03.304 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:02 smithi203 ceph-mon[36345]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T01:32:03.520 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:32:04.852 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:04 smithi046 ceph-mon[36108]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:04.852 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:04 smithi046 ceph-mon[36108]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T01:32:04.852 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:04 smithi046 ceph-mon[36108]: from='osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:32:04.852 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:04 smithi046 ceph-mon[36108]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T01:32:04.852 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:04.852 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:04 smithi046 ceph-mon[36108]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:32:04.853 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:32:04 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7[54267]: 2024-03-18T01:32:04.602+0000 7fa76b976700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T01:32:04.853 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:32:04 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7[54267]: 2024-03-18T01:32:04.613+0000 7fa767769700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:32:04.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:04 smithi203 ceph-mon[36345]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:04.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:04 smithi203 ceph-mon[36345]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T01:32:04.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:04 smithi203 ceph-mon[36345]: from='osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:32:04.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:04 smithi203 ceph-mon[36345]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T01:32:04.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:04.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:04 smithi203 ceph-mon[36345]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:32:05.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:04 smithi007 ceph-mon[31996]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:05.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:04 smithi007 ceph-mon[31996]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T01:32:05.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:04 smithi007 ceph-mon[31996]: from='osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:32:05.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:04 smithi007 ceph-mon[31996]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T01:32:05.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:05.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:04 smithi007 ceph-mon[31996]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:32:05.491 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T01:32:05.491 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T01:32:05.491 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-18T01:32:05.491 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-18T01:32:05.491 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116208 s, 902 MB/s 2024-03-18T01:32:05.491 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-18T01:32:05.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:05 smithi203 ceph-mon[36345]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:32:05.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:05 smithi203 ceph-mon[36345]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T01:32:05.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:05.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:05 smithi046 ceph-mon[36108]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:32:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:05 smithi046 ceph-mon[36108]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T01:32:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:05 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:05 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:05 smithi007 ceph-mon[31996]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:32:06.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:05 smithi007 ceph-mon[31996]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T01:32:06.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-03-18T01:32:06.417 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] boot 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:06.632 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:06.633 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] boot 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:06.797 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:06.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:06.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:06.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:06.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:06.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:06 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:07.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:06 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:32:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:32:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-18T01:32:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: osd.7 [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] boot 2024-03-18T01:32:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T01:32:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:07.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:06 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:07.851 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:07 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi046 2024-03-18T01:32:07.851 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:07 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi046 to 5694M 2024-03-18T01:32:07.851 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:07 smithi203 ceph-mon[36345]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T01:32:08.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:07 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi046 2024-03-18T01:32:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:07 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi046 to 5694M 2024-03-18T01:32:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:07 smithi046 ceph-mon[36108]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T01:32:08.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 -- 172.21.15.203:0/1207490355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944102300 msgr2=0x7f8944102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:08.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 --2- 172.21.15.203:0/1207490355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944102300 0x7f8944102720 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8934009990 tx=0x7f8934009ca0 comp rx=0 tx=0).stop 2024-03-18T01:32:08.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 -- 172.21.15.203:0/1207490355 shutdown_connections 2024-03-18T01:32:08.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 --2- 172.21.15.203:0/1207490355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8944103ea0 0x7f894410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:08.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 --2- 172.21.15.203:0/1207490355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944102300 0x7f8944102720 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:08.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 --2- 172.21.15.203:0/1207490355 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8944103500 0x7f8944103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:08.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 -- 172.21.15.203:0/1207490355 >> 172.21.15.203:0/1207490355 conn(0x7f89440fd8e0 msgr2=0x7f89440ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 -- 172.21.15.203:0/1207490355 shutdown_connections 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.046+0000 7f894913c700 1 -- 172.21.15.203:0/1207490355 wait complete. 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.047+0000 7f894913c700 1 Processor -- start 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.047+0000 7f894913c700 1 -- start start 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.047+0000 7f894913c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8944102300 0x7f8944198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.047+0000 7f894913c700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8944103500 0x7f8944198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894913c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944103ea0 0x7f894419fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894913c700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f894410df40 con 0x7f8944103500 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894913c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f894410ddc0 con 0x7f8944102300 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894913c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f894410e0c0 con 0x7f8944103ea0 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944103ea0 0x7f894419fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944103ea0 0x7f894419fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51016/0 (socket says 172.21.15.203:51016) 2024-03-18T01:32:08.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 -- 172.21.15.203:0/437094780 learned_addr learned my addr 172.21.15.203:0/437094780 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f8942d9d700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8944102300 0x7f8944198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 -- 172.21.15.203:0/437094780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8944102300 msgr2=0x7f8944198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8944102300 0x7f8944198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 -- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8944103500 msgr2=0x7f8944198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8944103500 0x7f8944198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 -- 172.21.15.203:0/437094780 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89441a0180 con 0x7f8944103ea0 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894359e700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944103ea0 0x7f894419fae0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f893c007d40 tx=0x7f893c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f892bfff700 1 -- 172.21.15.203:0/437094780 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c016020 con 0x7f8944103ea0 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f892bfff700 1 -- 172.21.15.203:0/437094780 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f893c00ad60 con 0x7f8944103ea0 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89441a0470 con 0x7f8944103ea0 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f892bfff700 1 -- 172.21.15.203:0/437094780 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893c011600 con 0x7f8944103ea0 2024-03-18T01:32:08.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.048+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89441a09e0 con 0x7f8944103ea0 2024-03-18T01:32:08.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.049+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8944068f90 con 0x7f8944103ea0 2024-03-18T01:32:08.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.050+0000 7f892bfff700 1 -- 172.21.15.203:0/437094780 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f893c019020 con 0x7f8944103ea0 2024-03-18T01:32:08.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.051+0000 7f892bfff700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f892c0a2890 0x7f892c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:32:08.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.051+0000 7f892bfff700 1 -- 172.21.15.203:0/437094780 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f893c0cdf70 con 0x7f8944103ea0 2024-03-18T01:32:08.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.051+0000 7f8942d9d700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f892c0a2890 0x7f892c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:32:08.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:07 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi046 2024-03-18T01:32:08.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:07 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi046 to 5694M 2024-03-18T01:32:08.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:07 smithi007 ceph-mon[31996]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T01:32:08.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.052+0000 7f8942d9d700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f892c0a2890 0x7f892c0a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8934003890 tx=0x7f8934003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:32:08.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.055+0000 7f892bfff700 1 -- 172.21.15.203:0/437094780 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f893c020020 con 0x7f8944103ea0 2024-03-18T01:32:08.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:08.260+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8944062710 con 0x7f892c0a2890 2024-03-18T01:32:08.911 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:08 smithi203 ceph-mon[36345]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:08.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:08 smithi203 ceph-mon[36345]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T01:32:08.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:32:08.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:32:08.912 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:08 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:09.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:08 smithi046 ceph-mon[36108]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:09.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:08 smithi046 ceph-mon[36108]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T01:32:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:32:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:32:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:08 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:08 smithi007 ceph-mon[31996]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:08 smithi007 ceph-mon[31996]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T01:32:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:32:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:32:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:08 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:09.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:09 smithi203 ceph-mon[36345]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:32:09.914 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:09 smithi203 ceph-mon[36345]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T01:32:10.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:09 smithi046 ceph-mon[36108]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:32:10.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:09 smithi046 ceph-mon[36108]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T01:32:10.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:09 smithi007 ceph-mon[31996]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:32:10.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:09 smithi007 ceph-mon[31996]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T01:32:11.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:10 smithi046 ceph-mon[36108]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:11.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:10 smithi046 ceph-mon[36108]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T01:32:11.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:10 smithi007 ceph-mon[31996]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:11.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:10 smithi007 ceph-mon[31996]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T01:32:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:10 smithi203 ceph-mon[36345]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:10 smithi203 ceph-mon[36345]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T01:32:12.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:11 smithi046 ceph-mon[36108]: from='client.? 172.21.15.203:0/1133391122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]: dispatch 2024-03-18T01:32:12.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:11 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]: dispatch 2024-03-18T01:32:12.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:11 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]': finished 2024-03-18T01:32:12.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:11 smithi046 ceph-mon[36108]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T01:32:12.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:11 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:12.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:11 smithi046 ceph-mon[36108]: from='client.? 172.21.15.203:0/3660129868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:32:12.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:11 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/1133391122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]: dispatch 2024-03-18T01:32:12.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:11 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]: dispatch 2024-03-18T01:32:12.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:11 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]': finished 2024-03-18T01:32:12.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:11 smithi007 ceph-mon[31996]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T01:32:12.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:11 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:12.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:11 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/3660129868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:32:12.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:11 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/1133391122' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]: dispatch 2024-03-18T01:32:12.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:11 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]: dispatch 2024-03-18T01:32:12.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:11 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db012442-b453-4810-8aba-51d2e9506797"}]': finished 2024-03-18T01:32:12.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:11 smithi203 ceph-mon[36345]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T01:32:12.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:11 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:12.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:11 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/3660129868' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:32:13.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:12 smithi046 ceph-mon[36108]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:13.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:12 smithi007 ceph-mon[31996]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:12 smithi203 ceph-mon[36345]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:14.835 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:14 smithi203 ceph-mon[36345]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:15.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:14 smithi046 ceph-mon[36108]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:15.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:14 smithi007 ceph-mon[31996]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:17.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:16 smithi046 ceph-mon[36108]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:17.028 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:16 smithi203 ceph-mon[36345]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:17.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:16 smithi007 ceph-mon[31996]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:19.009 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:18 smithi203 ceph-mon[36345]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:19.009 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T01:32:19.009 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:18 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:19.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:18 smithi046 ceph-mon[36108]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:19.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T01:32:19.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:18 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:19.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:18 smithi007 ceph-mon[31996]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:19.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T01:32:19.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:18 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:19.998 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:19 smithi203 ceph-mon[36345]: Deploying daemon osd.8 on smithi203 2024-03-18T01:32:20.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:19 smithi046 ceph-mon[36108]: Deploying daemon osd.8 on smithi203 2024-03-18T01:32:20.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:19 smithi007 ceph-mon[31996]: Deploying daemon osd.8 on smithi203 2024-03-18T01:32:21.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:20 smithi046 ceph-mon[36108]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:21.021 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:20 smithi203 ceph-mon[36345]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:21.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:20 smithi007 ceph-mon[31996]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:23.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:22 smithi046 ceph-mon[36108]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:23.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:22 smithi007 ceph-mon[31996]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:23.073 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:22 smithi203 ceph-mon[36345]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:24.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:23 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:24.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:23 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:24.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:23 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:24.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:23 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:24.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:23 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:24.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:23 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:24.059 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:23 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:24.059 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:23 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:24.059 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:23 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:25.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:24 smithi046 ceph-mon[36108]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:25.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:24 smithi007 ceph-mon[31996]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:25.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:24 smithi203 ceph-mon[36345]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:26.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:26 smithi203 ceph-mon[36345]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:27.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:26 smithi046 ceph-mon[36108]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:27.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:26 smithi007 ceph-mon[31996]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:27.395 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 8 on host 'smithi203' 2024-03-18T01:32:27.396 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.394+0000 7f892bfff700 1 -- 172.21.15.203:0/437094780 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8944062710 con 0x7f892c0a2890 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f892c0a2890 msgr2=0x7f892c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f892c0a2890 0x7f892c0a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8934003890 tx=0x7f8934003960 comp rx=0 tx=0).stop 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944103ea0 msgr2=0x7f894419fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944103ea0 0x7f894419fae0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f893c007d40 tx=0x7f893c00a340 comp rx=0 tx=0).stop 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 shutdown_connections 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f892c0a2890 0x7f892c0a4d50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8944103ea0 0x7f894419fae0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8944102300 0x7f8944198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 --2- 172.21.15.203:0/437094780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8944103500 0x7f8944198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 >> 172.21.15.203:0/437094780 conn(0x7f89440fd8e0 msgr2=0x7f8944106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 shutdown_connections 2024-03-18T01:32:27.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:32:27.397+0000 7f894913c700 1 -- 172.21.15.203:0/437094780 wait complete. 2024-03-18T01:32:28.040 DEBUG:teuthology.orchestra.run.smithi203:osd.8> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.8.service 2024-03-18T01:32:28.042 INFO:tasks.cephadm:Deploying osd.9 on smithi007 with /dev/vg_nvme/lv_1... 2024-03-18T01:32:28.042 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T01:32:28.523 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:32:28.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:28 smithi007 ceph-mon[31996]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:28.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:28 smithi007 ceph-mon[31996]: from='osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T01:32:28.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:28 smithi007 ceph-mon[31996]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T01:32:28.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:28 smithi203 ceph-mon[36345]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:28 smithi203 ceph-mon[36345]: from='osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T01:32:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:28 smithi203 ceph-mon[36345]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T01:32:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:28 smithi046 ceph-mon[36108]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:28 smithi046 ceph-mon[36108]: from='osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T01:32:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:28 smithi046 ceph-mon[36108]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T01:32:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:29 smithi203 ceph-mon[36345]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T01:32:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:29 smithi203 ceph-mon[36345]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T01:32:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:29 smithi203 ceph-mon[36345]: from='osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:32:29.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:29 smithi203 ceph-mon[36345]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:32:29.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:29 smithi007 ceph-mon[31996]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T01:32:29.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:29 smithi007 ceph-mon[31996]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T01:32:29.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:29.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:29 smithi007 ceph-mon[31996]: from='osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:32:29.699 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:29 smithi007 ceph-mon[31996]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:32:29.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:29 smithi046 ceph-mon[36108]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T01:32:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:29 smithi046 ceph-mon[36108]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T01:32:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:29 smithi046 ceph-mon[36108]: from='osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:32:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:29 smithi046 ceph-mon[36108]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:32:30.362 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:32:30.076+0000 7f93b123b700 -1 osd.8 0 waiting for initial osdmap 2024-03-18T01:32:30.362 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:32:30.087+0000 7f93a9026700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:32:30.452 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:30.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:32:30.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T01:32:30.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:30.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:30.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: from='osd.8 ' entity='osd.8' 2024-03-18T01:32:30.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] boot 2024-03-18T01:32:30.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T01:32:30.453 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:32:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T01:32:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:30.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:30.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: from='osd.8 ' entity='osd.8' 2024-03-18T01:32:30.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] boot 2024-03-18T01:32:30.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T01:32:30.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:30.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-03-18T01:32:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:32:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T01:32:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: from='osd.8 ' entity='osd.8' 2024-03-18T01:32:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: osd.8 [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] boot 2024-03-18T01:32:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T01:32:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T01:32:31.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:32:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:32:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T01:32:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:32:31.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.762 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:31.762 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:31.762 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:31.762 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T01:32:31.762 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T01:32:31.762 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-03-18T01:32:31.762 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-03-18T01:32:31.762 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0533924 s, 196 MB/s 2024-03-18T01:32:31.762 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-03-18T01:32:32.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:32 smithi203 ceph-mon[36345]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:32.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:32 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi203 2024-03-18T01:32:32.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:32 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi203 to 5693M 2024-03-18T01:32:32.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:32.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:32.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:32.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:32.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:32 smithi203 ceph-mon[36345]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T01:32:32.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:32 smithi046 ceph-mon[36108]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:32.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:32 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi203 2024-03-18T01:32:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:32 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi203 to 5693M 2024-03-18T01:32:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:32 smithi046 ceph-mon[36108]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T01:32:32.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:32 smithi007 ceph-mon[31996]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:32.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:32 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi203 2024-03-18T01:32:32.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:32 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi203 to 5693M 2024-03-18T01:32:32.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:32.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:32.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:32:32.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:32.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:32 smithi007 ceph-mon[31996]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T01:32:32.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_1 2024-03-18T01:32:33.440 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:32:34.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:34 smithi007 ceph-mon[31996]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:34.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:34 smithi007 ceph-mon[31996]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T01:32:34.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:34 smithi203 ceph-mon[36345]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:34.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:34 smithi203 ceph-mon[36345]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T01:32:34.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:34 smithi046 ceph-mon[36108]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:34.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:34 smithi046 ceph-mon[36108]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T01:32:35.297 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:35 smithi007 ceph-mon[31996]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T01:32:35.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:35 smithi046 ceph-mon[36108]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T01:32:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.621+0000 7fbfa0400700 1 -- 172.21.15.7:0/2773536251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98103eb0 msgr2=0x7fbf9810a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.621+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2773536251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98103eb0 0x7fbf9810a760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000b0d0 tx=0x7fbf9000b3e0 comp rx=0 tx=0).stop 2024-03-18T01:32:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.622+0000 7fbfa0400700 1 -- 172.21.15.7:0/2773536251 shutdown_connections 2024-03-18T01:32:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.622+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2773536251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98103eb0 0x7fbf9810a760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.622+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2773536251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf98102310 0x7fbf98102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.622+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2773536251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf98103510 0x7fbf98103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:35.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.622+0000 7fbfa0400700 1 -- 172.21.15.7:0/2773536251 >> 172.21.15.7:0/2773536251 conn(0x7fbf980fd870 msgr2=0x7fbf980ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:32:35.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.622+0000 7fbfa0400700 1 -- 172.21.15.7:0/2773536251 shutdown_connections 2024-03-18T01:32:35.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.622+0000 7fbfa0400700 1 -- 172.21.15.7:0/2773536251 wait complete. 2024-03-18T01:32:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.623+0000 7fbfa0400700 1 Processor -- start 2024-03-18T01:32:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.623+0000 7fbfa0400700 1 -- start start 2024-03-18T01:32:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.623+0000 7fbfa0400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98102310 0x7fbf9819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:32:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.623+0000 7fbfa0400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf98103510 0x7fbf9819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:32:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.623+0000 7fbfa0400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf98103eb0 0x7fbf981a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:32:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.623+0000 7fbfa0400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9810dee0 con 0x7fbf98102310 2024-03-18T01:32:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.623+0000 7fbfa0400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9810dd60 con 0x7fbf98103eb0 2024-03-18T01:32:35.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.623+0000 7fbfa0400700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9810e060 con 0x7fbf98103510 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9da00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98102310 0x7fbf9819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf98103eb0 0x7fbf981a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9d000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf98103510 0x7fbf9819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf98103eb0 0x7fbf981a1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:35142/0 (socket says 172.21.15.7:35142) 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9da00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98102310 0x7fbf9819a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57324/0 (socket says 172.21.15.7:57324) 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 -- 172.21.15.7:0/2100206888 learned_addr learned my addr 172.21.15.7:0/2100206888 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 -- 172.21.15.7:0/2100206888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf98103510 msgr2=0x7fbf9819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf98103510 0x7fbf9819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 -- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98102310 msgr2=0x7fbf9819a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98102310 0x7fbf9819a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 -- 172.21.15.7:0/2100206888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf981a22d0 con 0x7fbf98103eb0 2024-03-18T01:32:35.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9da00700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98102310 0x7fbf9819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:32:35.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.624+0000 7fbf9e400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf98103eb0 0x7fbf981a1c30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000d010 tx=0x7fbf90007a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:32:35.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.625+0000 7fbf96000700 1 -- 172.21.15.7:0/2100206888 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf90003c00 con 0x7fbf98103eb0 2024-03-18T01:32:35.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.625+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf981a2560 con 0x7fbf98103eb0 2024-03-18T01:32:35.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.625+0000 7fbf96000700 1 -- 172.21.15.7:0/2100206888 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf90022070 con 0x7fbf98103eb0 2024-03-18T01:32:35.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.625+0000 7fbf96000700 1 -- 172.21.15.7:0/2100206888 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9001b770 con 0x7fbf98103eb0 2024-03-18T01:32:35.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.625+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf981a29b0 con 0x7fbf98103eb0 2024-03-18T01:32:35.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.626+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf98068f90 con 0x7fbf98103eb0 2024-03-18T01:32:35.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.628+0000 7fbf96000700 1 -- 172.21.15.7:0/2100206888 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fbf9001d470 con 0x7fbf98103eb0 2024-03-18T01:32:35.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.630+0000 7fbf96000700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf800a2910 0x7fbf800a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:32:35.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.630+0000 7fbf96000700 1 -- 172.21.15.7:0/2100206888 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fbf900cf080 con 0x7fbf98103eb0 2024-03-18T01:32:35.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.630+0000 7fbf9da00700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf800a2910 0x7fbf800a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:32:35.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.631+0000 7fbf9da00700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf800a2910 0x7fbf800a4dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf98103370 tx=0x7fbf8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:32:35.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.635+0000 7fbf96000700 1 -- 172.21.15.7:0/2100206888 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf9009cac0 con 0x7fbf98103eb0 2024-03-18T01:32:35.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:35 smithi203 ceph-mon[36345]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T01:32:35.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:32:35.872+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbf98075c20 con 0x7fbf800a2910 2024-03-18T01:32:36.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:36 smithi046 ceph-mon[36108]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:36.532 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:36 smithi046 ceph-mon[36108]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:32:36.532 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:32:36.532 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:32:36.532 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:36.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:36 smithi007 ceph-mon[31996]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:36.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:36 smithi007 ceph-mon[31996]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:32:36.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:32:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:32:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:36 smithi203 ceph-mon[36345]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 488 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:36 smithi203 ceph-mon[36345]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:32:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:32:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:32:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:38.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:38 smithi046 ceph-mon[36108]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:32:38.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:38 smithi007 ceph-mon[31996]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:32:38.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:38 smithi203 ceph-mon[36345]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T01:32:40.443 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:40 smithi203 ceph-mon[36345]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-18T01:32:40.444 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:40 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2607815380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "352a39d7-32c4-461f-8c31-61badd6ba568"}]: dispatch 2024-03-18T01:32:40.444 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:40 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2607815380' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "352a39d7-32c4-461f-8c31-61badd6ba568"}]': finished 2024-03-18T01:32:40.444 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:40 smithi203 ceph-mon[36345]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T01:32:40.444 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:32:40.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:40 smithi046 ceph-mon[36108]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-18T01:32:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:40 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2607815380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "352a39d7-32c4-461f-8c31-61badd6ba568"}]: dispatch 2024-03-18T01:32:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:40 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2607815380' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "352a39d7-32c4-461f-8c31-61badd6ba568"}]': finished 2024-03-18T01:32:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:40 smithi046 ceph-mon[36108]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T01:32:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:32:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:40 smithi007 ceph-mon[31996]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-18T01:32:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:40 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2607815380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "352a39d7-32c4-461f-8c31-61badd6ba568"}]: dispatch 2024-03-18T01:32:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:40 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2607815380' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "352a39d7-32c4-461f-8c31-61badd6ba568"}]': finished 2024-03-18T01:32:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:40 smithi007 ceph-mon[31996]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T01:32:40.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:32:41.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:41 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/4184952100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:32:41.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:41 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4184952100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:32:41.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:41 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4184952100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:32:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:42 smithi046 ceph-mon[36108]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:32:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:42 smithi007 ceph-mon[31996]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:32:42.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:42 smithi203 ceph-mon[36345]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:32:44.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:44 smithi046 ceph-mon[36108]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T01:32:44.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:44 smithi007 ceph-mon[31996]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T01:32:44.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:44 smithi203 ceph-mon[36345]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-18T01:32:46.446 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:46 smithi007 ceph-mon[31996]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T01:32:46.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:46 smithi046 ceph-mon[36108]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T01:32:46.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:46 smithi203 ceph-mon[36345]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T01:32:48.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:48 smithi007 ceph-mon[31996]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:48.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:48 smithi046 ceph-mon[36108]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:48 smithi203 ceph-mon[36345]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:50.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:50 smithi007 ceph-mon[31996]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:50.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T01:32:50.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:50.445 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:50 smithi007 ceph-mon[31996]: Deploying daemon osd.9 on smithi007 2024-03-18T01:32:50.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:50 smithi046 ceph-mon[36108]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:50.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T01:32:50.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:50.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:50 smithi046 ceph-mon[36108]: Deploying daemon osd.9 on smithi007 2024-03-18T01:32:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:50 smithi203 ceph-mon[36345]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T01:32:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:32:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:50 smithi203 ceph-mon[36345]: Deploying daemon osd.9 on smithi007 2024-03-18T01:32:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:52 smithi007 ceph-mon[31996]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:52.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:52 smithi046 ceph-mon[36108]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:52.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:52 smithi203 ceph-mon[36345]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:54.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:54 smithi046 ceph-mon[36108]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:54.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:54 smithi007 ceph-mon[31996]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:54.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:54 smithi203 ceph-mon[36345]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:56.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:56 smithi046 ceph-mon[36108]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:56.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:56 smithi007 ceph-mon[31996]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:56.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:56 smithi203 ceph-mon[36345]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:59.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:58 smithi007 ceph-mon[31996]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:32:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:58 smithi203 ceph-mon[36345]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:32:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:32:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:58 smithi046 ceph-mon[36108]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:32:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:32:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:32:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:00 smithi203 ceph-mon[36345]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:01.142 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:00 smithi007 ceph-mon[31996]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:01.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:00 smithi046 ceph-mon[36108]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:02.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.880+0000 7fbf96000700 1 -- 172.21.15.7:0/2100206888 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbf98075c20 con 0x7fbf800a2910 2024-03-18T01:33:02.881 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 9 on host 'smithi007' 2024-03-18T01:33:02.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf800a2910 msgr2=0x7fbf800a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:02.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf800a2910 0x7fbf800a4dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbf98103370 tx=0x7fbf8c009260 comp rx=0 tx=0).stop 2024-03-18T01:33:02.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf98103eb0 msgr2=0x7fbf981a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:02.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf98103eb0 0x7fbf981a1c30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000d010 tx=0x7fbf90007a00 comp rx=0 tx=0).stop 2024-03-18T01:33:02.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 shutdown_connections 2024-03-18T01:33:02.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf800a2910 0x7fbf800a4dd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:02.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf98103eb0 0x7fbf981a1c30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:02.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf98102310 0x7fbf9819a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:02.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 --2- 172.21.15.7:0/2100206888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf98103510 0x7fbf9819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:02.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 >> 172.21.15.7:0/2100206888 conn(0x7fbf980fd870 msgr2=0x7fbf98074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:33:02.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.884+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 shutdown_connections 2024-03-18T01:33:02.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:02.885+0000 7fbfa0400700 1 -- 172.21.15.7:0/2100206888 wait complete. 2024-03-18T01:33:03.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:02 smithi007 ceph-mon[31996]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:03.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:03.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:02 smithi007 ceph-mon[31996]: from='osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T01:33:03.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:02 smithi007 ceph-mon[31996]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T01:33:03.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:02 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:02 smithi203 ceph-mon[36345]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:02 smithi203 ceph-mon[36345]: from='osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T01:33:03.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:02 smithi203 ceph-mon[36345]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T01:33:03.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:02 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:03.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:02 smithi046 ceph-mon[36108]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:02 smithi046 ceph-mon[36108]: from='osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T01:33:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:02 smithi046 ceph-mon[36108]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T01:33:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:02 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:03.763 DEBUG:teuthology.orchestra.run.smithi007:osd.9> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.9.service 2024-03-18T01:33:03.770 INFO:tasks.cephadm:Deploying osd.10 on smithi046 with /dev/vg_nvme/lv_1... 2024-03-18T01:33:03.770 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T01:33:04.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T01:33:04.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: from='osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:33:04.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T01:33:04.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:04.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:33:04.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:04.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:04.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:33:04.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T01:33:04.013 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:03 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:33:04.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T01:33:04.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: from='osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:33:04.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T01:33:04.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:04.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:33:04.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:04.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:04.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:33:04.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T01:33:04.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:03 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T01:33:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: from='osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:33:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T01:33:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:04.154 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-03-18T01:33:04.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:04.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:04.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-03-18T01:33:04.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T01:33:04.155 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:03 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:04 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:33:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:04 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:33:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:04 smithi203 ceph-mon[36345]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:05.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:04 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:33:05.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:04 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:33:05.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:04 smithi007 ceph-mon[31996]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:05.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:05.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:04 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:33:05.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:04 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:33:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:04 smithi046 ceph-mon[36108]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:05 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:05 smithi046 ceph-mon[36108]: osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] boot 2024-03-18T01:33:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:05 smithi046 ceph-mon[36108]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T01:33:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:05 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:06.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:06.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:05 smithi203 ceph-mon[36345]: osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] boot 2024-03-18T01:33:06.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:05 smithi203 ceph-mon[36345]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T01:33:06.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:06.187 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T01:33:06.188 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T01:33:06.188 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-18T01:33:06.188 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-18T01:33:06.188 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00808757 s, 1.3 GB/s 2024-03-18T01:33:06.188 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-18T01:33:06.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:06.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:05 smithi007 ceph-mon[31996]: osd.9 [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] boot 2024-03-18T01:33:06.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:05 smithi007 ceph-mon[31996]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T01:33:06.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T01:33:06.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_1 2024-03-18T01:33:07.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:06 smithi203 ceph-mon[36345]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:07.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:06 smithi203 ceph-mon[36345]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T01:33:07.144 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:33:07.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:06 smithi046 ceph-mon[36108]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:07.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:06 smithi046 ceph-mon[36108]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T01:33:07.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:06 smithi007 ceph-mon[31996]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-18T01:33:07.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:06 smithi007 ceph-mon[31996]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T01:33:08.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 -- 172.21.15.46:0/1452381236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468102d60 msgr2=0x7f4468105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:08.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 --2- 172.21.15.46:0/1452381236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468102d60 0x7f4468105150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f445800b0a0 tx=0x7f445800b3b0 comp rx=0 tx=0).stop 2024-03-18T01:33:08.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 -- 172.21.15.46:0/1452381236 shutdown_connections 2024-03-18T01:33:08.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 --2- 172.21.15.46:0/1452381236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4468107fc0 0x7f446810a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:08.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 --2- 172.21.15.46:0/1452381236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468102d60 0x7f4468105150 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:08.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 --2- 172.21.15.46:0/1452381236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4468105690 0x7f4468107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:08.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 -- 172.21.15.46:0/1452381236 >> 172.21.15.46:0/1452381236 conn(0x7f44680fc770 msgr2=0x7f44680febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:33:08.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 -- 172.21.15.46:0/1452381236 shutdown_connections 2024-03-18T01:33:08.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.852+0000 7f446fa96700 1 -- 172.21.15.46:0/1452381236 wait complete. 2024-03-18T01:33:08.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446fa96700 1 Processor -- start 2024-03-18T01:33:08.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446fa96700 1 -- start start 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446fa96700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4468102d60 0x7f44681026c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446fa96700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4468105690 0x7f4468100d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446fa96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468107fc0 0x7f4468101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446fa96700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f446810df20 con 0x7f4468105690 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446fa96700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f446810dda0 con 0x7f4468107fc0 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446fa96700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f446810e0a0 con 0x7f4468102d60 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446e033700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468107fc0 0x7f4468101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446e033700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468107fc0 0x7f4468101250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49120/0 (socket says 172.21.15.46:49120) 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446e033700 1 -- 172.21.15.46:0/3148516078 learned_addr learned my addr 172.21.15.46:0/3148516078 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-18T01:33:08.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446d832700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4468102d60 0x7f44681026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.853+0000 7f446e033700 1 -- 172.21.15.46:0/3148516078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4468102d60 msgr2=0x7f44681026c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f446e033700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4468102d60 0x7f44681026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f446e033700 1 -- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4468105690 msgr2=0x7f4468100d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f446e033700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4468105690 0x7f4468100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f446e033700 1 -- 172.21.15.46:0/3148516078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4468101b00 con 0x7f4468107fc0 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f446e033700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468107fc0 0x7f4468101250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f446000c8d0 tx=0x7f446000cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f4466ffd700 1 -- 172.21.15.46:0/3148516078 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460004460 con 0x7f4468107fc0 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f4466ffd700 1 -- 172.21.15.46:0/3148516078 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4460007bd0 con 0x7f4468107fc0 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44681a26d0 con 0x7f4468107fc0 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f4466ffd700 1 -- 172.21.15.46:0/3148516078 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446000f450 con 0x7f4468107fc0 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.854+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44681a2b90 con 0x7f4468107fc0 2024-03-18T01:33:08.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.855+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4468068f90 con 0x7f4468107fc0 2024-03-18T01:33:08.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.856+0000 7f4466ffd700 1 -- 172.21.15.46:0/3148516078 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f44600076f0 con 0x7f4468107fc0 2024-03-18T01:33:08.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.857+0000 7f4466ffd700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f44500a2890 0x7f44500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:08.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.857+0000 7f4466ffd700 1 -- 172.21.15.46:0/3148516078 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f44600ce200 con 0x7f4468107fc0 2024-03-18T01:33:08.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.857+0000 7f446d832700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f44500a2890 0x7f44500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:08.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.858+0000 7f446d832700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f44500a2890 0x7f44500a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f445800b830 tx=0x7f44580060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:33:08.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:08.861+0000 7f4466ffd700 1 -- 172.21.15.46:0/3148516078 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4460013030 con 0x7f4468107fc0 2024-03-18T01:33:09.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:08 smithi007 ceph-mon[31996]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:09.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:09.078+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f4468062710 con 0x7f44500a2890 2024-03-18T01:33:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:08 smithi203 ceph-mon[36345]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:09.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:08 smithi046 ceph-mon[36108]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 296 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:10.004 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi007 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi007 to 4273M 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.005 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:33:10.006 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:33:10.006 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.006 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:10.006 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.006 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:10.006 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi007 2024-03-18T01:33:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi007 to 4273M 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:10.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:10.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi007 2024-03-18T01:33:10.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi007 to 4273M 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:33:10.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:10.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:10.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:10.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:11.145 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:10 smithi046 ceph-mon[36108]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:33:11.145 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:10 smithi046 ceph-mon[36108]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:11.199 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:10 smithi007 ceph-mon[31996]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:33:11.199 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:10 smithi007 ceph-mon[31996]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:11.248 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:10 smithi203 ceph-mon[36345]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:33:11.248 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:10 smithi203 ceph-mon[36345]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:12.258 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:11 smithi046 ceph-mon[36108]: from='client.? 172.21.15.46:0/1709839087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]: dispatch 2024-03-18T01:33:12.258 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:11 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]: dispatch 2024-03-18T01:33:12.258 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:11 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]': finished 2024-03-18T01:33:12.258 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:11 smithi046 ceph-mon[36108]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T01:33:12.258 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:11 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:12.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:11 smithi007 ceph-mon[31996]: from='client.? 172.21.15.46:0/1709839087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]: dispatch 2024-03-18T01:33:12.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:11 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]: dispatch 2024-03-18T01:33:12.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:11 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]': finished 2024-03-18T01:33:12.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:11 smithi007 ceph-mon[31996]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T01:33:12.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:11 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:12.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:11 smithi203 ceph-mon[36345]: from='client.? 172.21.15.46:0/1709839087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]: dispatch 2024-03-18T01:33:12.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:11 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]: dispatch 2024-03-18T01:33:12.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:11 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d452ca28-0c10-48a8-bd4a-f01b4e4986c0"}]': finished 2024-03-18T01:33:12.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:11 smithi203 ceph-mon[36345]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T01:33:12.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:11 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:13.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:12 smithi046 ceph-mon[36108]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:13.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:12 smithi046 ceph-mon[36108]: from='client.? 172.21.15.46:0/2235152577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:33:13.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:12 smithi007 ceph-mon[31996]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:13.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:12 smithi007 ceph-mon[31996]: from='client.? 172.21.15.46:0/2235152577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:33:13.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:12 smithi203 ceph-mon[36345]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:13.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:12 smithi203 ceph-mon[36345]: from='client.? 172.21.15.46:0/2235152577' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:33:15.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:14 smithi046 ceph-mon[36108]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:15.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:14 smithi007 ceph-mon[31996]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:15.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:14 smithi203 ceph-mon[36345]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:16.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:16 smithi046 ceph-mon[36108]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:16.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:16 smithi007 ceph-mon[31996]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:16.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:16 smithi203 ceph-mon[36345]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:18.394 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:18 smithi046 ceph-mon[36108]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:18.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:18 smithi007 ceph-mon[31996]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:18 smithi203 ceph-mon[36345]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:20.474 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:20 smithi046 ceph-mon[36108]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:20.475 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T01:33:20.475 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:20.475 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:20 smithi046 ceph-mon[36108]: Deploying daemon osd.10 on smithi046 2024-03-18T01:33:20.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:20 smithi007 ceph-mon[31996]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:20.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T01:33:20.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:20.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:20 smithi007 ceph-mon[31996]: Deploying daemon osd.10 on smithi046 2024-03-18T01:33:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:20 smithi203 ceph-mon[36345]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T01:33:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:20 smithi203 ceph-mon[36345]: Deploying daemon osd.10 on smithi046 2024-03-18T01:33:22.457 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:22 smithi046 ceph-mon[36108]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:22.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:22 smithi007 ceph-mon[31996]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:22.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:22 smithi203 ceph-mon[36345]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:24.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:24 smithi046 ceph-mon[36108]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:24.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:24 smithi007 ceph-mon[31996]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:24.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:24 smithi203 ceph-mon[36345]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:26 smithi046 ceph-mon[36108]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:26.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:26 smithi007 ceph-mon[31996]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:26.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:26 smithi203 ceph-mon[36345]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:26.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:26.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:26.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:28.767 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:28 smithi046 ceph-mon[36108]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:28.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:28 smithi007 ceph-mon[31996]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:28.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:28 smithi203 ceph-mon[36345]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:29.517 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 10 on host 'smithi046' 2024-03-18T01:33:29.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.516+0000 7f4466ffd700 1 -- 172.21.15.46:0/3148516078 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f4468062710 con 0x7f44500a2890 2024-03-18T01:33:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f44500a2890 msgr2=0x7f44500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f44500a2890 0x7f44500a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f445800b830 tx=0x7f44580060c0 comp rx=0 tx=0).stop 2024-03-18T01:33:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468107fc0 msgr2=0x7f4468101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468107fc0 0x7f4468101250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f446000c8d0 tx=0x7f446000cc90 comp rx=0 tx=0).stop 2024-03-18T01:33:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 shutdown_connections 2024-03-18T01:33:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f44500a2890 0x7f44500a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468107fc0 0x7f4468101250 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4468102d60 0x7f44681026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 --2- 172.21.15.46:0/3148516078 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4468105690 0x7f4468100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.518+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 >> 172.21.15.46:0/3148516078 conn(0x7f44680fc770 msgr2=0x7f44681036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:33:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.519+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 shutdown_connections 2024-03-18T01:33:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-18T01:33:29.519+0000 7f446fa96700 1 -- 172.21.15.46:0/3148516078 wait complete. 2024-03-18T01:33:30.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:30.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:30.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T01:33:30.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T01:33:30.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:30.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:29 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:30.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:30.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T01:33:30.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T01:33:30.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:30.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:29 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:30.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:30.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T01:33:30.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T01:33:30.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:30.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:29 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:30.147 DEBUG:teuthology.orchestra.run.smithi046:osd.10> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.10.service 2024-03-18T01:33:30.150 INFO:tasks.cephadm:Deploying osd.11 on smithi203 with /dev/vg_nvme/lv_1... 2024-03-18T01:33:30.150 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T01:33:30.403 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:33:30.775 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:30 smithi203 ceph-mon[36345]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:30.776 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:30 smithi203 ceph-mon[36345]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T01:33:30.776 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:30 smithi203 ceph-mon[36345]: from='osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:33:30.776 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:30 smithi203 ceph-mon[36345]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T01:33:30.776 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:30.776 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:30 smithi203 ceph-mon[36345]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:33:30.989 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:30 smithi046 ceph-mon[36108]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:30.989 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:30 smithi046 ceph-mon[36108]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T01:33:30.989 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:30 smithi046 ceph-mon[36108]: from='osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:33:30.989 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:30 smithi046 ceph-mon[36108]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T01:33:30.989 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:30.989 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:30 smithi046 ceph-mon[36108]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:33:30.989 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:33:30 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:33:30.750+0000 7f71e3b43700 -1 osd.10 0 waiting for initial osdmap 2024-03-18T01:33:30.990 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:33:30 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:33:30.766+0000 7f71df936700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:33:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:30 smithi007 ceph-mon[31996]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:30 smithi007 ceph-mon[31996]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T01:33:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:30 smithi007 ceph-mon[31996]: from='osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:33:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:30 smithi007 ceph-mon[31996]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T01:33:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:30 smithi007 ceph-mon[31996]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-18T01:33:31.845 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:31 smithi203 ceph-mon[36345]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:33:31.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:31 smithi203 ceph-mon[36345]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T01:33:31.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:31.846 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:32.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:31 smithi046 ceph-mon[36108]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:33:32.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:31 smithi046 ceph-mon[36108]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T01:33:32.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:32.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:32.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:31 smithi007 ceph-mon[31996]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-18T01:33:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:31 smithi007 ceph-mon[31996]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T01:33:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:32.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:32.542 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T01:33:32.542 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T01:33:32.542 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-03-18T01:33:32.542 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-03-18T01:33:32.542 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011485 s, 913 MB/s 2024-03-18T01:33:32.543 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 2024-03-18T01:33:32.543 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-03-18T01:33:33.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:32 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:33:33.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:32 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:33:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:32 smithi046 ceph-mon[36108]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:32 smithi046 ceph-mon[36108]: osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] boot 2024-03-18T01:33:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:32 smithi046 ceph-mon[36108]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T01:33:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:32 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:32 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:33:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:32 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:33:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:32 smithi007 ceph-mon[31996]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:32 smithi007 ceph-mon[31996]: osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] boot 2024-03-18T01:33:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:32 smithi007 ceph-mon[31996]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T01:33:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:32 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:32 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:33:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:32 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:33:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:32 smithi203 ceph-mon[36345]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 297 MiB used, 894 GiB / 894 GiB avail 2024-03-18T01:33:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:32 smithi203 ceph-mon[36345]: osd.10 [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] boot 2024-03-18T01:33:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:32 smithi203 ceph-mon[36345]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T01:33:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:32 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T01:33:33.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-03-18T01:33:33.463 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:33:33.857 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T01:33:33.857 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:33.857 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:33.857 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:33.857 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:33.858 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:33.858 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:33.858 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:33.858 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:33.858 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:33.858 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:33.858 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:33 smithi203 ceph-mon[36345]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T01:33:34.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T01:33:34.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:34.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:34.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:34.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:34.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:34.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:34.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:34.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:34.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:34.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:34.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:33 smithi007 ceph-mon[31996]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T01:33:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T01:33:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:34.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:34.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:34.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:34.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:33 smithi046 ceph-mon[36108]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T01:33:34.859 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:34 smithi203 ceph-mon[36345]: pgmap v261: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:34.859 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:34 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi046 2024-03-18T01:33:34.859 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:34 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi046 to 4271M 2024-03-18T01:33:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:34 smithi007 ceph-mon[31996]: pgmap v261: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:34 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi046 2024-03-18T01:33:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:34 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi046 to 4271M 2024-03-18T01:33:35.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 -- 172.21.15.203:0/1610412671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103520 msgr2=0x7f17b8103980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:35.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1610412671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103520 0x7f17b8103980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009990 tx=0x7f17ac009ca0 comp rx=0 tx=0).stop 2024-03-18T01:33:35.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 -- 172.21.15.203:0/1610412671 shutdown_connections 2024-03-18T01:33:35.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1610412671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b8103ec0 0x7f17b810a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:35.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1610412671 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17b8102320 0x7f17b8102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:35.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1610412671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103520 0x7f17b8103980 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:35.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 -- 172.21.15.203:0/1610412671 >> 172.21.15.203:0/1610412671 conn(0x7f17b80fd8e0 msgr2=0x7f17b80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:33:35.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 -- 172.21.15.203:0/1610412671 shutdown_connections 2024-03-18T01:33:35.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.070+0000 7f17bcf39700 1 -- 172.21.15.203:0/1610412671 wait complete. 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17bcf39700 1 Processor -- start 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17bcf39700 1 -- start start 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17bcf39700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b8102320 0x7f17b8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17bcf39700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17b8103520 0x7f17b8198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17bcf39700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103ec0 0x7f17b819fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17bcf39700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810df80 con 0x7f17b8103520 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17bcf39700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810de00 con 0x7f17b8102320 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17bcf39700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17b810e100 con 0x7f17b8103ec0 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17b6d9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103ec0 0x7f17b819fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:35.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17b6d9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103ec0 0x7f17b819fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:33776/0 (socket says 172.21.15.203:33776) 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17b6d9d700 1 -- 172.21.15.203:0/1870679612 learned_addr learned my addr 172.21.15.203:0/1870679612 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.071+0000 7f17b659c700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b8102320 0x7f17b8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17b6d9d700 1 -- 172.21.15.203:0/1870679612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b8102320 msgr2=0x7f17b8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17b6d9d700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b8102320 0x7f17b8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17b6d9d700 1 -- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17b8103520 msgr2=0x7f17b8198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17b6d9d700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17b8103520 0x7f17b8198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17b6d9d700 1 -- 172.21.15.203:0/1870679612 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b81a01c0 con 0x7f17b8103ec0 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17b6d9d700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103ec0 0x7f17b819fb20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f17b000ef60 tx=0x7f17b000ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17a77fe700 1 -- 172.21.15.203:0/1870679612 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b00092e0 con 0x7f17b8103ec0 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17a77fe700 1 -- 172.21.15.203:0/1870679612 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b0004bb0 con 0x7f17b8103ec0 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b81a0450 con 0x7f17b8103ec0 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17a77fe700 1 -- 172.21.15.203:0/1870679612 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b0004280 con 0x7f17b8103ec0 2024-03-18T01:33:35.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.072+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b81a0a10 con 0x7f17b8103ec0 2024-03-18T01:33:35.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.073+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b8068f90 con 0x7f17b8103ec0 2024-03-18T01:33:35.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.074+0000 7f17a77fe700 1 -- 172.21.15.203:0/1870679612 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f17b001b030 con 0x7f17b8103ec0 2024-03-18T01:33:35.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.074+0000 7f17a77fe700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17a00a2890 0x7f17a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:35.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.075+0000 7f17a77fe700 1 -- 172.21.15.203:0/1870679612 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f17b00cdf90 con 0x7f17b8103ec0 2024-03-18T01:33:35.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.075+0000 7f17b659c700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17a00a2890 0x7f17a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:35.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.076+0000 7f17b659c700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17a00a2890 0x7f17a00a4d50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17a8006fd0 tx=0x7f17a8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:33:35.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.079+0000 7f17a77fe700 1 -- 172.21.15.203:0/1870679612 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17b0022020 con 0x7f17b8103ec0 2024-03-18T01:33:35.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:34 smithi046 ceph-mon[36108]: pgmap v261: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:35.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:34 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi046 2024-03-18T01:33:35.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:34 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi046 to 4271M 2024-03-18T01:33:35.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:35.288+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f17b8062710 con 0x7f17a00a2890 2024-03-18T01:33:36.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:35 smithi203 ceph-mon[36345]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T01:33:36.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:35 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:33:36.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:35 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:33:36.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:35 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:36.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:35 smithi046 ceph-mon[36108]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T01:33:36.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:35 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:33:36.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:35 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:33:36.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:35 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:36.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:35 smithi007 ceph-mon[31996]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T01:33:36.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:35 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T01:33:36.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:35 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T01:33:36.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:35 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:37.082 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:36 smithi203 ceph-mon[36345]: pgmap v264: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:37.083 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:36 smithi203 ceph-mon[36345]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:33:37.083 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:36 smithi203 ceph-mon[36345]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T01:33:37.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:36 smithi046 ceph-mon[36108]: pgmap v264: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:36 smithi046 ceph-mon[36108]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:33:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:36 smithi046 ceph-mon[36108]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T01:33:37.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:36 smithi007 ceph-mon[31996]: pgmap v264: 1 pgs: 1 peering; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:37.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:36 smithi007 ceph-mon[31996]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:33:37.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:36 smithi007 ceph-mon[31996]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T01:33:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:38 smithi203 ceph-mon[36345]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-18T01:33:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:38 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/4141335316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]: dispatch 2024-03-18T01:33:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:38 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]: dispatch 2024-03-18T01:33:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:38 smithi203 ceph-mon[36345]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]': finished 2024-03-18T01:33:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:38 smithi203 ceph-mon[36345]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T01:33:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:38 smithi203 ceph-mon[36345]: from='client.? 172.21.15.203:0/2363943723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:33:39.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:38 smithi046 ceph-mon[36108]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-18T01:33:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:38 smithi046 ceph-mon[36108]: from='client.? 172.21.15.203:0/4141335316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]: dispatch 2024-03-18T01:33:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:38 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]: dispatch 2024-03-18T01:33:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:38 smithi046 ceph-mon[36108]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]': finished 2024-03-18T01:33:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:38 smithi046 ceph-mon[36108]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T01:33:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:38 smithi046 ceph-mon[36108]: from='client.? 172.21.15.203:0/2363943723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:33:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:38 smithi007 ceph-mon[31996]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-18T01:33:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:38 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/4141335316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]: dispatch 2024-03-18T01:33:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:38 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]: dispatch 2024-03-18T01:33:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:38 smithi007 ceph-mon[31996]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b43ff2d-9d94-4fd3-9a3d-9a2da004016b"}]': finished 2024-03-18T01:33:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:38 smithi007 ceph-mon[31996]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T01:33:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:38 smithi007 ceph-mon[31996]: from='client.? 172.21.15.203:0/2363943723' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T01:33:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:40 smithi203 ceph-mon[36345]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-18T01:33:41.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:40 smithi046 ceph-mon[36108]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-18T01:33:41.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:40 smithi007 ceph-mon[31996]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-18T01:33:43.085 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:42 smithi203 ceph-mon[36345]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-18T01:33:43.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:42 smithi046 ceph-mon[36108]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-18T01:33:43.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:42 smithi007 ceph-mon[31996]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-18T01:33:45.043 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:44 smithi203 ceph-mon[36345]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:33:45.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:44 smithi046 ceph-mon[36108]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:33:45.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:44 smithi007 ceph-mon[31996]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T01:33:46.135 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:45 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T01:33:46.135 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:45 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:46.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:45 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T01:33:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:45 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:46.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:45 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T01:33:46.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:45 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:47.003 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:46 smithi203 ceph-mon[36345]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-18T01:33:47.003 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:46 smithi203 ceph-mon[36345]: Deploying daemon osd.11 on smithi203 2024-03-18T01:33:47.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:46 smithi046 ceph-mon[36108]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-18T01:33:47.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:46 smithi046 ceph-mon[36108]: Deploying daemon osd.11 on smithi203 2024-03-18T01:33:47.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:46 smithi007 ceph-mon[31996]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-18T01:33:47.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:46 smithi007 ceph-mon[31996]: Deploying daemon osd.11 on smithi203 2024-03-18T01:33:49.085 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:48 smithi203 ceph-mon[36345]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:48 smithi046 ceph-mon[36108]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:49.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:48 smithi007 ceph-mon[31996]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:50 smithi203 ceph-mon[36345]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:50 smithi046 ceph-mon[36108]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:51.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:50 smithi007 ceph-mon[31996]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:52 smithi007 ceph-mon[31996]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:52.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:52.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:52.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:52.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:52 smithi203 ceph-mon[36345]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:52.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:52.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:52.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:52.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:52 smithi046 ceph-mon[36108]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:54.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:54 smithi203 ceph-mon[36345]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:54.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:54.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:54.518 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:54 smithi046 ceph-mon[36108]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:54.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:54.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:54.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:54 smithi007 ceph-mon[31996]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:54.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:54.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:54.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:54.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.821+0000 7f17a77fe700 1 -- 172.21.15.203:0/1870679612 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f17b8062710 con 0x7f17a00a2890 2024-03-18T01:33:54.822 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 11 on host 'smithi203' 2024-03-18T01:33:54.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17a00a2890 msgr2=0x7f17a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17a00a2890 0x7f17a00a4d50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f17a8006fd0 tx=0x7f17a8008040 comp rx=0 tx=0).stop 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103ec0 msgr2=0x7f17b819fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103ec0 0x7f17b819fb20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f17b000ef60 tx=0x7f17b000ef90 comp rx=0 tx=0).stop 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 shutdown_connections 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17a00a2890 0x7f17a00a4d50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17b8103ec0 0x7f17b819fb20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17b8102320 0x7f17b8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 --2- 172.21.15.203:0/1870679612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17b8103520 0x7f17b8198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 >> 172.21.15.203:0/1870679612 conn(0x7f17b80fd8e0 msgr2=0x7f17b8106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 shutdown_connections 2024-03-18T01:33:54.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:33:54.824+0000 7f17bcf39700 1 -- 172.21.15.203:0/1870679612 wait complete. 2024-03-18T01:33:55.462 DEBUG:teuthology.orchestra.run.smithi203:osd.11> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.11.service 2024-03-18T01:33:55.464 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-18T01:33:55.464 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T01:33:55.947 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:33:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:55 smithi007 ceph-mon[31996]: from='osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T01:33:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:55 smithi007 ceph-mon[31996]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T01:33:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:55 smithi203 ceph-mon[36345]: from='osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T01:33:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:55 smithi203 ceph-mon[36345]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T01:33:56.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:33:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:55 smithi046 ceph-mon[36108]: from='osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T01:33:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:55 smithi046 ceph-mon[36108]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T01:33:56.977 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:56 smithi203 ceph-mon[36345]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:56.977 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:56 smithi203 ceph-mon[36345]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T01:33:56.977 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:56 smithi203 ceph-mon[36345]: from='osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:33:56.977 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:56 smithi203 ceph-mon[36345]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T01:33:56.978 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:56.978 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:56 smithi203 ceph-mon[36345]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:33:56.978 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:33:56 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:33:56.822+0000 7fbb0d94c700 -1 osd.11 0 waiting for initial osdmap 2024-03-18T01:33:56.978 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:33:56 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:33:56.834+0000 7fbb0973f700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T01:33:57.135 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:56 smithi007 ceph-mon[31996]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:57.135 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:56 smithi007 ceph-mon[31996]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T01:33:57.135 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:56 smithi007 ceph-mon[31996]: from='osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:33:57.135 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:56 smithi007 ceph-mon[31996]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T01:33:57.135 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:57.135 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:56 smithi007 ceph-mon[31996]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:33:57.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:56 smithi046 ceph-mon[36108]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:56 smithi046 ceph-mon[36108]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T01:33:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:56 smithi046 ceph-mon[36108]: from='osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:33:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:56 smithi046 ceph-mon[36108]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T01:33:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:56 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:56 smithi046 ceph-mon[36108]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-03-18T01:33:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:57 smithi007 ceph-mon[31996]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:33:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:57 smithi007 ceph-mon[31996]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T01:33:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:58.055 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:57 smithi203 ceph-mon[36345]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:33:58.055 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:57 smithi203 ceph-mon[36345]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T01:33:58.055 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:58.055 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:58.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:57 smithi046 ceph-mon[36108]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-03-18T01:33:58.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:57 smithi046 ceph-mon[36108]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T01:33:58.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:58.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 -- 172.21.15.7:0/1737114742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc100b80 msgr2=0x7f63fc102fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:58.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 --2- 172.21.15.7:0/1737114742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc100b80 0x7f63fc102fb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009990 tx=0x7f63ec009ca0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 -- 172.21.15.7:0/1737114742 shutdown_connections 2024-03-18T01:33:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 --2- 172.21.15.7:0/1737114742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63fc105ec0 0x7f63fc106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 --2- 172.21.15.7:0/1737114742 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc100b80 0x7f63fc102fb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 --2- 172.21.15.7:0/1737114742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63fc1034f0 0x7f63fc105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 -- 172.21.15.7:0/1737114742 >> 172.21.15.7:0/1737114742 conn(0x7f63fc0fc7a0 msgr2=0x7f63fc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:33:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 -- 172.21.15.7:0/1737114742 shutdown_connections 2024-03-18T01:33:58.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.274+0000 7f6403c00700 1 -- 172.21.15.7:0/1737114742 wait complete. 2024-03-18T01:33:58.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.275+0000 7f6403c00700 1 Processor -- start 2024-03-18T01:33:58.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.275+0000 7f6403c00700 1 -- start start 2024-03-18T01:33:58.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6403c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63fc100b80 0x7f63fc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:58.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6403c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63fc1034f0 0x7f63fc19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:58.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6403c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc105ec0 0x7f63fc1a6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:58.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6403c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc1144b0 con 0x7f63fc105ec0 2024-03-18T01:33:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6403c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc114330 con 0x7f63fc1034f0 2024-03-18T01:33:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6403c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f63fc114630 con 0x7f63fc100b80 2024-03-18T01:33:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63fc100b80 0x7f63fc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc105ec0 0x7f63fc1a6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63fc100b80 0x7f63fc19eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36338/0 (socket says 172.21.15.7:36338) 2024-03-18T01:33:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401200700 1 -- 172.21.15.7:0/1401718851 learned_addr learned my addr 172.21.15.7:0/1401718851 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:33:58.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f63f3e00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63fc1034f0 0x7f63fc19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401c00700 1 -- 172.21.15.7:0/1401718851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63fc100b80 msgr2=0x7f63fc19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63fc100b80 0x7f63fc19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401c00700 1 -- 172.21.15.7:0/1401718851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63fc1034f0 msgr2=0x7f63fc19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63fc1034f0 0x7f63fc19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.276+0000 7f6401c00700 1 -- 172.21.15.7:0/1401718851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63fc1a67c0 con 0x7f63fc105ec0 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.277+0000 7f6401c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc105ec0 0x7f63fc1a6120 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009f60 tx=0x7f63f4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.277+0000 7f63fa000700 1 -- 172.21.15.7:0/1401718851 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f40092e0 con 0x7f63fc105ec0 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.277+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63fc1a6a50 con 0x7f63fc105ec0 2024-03-18T01:33:58.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.277+0000 7f63fa000700 1 -- 172.21.15.7:0/1401718851 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f4004bb0 con 0x7f63fc105ec0 2024-03-18T01:33:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.277+0000 7f63fa000700 1 -- 172.21.15.7:0/1401718851 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4004280 con 0x7f63fc105ec0 2024-03-18T01:33:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.277+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63fc1a6f90 con 0x7f63fc105ec0 2024-03-18T01:33:58.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.278+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63fc068f90 con 0x7f63fc105ec0 2024-03-18T01:33:58.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.280+0000 7f63fa000700 1 -- 172.21.15.7:0/1401718851 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f63f401b030 con 0x7f63fc105ec0 2024-03-18T01:33:58.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.282+0000 7f63fa000700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f63e00a2890 0x7f63e00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:33:58.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.282+0000 7f63fa000700 1 -- 172.21.15.7:0/1401718851 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f63f40ce060 con 0x7f63fc105ec0 2024-03-18T01:33:58.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.282+0000 7f6401200700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f63e00a2890 0x7f63e00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:33:58.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.283+0000 7f6401200700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f63e00a2890 0x7f63e00a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00b7b0 tx=0x7f63ec00b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:33:58.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.287+0000 7f63fa000700 1 -- 172.21.15.7:0/1401718851 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f63f4022020 con 0x7f63fc105ec0 2024-03-18T01:33:58.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.519+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f63fc062ee0 con 0x7f63fc105ec0 2024-03-18T01:33:58.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.521+0000 7f63fa000700 1 -- 172.21.15.7:0/1401718851 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f63f4010040 con 0x7f63fc105ec0 2024-03-18T01:33:58.521 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:33:58.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f63e00a2890 msgr2=0x7f63e00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:58.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f63e00a2890 0x7f63e00a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00b7b0 tx=0x7f63ec00b880 comp rx=0 tx=0).stop 2024-03-18T01:33:58.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc105ec0 msgr2=0x7f63fc1a6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc105ec0 0x7f63fc1a6120 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009f60 tx=0x7f63f4009f90 comp rx=0 tx=0).stop 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 shutdown_connections 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f63e00a2890 0x7f63e00a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f63fc105ec0 0x7f63fc1a6120 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63fc100b80 0x7f63fc19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 --2- 172.21.15.7:0/1401718851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63fc1034f0 0x7f63fc19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 >> 172.21.15.7:0/1401718851 conn(0x7f63fc0fc7a0 msgr2=0x7f63fc103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 shutdown_connections 2024-03-18T01:33:58.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:33:58.525+0000 7f6403c00700 1 -- 172.21.15.7:0/1401718851 wait complete. 2024-03-18T01:33:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: purged_snaps scrub starts 2024-03-18T01:33:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: purged_snaps scrub ok 2024-03-18T01:33:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] boot 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1401718851' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: purged_snaps scrub starts 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: purged_snaps scrub ok 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] boot 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1401718851' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T01:33:59.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:59.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:59.224 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: purged_snaps scrub starts 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: purged_snaps scrub ok 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: osd.11 [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] boot 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:33:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1401718851' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T01:33:59.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:33:59.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:33:59.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:33:59.348 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1710725637,"num_in_osds":12,"osd_in_since":1710725618,"num_remapped_pgs":0} 2024-03-18T01:33:59.349 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T01:33:59.812 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:34:00.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:59 smithi007 ceph-mon[31996]: Detected new or changed devices on smithi203 2024-03-18T01:34:00.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:59 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi203 to 4270M 2024-03-18T01:34:00.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:33:59 smithi007 ceph-mon[31996]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T01:34:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:59 smithi203 ceph-mon[36345]: Detected new or changed devices on smithi203 2024-03-18T01:34:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:59 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi203 to 4270M 2024-03-18T01:34:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:33:59 smithi203 ceph-mon[36345]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T01:34:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:59 smithi046 ceph-mon[36108]: Detected new or changed devices on smithi203 2024-03-18T01:34:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:59 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi203 to 4270M 2024-03-18T01:34:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:33:59 smithi046 ceph-mon[36108]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T01:34:01.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:00 smithi007 ceph-mon[31996]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:00 smithi203 ceph-mon[36345]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:01.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:00 smithi046 ceph-mon[36108]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:02.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 -- 172.21.15.7:0/2936728705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd18101b70 msgr2=0x7fdd1810e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:02.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/2936728705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd18101b70 0x7fdd1810e050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000b0a0 tx=0x7fdd1000b3b0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 -- 172.21.15.7:0/2936728705 shutdown_connections 2024-03-18T01:34:02.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/2936728705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd18101b70 0x7fdd1810e050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/2936728705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd18107210 0x7fdd181075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/2936728705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd181011d0 0x7fdd18101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 -- 172.21.15.7:0/2936728705 >> 172.21.15.7:0/2936728705 conn(0x7fdd18078100 msgr2=0x7fdd18078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:02.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 -- 172.21.15.7:0/2936728705 shutdown_connections 2024-03-18T01:34:02.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.035+0000 7fdd1fa00700 1 -- 172.21.15.7:0/2936728705 wait complete. 2024-03-18T01:34:02.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.036+0000 7fdd1fa00700 1 Processor -- start 2024-03-18T01:34:02.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1fa00700 1 -- start start 2024-03-18T01:34:02.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1fa00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd181011d0 0x7fdd1819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:02.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1fa00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd18101b70 0x7fdd1819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:02.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1fa00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd18107210 0x7fdd18198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:02.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1fa00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdd181114e0 con 0x7fdd18101b70 2024-03-18T01:34:02.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1fa00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd18111360 con 0x7fdd18107210 2024-03-18T01:34:02.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1fa00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd18111660 con 0x7fdd181011d0 2024-03-18T01:34:02.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1d000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd181011d0 0x7fdd1819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:02.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1d000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd181011d0 0x7fdd1819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36348/0 (socket says 172.21.15.7:36348) 2024-03-18T01:34:02.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd17e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd18101b70 0x7fdd1819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:02.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1d000700 1 -- 172.21.15.7:0/1685505084 learned_addr learned my addr 172.21.15.7:0/1685505084 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:02.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.037+0000 7fdd1da00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd18107210 0x7fdd18198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:02.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.038+0000 7fdd1d000700 1 -- 172.21.15.7:0/1685505084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd18107210 msgr2=0x7fdd18198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:02.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.038+0000 7fdd1d000700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd18107210 0x7fdd18198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.038+0000 7fdd1d000700 1 -- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd18101b70 msgr2=0x7fdd1819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:02.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.038+0000 7fdd1d000700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd18101b70 0x7fdd1819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.038+0000 7fdd1d000700 1 -- 172.21.15.7:0/1685505084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd18198fd0 con 0x7fdd181011d0 2024-03-18T01:34:02.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.038+0000 7fdd1d000700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd181011d0 0x7fdd1819e6b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00e9c0 tx=0x7fdd0c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:02.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.038+0000 7fdd15600700 1 -- 172.21.15.7:0/1685505084 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00cbd0 con 0x7fdd181011d0 2024-03-18T01:34:02.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.038+0000 7fdd15600700 1 -- 172.21.15.7:0/1685505084 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd0c00ee60 con 0x7fdd181011d0 2024-03-18T01:34:02.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.039+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd181992c0 con 0x7fdd181011d0 2024-03-18T01:34:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.039+0000 7fdd15600700 1 -- 172.21.15.7:0/1685505084 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c005630 con 0x7fdd181011d0 2024-03-18T01:34:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.039+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd181a5580 con 0x7fdd181011d0 2024-03-18T01:34:02.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.040+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd18068f90 con 0x7fdd181011d0 2024-03-18T01:34:02.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.042+0000 7fdd15600700 1 -- 172.21.15.7:0/1685505084 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fdd0c004750 con 0x7fdd181011d0 2024-03-18T01:34:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.043+0000 7fdd15600700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd000a2890 0x7fdd000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.043+0000 7fdd15600700 1 -- 172.21.15.7:0/1685505084 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdd0c0ce890 con 0x7fdd181011d0 2024-03-18T01:34:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.043+0000 7fdd17e00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd000a2890 0x7fdd000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:02.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.044+0000 7fdd17e00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd000a2890 0x7fdd000a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdd18078670 tx=0x7fdd04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:02.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.047+0000 7fdd15600700 1 -- 172.21.15.7:0/1685505084 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd0c09bd30 con 0x7fdd181011d0 2024-03-18T01:34:02.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.272+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd181a57b0 con 0x7fdd181011d0 2024-03-18T01:34:02.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.273+0000 7fdd15600700 1 -- 172.21.15.7:0/1685505084 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18340 (secure 0 0 0) 0x7fdd0c09b630 con 0x7fdd181011d0 2024-03-18T01:34:02.274 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:02.275 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":81,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:33:58.842081+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:34:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.276+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd000a2890 msgr2=0x7fdd000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.276+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd000a2890 0x7fdd000a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdd18078670 tx=0x7fdd04009260 comp rx=0 tx=0).stop 2024-03-18T01:34:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.276+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd181011d0 msgr2=0x7fdd1819e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.276+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd181011d0 0x7fdd1819e6b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00e9c0 tx=0x7fdd0c00ecd0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.277+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 shutdown_connections 2024-03-18T01:34:02.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.277+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd000a2890 0x7fdd000a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.277+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd18107210 0x7fdd18198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.277+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd181011d0 0x7fdd1819e6b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.277+0000 7fdd1fa00700 1 --2- 172.21.15.7:0/1685505084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd18101b70 0x7fdd1819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.277+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 >> 172.21.15.7:0/1685505084 conn(0x7fdd18078100 msgr2=0x7fdd181004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.277+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 shutdown_connections 2024-03-18T01:34:02.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:02.277+0000 7fdd1fa00700 1 -- 172.21.15.7:0/1685505084 wait complete. 2024-03-18T01:34:02.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:02 smithi007 ceph-mon[31996]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:02.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:02 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1685505084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:03.030 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T01:34:03.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T01:34:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:02 smithi203 ceph-mon[36345]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:02 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1685505084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:03.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:02 smithi046 ceph-mon[36108]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:02 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1685505084' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:03.514 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:34:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:04 smithi203 ceph-mon[36345]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:05.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:04 smithi007 ceph-mon[31996]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:04 smithi046 ceph-mon[36108]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:05.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 -- 172.21.15.7:0/1643972933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24010ecd0 msgr2=0x7fd2401110c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:05.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 --2- 172.21.15.7:0/1643972933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24010ecd0 0x7fd2401110c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd2380098f0 tx=0x7fd238009c00 comp rx=0 tx=0).stop 2024-03-18T01:34:05.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 -- 172.21.15.7:0/1643972933 shutdown_connections 2024-03-18T01:34:05.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 --2- 172.21.15.7:0/1643972933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd24010ecd0 0x7fd2401110c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 --2- 172.21.15.7:0/1643972933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd240074500 0x7fd2400748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 --2- 172.21.15.7:0/1643972933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd240074e20 0x7fd24010e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 -- 172.21.15.7:0/1643972933 >> 172.21.15.7:0/1643972933 conn(0x7fd240079430 msgr2=0x7fd240079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:05.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 -- 172.21.15.7:0/1643972933 shutdown_connections 2024-03-18T01:34:05.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.719+0000 7fd247200700 1 -- 172.21.15.7:0/1643972933 wait complete. 2024-03-18T01:34:05.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.720+0000 7fd247200700 1 Processor -- start 2024-03-18T01:34:05.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.720+0000 7fd247200700 1 -- start start 2024-03-18T01:34:05.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd247200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd240074500 0x7fd2401a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:05.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd247200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd240074e20 0x7fd2401a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:05.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd247200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd24010ecd0 0x7fd24019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:05.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd247200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd240114930 con 0x7fd240074e20 2024-03-18T01:34:05.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd247200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd2401147b0 con 0x7fd240074500 2024-03-18T01:34:05.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd247200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd240114ab0 con 0x7fd24010ecd0 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd237400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd240074e20 0x7fd2401a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd237400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd240074e20 0x7fd2401a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:55350/0 (socket says 172.21.15.7:55350) 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd245200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd24010ecd0 0x7fd24019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd237400700 1 -- 172.21.15.7:0/302589600 learned_addr learned my addr 172.21.15.7:0/302589600 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.721+0000 7fd23fe00700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd240074500 0x7fd2401a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.722+0000 7fd237400700 1 -- 172.21.15.7:0/302589600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd24010ecd0 msgr2=0x7fd24019ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.722+0000 7fd237400700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd24010ecd0 0x7fd24019ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.722+0000 7fd237400700 1 -- 172.21.15.7:0/302589600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd240074500 msgr2=0x7fd2401a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.722+0000 7fd237400700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd240074500 0x7fd2401a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.722+0000 7fd237400700 1 -- 172.21.15.7:0/302589600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24019b290 con 0x7fd240074e20 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.722+0000 7fd245200700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd24010ecd0 0x7fd24019ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:05.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.722+0000 7fd237400700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd240074e20 0x7fd2401a0e60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd22800e3c0 tx=0x7fd22800e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:05.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.722+0000 7fd23d600700 1 -- 172.21.15.7:0/302589600 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22800ee40 con 0x7fd240074e20 2024-03-18T01:34:05.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.723+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd24019b520 con 0x7fd240074e20 2024-03-18T01:34:05.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.723+0000 7fd23d600700 1 -- 172.21.15.7:0/302589600 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd228003a00 con 0x7fd240074e20 2024-03-18T01:34:05.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.723+0000 7fd23d600700 1 -- 172.21.15.7:0/302589600 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd228004790 con 0x7fd240074e20 2024-03-18T01:34:05.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.723+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2401a77f0 con 0x7fd240074e20 2024-03-18T01:34:05.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.724+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd240068f90 con 0x7fd240074e20 2024-03-18T01:34:05.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.726+0000 7fd23d600700 1 -- 172.21.15.7:0/302589600 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fd22801b030 con 0x7fd240074e20 2024-03-18T01:34:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.727+0000 7fd23d600700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2240a2890 0x7fd2240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.727+0000 7fd23d600700 1 -- 172.21.15.7:0/302589600 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd2280cdf90 con 0x7fd240074e20 2024-03-18T01:34:05.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.728+0000 7fd23fe00700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2240a2890 0x7fd2240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:05.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.729+0000 7fd23fe00700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2240a2890 0x7fd2240a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd230009c60 tx=0x7fd230009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:05.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.737+0000 7fd23d600700 1 -- 172.21.15.7:0/302589600 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd228022060 con 0x7fd240074e20 2024-03-18T01:34:05.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.967+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2401a79c0 con 0x7fd240074e20 2024-03-18T01:34:05.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.968+0000 7fd23d600700 1 -- 172.21.15.7:0/302589600 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fd228010040 con 0x7fd240074e20 2024-03-18T01:34:05.968 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:34:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2240a2890 msgr2=0x7fd2240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:05.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2240a2890 0x7fd2240a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd230009c60 tx=0x7fd230009260 comp rx=0 tx=0).stop 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd240074e20 msgr2=0x7fd2401a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd240074e20 0x7fd2401a0e60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd22800e3c0 tx=0x7fd22800e6d0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 shutdown_connections 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd2240a2890 0x7fd2240a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd24010ecd0 0x7fd24019ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd240074500 0x7fd2401a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 --2- 172.21.15.7:0/302589600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd240074e20 0x7fd2401a0e60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 >> 172.21.15.7:0/302589600 conn(0x7fd240079430 msgr2=0x7fd2400ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 shutdown_connections 2024-03-18T01:34:05.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:05.971+0000 7fd247200700 1 -- 172.21.15.7:0/302589600 wait complete. 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.a on smithi007 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.d on smithi007 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.g on smithi007 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.j on smithi007 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.b on smithi046 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.e on smithi046 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.h on smithi046 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.k on smithi046 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.c on smithi203 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.f on smithi203 2024-03-18T01:34:06.753 INFO:tasks.cephadm:Adding mds.i on smithi203 2024-03-18T01:34:06.754 INFO:tasks.cephadm:Adding mds.l on smithi203 2024-03-18T01:34:06.754 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l' 2024-03-18T01:34:07.008 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:34:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:06 smithi007 ceph-mon[31996]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:06 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/302589600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:07.130 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:06 smithi203 ceph-mon[36345]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:07.131 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:06 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/302589600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:06 smithi046 ceph-mon[36108]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:06 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/302589600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:08.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 -- 172.21.15.203:0/2225192252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 msgr2=0x7f874c109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:08.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 --2- 172.21.15.203:0/2225192252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 0x7f874c109650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f874400b0a0 tx=0x7f874400b3b0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 -- 172.21.15.203:0/2225192252 shutdown_connections 2024-03-18T01:34:08.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 --2- 172.21.15.203:0/2225192252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 0x7f874c109650 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 --2- 172.21.15.203:0/2225192252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f874c101200 0x7f874c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 --2- 172.21.15.203:0/2225192252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f874c102400 0x7f874c102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 -- 172.21.15.203:0/2225192252 >> 172.21.15.203:0/2225192252 conn(0x7f874c0fc7c0 msgr2=0x7f874c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 -- 172.21.15.203:0/2225192252 shutdown_connections 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 -- 172.21.15.203:0/2225192252 wait complete. 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.687+0000 7f8751317700 1 Processor -- start 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f8751317700 1 -- start start 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f8751317700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f874c101200 0x7f874c06b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f8751317700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f874c102400 0x7f874c19d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f8751317700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 0x7f874c19f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f8751317700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f874c10ce70 con 0x7f874c101200 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f8751317700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f874c10ccf0 con 0x7f874c102400 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f8751317700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f874c10cff0 con 0x7f874c102da0 2024-03-18T01:34:08.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 0x7f874c19f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 0x7f874c19f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:35372/0 (socket says 172.21.15.203:35372) 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 -- 172.21.15.203:0/2591353554 learned_addr learned my addr 172.21.15.203:0/2591353554 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874a7fc700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f874c102400 0x7f874c19d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 -- 172.21.15.203:0/2591353554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f874c102400 msgr2=0x7f874c19d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f874c102400 0x7f874c19d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 -- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f874c101200 msgr2=0x7f874c06b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f874c101200 0x7f874c06b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 -- 172.21.15.203:0/2591353554 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874c1a0140 con 0x7f874c102da0 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.688+0000 7f874b7fe700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 0x7f874c19f870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f874400b070 tx=0x7f8744009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:08.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.689+0000 7f8733fff700 1 -- 172.21.15.203:0/2591353554 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744021070 con 0x7f874c102da0 2024-03-18T01:34:08.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.689+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f874c1a03d0 con 0x7f874c102da0 2024-03-18T01:34:08.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.689+0000 7f8733fff700 1 -- 172.21.15.203:0/2591353554 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8744007d40 con 0x7f874c102da0 2024-03-18T01:34:08.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.689+0000 7f8733fff700 1 -- 172.21.15.203:0/2591353554 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874401b850 con 0x7f874c102da0 2024-03-18T01:34:08.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.689+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f874c1a0820 con 0x7f874c102da0 2024-03-18T01:34:08.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.690+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f874c02db70 con 0x7f874c102da0 2024-03-18T01:34:08.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.692+0000 7f8733fff700 1 -- 172.21.15.203:0/2591353554 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f8744024070 con 0x7f874c102da0 2024-03-18T01:34:08.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.692+0000 7f8733fff700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87340a2890 0x7f87340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:08.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.692+0000 7f8733fff700 1 -- 172.21.15.203:0/2591353554 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f874400f020 con 0x7f874c102da0 2024-03-18T01:34:08.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.693+0000 7f874affd700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87340a2890 0x7f87340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:08.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.694+0000 7f874affd700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87340a2890 0x7f87340a4d50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f874c102260 tx=0x7f873c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:08.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.695+0000 7f8733fff700 1 -- 172.21.15.203:0/2591353554 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f874402b020 con 0x7f874c102da0 2024-03-18T01:34:08.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.906+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}) -- 0x7f874c062810 con 0x7f87340a2890 2024-03-18T01:34:08.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.923+0000 7f8733fff700 1 -- 172.21.15.203:0/2591353554 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f874c062810 con 0x7f87340a2890 2024-03-18T01:34:08.924 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mds.all update... 2024-03-18T01:34:08.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87340a2890 msgr2=0x7f87340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:08.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87340a2890 0x7f87340a4d50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f874c102260 tx=0x7f873c006d20 comp rx=0 tx=0).stop 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 msgr2=0x7f874c19f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 0x7f874c19f870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f874400b070 tx=0x7f8744009800 comp rx=0 tx=0).stop 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 shutdown_connections 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87340a2890 0x7f87340a4d50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f874c102da0 0x7f874c19f870 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f874c101200 0x7f874c06b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 --2- 172.21.15.203:0/2591353554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f874c102400 0x7f874c19d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 >> 172.21.15.203:0/2591353554 conn(0x7f874c0fc7c0 msgr2=0x7f874c1056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 shutdown_connections 2024-03-18T01:34:08.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-18T01:34:08.925+0000 7f8751317700 1 -- 172.21.15.203:0/2591353554 wait complete. 2024-03-18T01:34:09.017 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:08 smithi203 ceph-mon[36345]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:09.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:08 smithi046 ceph-mon[36108]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:09.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:08 smithi007 ceph-mon[31996]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:09.547 DEBUG:teuthology.orchestra.run.smithi007:mds.a> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.a.service 2024-03-18T01:34:09.552 DEBUG:teuthology.orchestra.run.smithi007:mds.d> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.d.service 2024-03-18T01:34:09.599 DEBUG:teuthology.orchestra.run.smithi007:mds.g> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.g.service 2024-03-18T01:34:09.646 DEBUG:teuthology.orchestra.run.smithi007:mds.j> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.j.service 2024-03-18T01:34:09.692 DEBUG:teuthology.orchestra.run.smithi046:mds.b> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.b.service 2024-03-18T01:34:09.695 DEBUG:teuthology.orchestra.run.smithi046:mds.e> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.e.service 2024-03-18T01:34:09.738 DEBUG:teuthology.orchestra.run.smithi046:mds.h> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.h.service 2024-03-18T01:34:09.781 DEBUG:teuthology.orchestra.run.smithi046:mds.k> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.k.service 2024-03-18T01:34:09.824 DEBUG:teuthology.orchestra.run.smithi203:mds.c> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.c.service 2024-03-18T01:34:09.827 DEBUG:teuthology.orchestra.run.smithi203:mds.f> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.f.service 2024-03-18T01:34:09.869 DEBUG:teuthology.orchestra.run.smithi203:mds.i> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.i.service 2024-03-18T01:34:09.912 DEBUG:teuthology.orchestra.run.smithi203:mds.l> sudo journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.l.service 2024-03-18T01:34:09.955 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-18T01:34:09.955 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:34:10.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:09 smithi046 ceph-mon[36108]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:34:10.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:09 smithi046 ceph-mon[36108]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-03-18T01:34:10.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:10.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:09 smithi203 ceph-mon[36345]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:34:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:09 smithi203 ceph-mon[36345]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-03-18T01:34:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:10.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 -- 172.21.15.7:0/1632032258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24075820 msgr2=0x7fda24073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 --2- 172.21.15.7:0/1632032258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24075820 0x7fda24073c70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fda14009ca0 tx=0x7fda1402c100 comp rx=0 tx=0).stop 2024-03-18T01:34:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 -- 172.21.15.7:0/1632032258 shutdown_connections 2024-03-18T01:34:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 --2- 172.21.15.7:0/1632032258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda241096a0 0x7fda2410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 --2- 172.21.15.7:0/1632032258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda24074240 0x7fda240746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 --2- 172.21.15.7:0/1632032258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24075820 0x7fda24073c70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 -- 172.21.15.7:0/1632032258 >> 172.21.15.7:0/1632032258 conn(0x7fda240fcd50 msgr2=0x7fda240ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 -- 172.21.15.7:0/1632032258 shutdown_connections 2024-03-18T01:34:10.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.173+0000 7fda2b800640 1 -- 172.21.15.7:0/1632032258 wait complete. 2024-03-18T01:34:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.174+0000 7fda2b800640 1 Processor -- start 2024-03-18T01:34:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.174+0000 7fda2b800640 1 -- start start 2024-03-18T01:34:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda2b800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24074240 0x7fda24199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda2b800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda24075820 0x7fda2419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:10.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda2b800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda241096a0 0x7fda241a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda2b800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fda2410f620 con 0x7fda241096a0 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda2b800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fda2410f4a0 con 0x7fda24075820 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda2b800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fda2410f7a0 con 0x7fda24074240 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda28e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24074240 0x7fda24199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda28e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24074240 0x7fda24199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36030/0 (socket says 172.21.15.7:36030) 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda28e00640 1 -- 172.21.15.7:0/1872523342 learned_addr learned my addr 172.21.15.7:0/1872523342 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda1be00640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda24075820 0x7fda2419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda28e00640 1 -- 172.21.15.7:0/1872523342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda24075820 msgr2=0x7fda2419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda28e00640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda24075820 0x7fda2419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda28e00640 1 -- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda241096a0 msgr2=0x7fda241a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.175+0000 7fda29800640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda241096a0 0x7fda241a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.176+0000 7fda28e00640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda241096a0 0x7fda241a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.176+0000 7fda28e00640 1 -- 172.21.15.7:0/1872523342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda241a1940 con 0x7fda24074240 2024-03-18T01:34:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.176+0000 7fda29800640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda241096a0 0x7fda241a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:34:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.176+0000 7fda28e00640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24074240 0x7fda24199d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda140325f0 tx=0x7fda14035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.176+0000 7fda22000640 1 -- 172.21.15.7:0/1872523342 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14039040 con 0x7fda24074240 2024-03-18T01:34:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.176+0000 7fda22000640 1 -- 172.21.15.7:0/1872523342 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda14044070 con 0x7fda24074240 2024-03-18T01:34:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.176+0000 7fda22000640 1 -- 172.21.15.7:0/1872523342 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda14004440 con 0x7fda24074240 2024-03-18T01:34:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.176+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda241a1bd0 con 0x7fda24074240 2024-03-18T01:34:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.177+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda241a2040 con 0x7fda24074240 2024-03-18T01:34:10.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.178+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda241083d0 con 0x7fda24074240 2024-03-18T01:34:10.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.180+0000 7fda22000640 1 -- 172.21.15.7:0/1872523342 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fda140045e0 con 0x7fda24074240 2024-03-18T01:34:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.181+0000 7fda22000640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda080a2880 0x7fda080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.181+0000 7fda22000640 1 -- 172.21.15.7:0/1872523342 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fda140f0c40 con 0x7fda24074240 2024-03-18T01:34:10.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.181+0000 7fda1be00640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda080a2880 0x7fda080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:10.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.182+0000 7fda1be00640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda080a2880 0x7fda080a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009bd0 tx=0x7fda0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:10.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.186+0000 7fda22000640 1 -- 172.21.15.7:0/1872523342 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fda140be0e0 con 0x7fda24074240 2024-03-18T01:34:10.297 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:09 smithi007 ceph-mon[31996]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:34:10.297 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:09 smithi007 ceph-mon[31996]: Saving service mds.all spec with placement smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-03-18T01:34:10.298 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:10.298 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:10.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.393+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda24062410 con 0x7fda24074240 2024-03-18T01:34:10.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.395+0000 7fda22000640 1 -- 172.21.15.7:0/1872523342 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18340 (secure 0 0 0) 0x7fda140bd9e0 con 0x7fda24074240 2024-03-18T01:34:10.395 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:10.395 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":81,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:33:58.842081+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:34:10.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda080a2880 msgr2=0x7fda080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda080a2880 0x7fda080a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009bd0 tx=0x7fda0c005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24074240 msgr2=0x7fda24199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24074240 0x7fda24199d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda140325f0 tx=0x7fda14035d10 comp rx=0 tx=0).stop 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 shutdown_connections 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda080a2880 0x7fda080a4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fda241096a0 0x7fda241a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda24075820 0x7fda2419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 --2- 172.21.15.7:0/1872523342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fda24074240 0x7fda24199d30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.397+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 >> 172.21.15.7:0/1872523342 conn(0x7fda240fcd50 msgr2=0x7fda241055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.398+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 shutdown_connections 2024-03-18T01:34:10.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.398+0000 7fda2b800640 1 -- 172.21.15.7:0/1872523342 wait complete. 2024-03-18T01:34:10.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T01:34:10.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:34:10.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 -- 172.21.15.7:0/1627149429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5340103840 msgr2=0x7f5340103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 --2- 172.21.15.7:0/1627149429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5340103840 0x7f5340103c60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5330009ca0 tx=0x7f533002c100 comp rx=0 tx=0).stop 2024-03-18T01:34:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 -- 172.21.15.7:0/1627149429 shutdown_connections 2024-03-18T01:34:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 --2- 172.21.15.7:0/1627149429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53401053f0 0x7f534010bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 --2- 172.21.15.7:0/1627149429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5340104a30 0x7f5340104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 --2- 172.21.15.7:0/1627149429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5340103840 0x7f5340103c60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 -- 172.21.15.7:0/1627149429 >> 172.21.15.7:0/1627149429 conn(0x7f53400fedb0 msgr2=0x7f53401011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 -- 172.21.15.7:0/1627149429 shutdown_connections 2024-03-18T01:34:10.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.650+0000 7f533fe00640 1 -- 172.21.15.7:0/1627149429 wait complete. 2024-03-18T01:34:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.651+0000 7f533fe00640 1 Processor -- start 2024-03-18T01:34:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.651+0000 7f533fe00640 1 -- start start 2024-03-18T01:34:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5340103840 0x7f5340199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5340104a30 0x7f534019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53401053f0 0x7f53401a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:10.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533fe00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f534010f4c0 con 0x7f5340103840 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533fe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f534010f340 con 0x7f53401053f0 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f534010f640 con 0x7f5340104a30 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5340104a30 0x7f534019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533ea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5340103840 0x7f5340199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5340104a30 0x7f534019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36056/0 (socket says 172.21.15.7:36056) 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533e000640 1 -- 172.21.15.7:0/3728332599 learned_addr learned my addr 172.21.15.7:0/3728332599 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533f400640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53401053f0 0x7f53401a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533e000640 1 -- 172.21.15.7:0/3728332599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53401053f0 msgr2=0x7f53401a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533e000640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53401053f0 0x7f53401a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533e000640 1 -- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5340103840 msgr2=0x7f5340199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533e000640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5340103840 0x7f5340199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533e000640 1 -- 172.21.15.7:0/3728332599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53401a1750 con 0x7f5340104a30 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.652+0000 7f533ea00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5340103840 0x7f5340199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.653+0000 7f533e000640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5340104a30 0x7f534019a070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f532800e9f0 tx=0x7f532800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.653+0000 7f5337400640 1 -- 172.21.15.7:0/3728332599 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532800cd80 con 0x7f5340104a30 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.653+0000 7f5337400640 1 -- 172.21.15.7:0/3728332599 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5328010910 con 0x7f5340104a30 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.653+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53401a1a40 con 0x7f5340104a30 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.653+0000 7f5337400640 1 -- 172.21.15.7:0/3728332599 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53280188f0 con 0x7f5340104a30 2024-03-18T01:34:10.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.653+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53401a1fe0 con 0x7f5340104a30 2024-03-18T01:34:10.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.654+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534010abe0 con 0x7f5340104a30 2024-03-18T01:34:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.657+0000 7f5337400640 1 -- 172.21.15.7:0/3728332599 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f5328010ac0 con 0x7f5340104a30 2024-03-18T01:34:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.658+0000 7f5337400640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53240a2880 0x7f53240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:10.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.658+0000 7f5337400640 1 -- 172.21.15.7:0/3728332599 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f53280ce960 con 0x7f5340104a30 2024-03-18T01:34:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.659+0000 7f533ea00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53240a2880 0x7f53240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:10.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.659+0000 7f533ea00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53240a2880 0x7f53240a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f533002c710 tx=0x7f5330032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:10.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.663+0000 7f5337400640 1 -- 172.21.15.7:0/3728332599 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f532809b7b0 con 0x7f5340104a30 2024-03-18T01:34:10.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.867+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53400623e0 con 0x7f5340104a30 2024-03-18T01:34:10.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.868+0000 7f5337400640 1 -- 172.21.15.7:0/3728332599 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f532809f850 con 0x7f5340104a30 2024-03-18T01:34:10.868 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:34:10.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53240a2880 msgr2=0x7f53240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53240a2880 0x7f53240a4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f533002c710 tx=0x7f5330032590 comp rx=0 tx=0).stop 2024-03-18T01:34:10.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5340104a30 msgr2=0x7f534019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:10.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5340104a30 0x7f534019a070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f532800e9f0 tx=0x7f532800eeb0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 shutdown_connections 2024-03-18T01:34:10.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53401053f0 0x7f53401a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5340104a30 0x7f534019a070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53240a2880 0x7f53240a4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 --2- 172.21.15.7:0/3728332599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5340103840 0x7f5340199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:10.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 >> 172.21.15.7:0/3728332599 conn(0x7f53400fedb0 msgr2=0x7f5340107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:10.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 shutdown_connections 2024-03-18T01:34:10.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:10.871+0000 7f533fe00640 1 -- 172.21.15.7:0/3728332599 wait complete. 2024-03-18T01:34:10.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:11.044 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:11.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:11.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:11.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:11.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:11.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:11.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:11.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1872523342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:11.045 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:10 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3728332599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:11.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.106+0000 7f27e9200640 1 -- 172.21.15.7:0/1747084437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27e4108e10 msgr2=0x7f27e4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:11.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.106+0000 7f27e9200640 1 --2- 172.21.15.7:0/1747084437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27e4108e10 0x7f27e4109290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009900 tx=0x7f27d002bc40 comp rx=0 tx=0).stop 2024-03-18T01:34:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.106+0000 7f27e9200640 1 -- 172.21.15.7:0/1747084437 shutdown_connections 2024-03-18T01:34:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.106+0000 7f27e9200640 1 --2- 172.21.15.7:0/1747084437 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27e41097d0 0x7f27e4110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.106+0000 7f27e9200640 1 --2- 172.21.15.7:0/1747084437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27e4108e10 0x7f27e4109290 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.106+0000 7f27e9200640 1 --2- 172.21.15.7:0/1747084437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27e4107c20 0x7f27e4108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.106+0000 7f27e9200640 1 -- 172.21.15.7:0/1747084437 >> 172.21.15.7:0/1747084437 conn(0x7f27e4076bf0 msgr2=0x7f27e4079030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.106+0000 7f27e9200640 1 -- 172.21.15.7:0/1747084437 shutdown_connections 2024-03-18T01:34:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.107+0000 7f27e9200640 1 -- 172.21.15.7:0/1747084437 wait complete. 2024-03-18T01:34:11.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.108+0000 7f27e9200640 1 Processor -- start 2024-03-18T01:34:11.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.108+0000 7f27e9200640 1 -- start start 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e9200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27e4107c20 0x7f27e419de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e9200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27e4108e10 0x7f27e419e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e9200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27e41097d0 0x7f27e41a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e9200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f27e41138f0 con 0x7f27e41097d0 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e9200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27e4113770 con 0x7f27e4108e10 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e9200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f27e4113a70 con 0x7f27e4107c20 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27e41097d0 0x7f27e41a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27e4107c20 0x7f27e419de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27e41097d0 0x7f27e41a5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36884/0 (socket says 172.21.15.7:36884) 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27e4107c20 0x7f27e419de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36074/0 (socket says 172.21.15.7:36074) 2024-03-18T01:34:11.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3400640 1 -- 172.21.15.7:0/736240556 learned_addr learned my addr 172.21.15.7:0/736240556 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27e4108e10 0x7f27e419e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:11.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3400640 1 -- 172.21.15.7:0/736240556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27e4108e10 msgr2=0x7f27e419e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3400640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27e4108e10 0x7f27e419e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3400640 1 -- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27e41097d0 msgr2=0x7f27e41a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3400640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27e41097d0 0x7f27e41a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.109+0000 7f27e3400640 1 -- 172.21.15.7:0/736240556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e41a5aa0 con 0x7f27e4107c20 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.110+0000 7f27e3400640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27e4107c20 0x7f27e419de80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f27d400e9c0 tx=0x7f27d400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.110+0000 7f27dbe00640 1 -- 172.21.15.7:0/736240556 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d400ccc0 con 0x7f27e4107c20 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.110+0000 7f27dbe00640 1 -- 172.21.15.7:0/736240556 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27d40035f0 con 0x7f27e4107c20 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.110+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e41a5d90 con 0x7f27e4107c20 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.110+0000 7f27dbe00640 1 -- 172.21.15.7:0/736240556 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d400ce60 con 0x7f27e4107c20 2024-03-18T01:34:11.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.110+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e41a61a0 con 0x7f27e4107c20 2024-03-18T01:34:11.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.111+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e410ee70 con 0x7f27e4107c20 2024-03-18T01:34:11.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.113+0000 7f27dbe00640 1 -- 172.21.15.7:0/736240556 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f27d40037a0 con 0x7f27e4107c20 2024-03-18T01:34:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.115+0000 7f27dbe00640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27c80a28a0 0x7f27c80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.115+0000 7f27dbe00640 1 -- 172.21.15.7:0/736240556 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f27d40ce9a0 con 0x7f27e4107c20 2024-03-18T01:34:11.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.115+0000 7f27e2a00640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27c80a28a0 0x7f27c80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:11.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.116+0000 7f27e2a00640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27c80a28a0 0x7f27c80a4d60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f27d002c250 tx=0x7f27d002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:11.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.119+0000 7f27dbe00640 1 -- 172.21.15.7:0/736240556 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27d409be40 con 0x7f27e4107c20 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1872523342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:10 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3728332599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:11.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:11.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1872523342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:11.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3728332599' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:11.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.366+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27e402d1b0 con 0x7f27e4107c20 2024-03-18T01:34:11.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.367+0000 7f27dbe00640 1 -- 172.21.15.7:0/736240556 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f27d409b740 con 0x7f27e4107c20 2024-03-18T01:34:11.367 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:11.367 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T01:34:11.367 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 1 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27c80a28a0 msgr2=0x7f27c80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27c80a28a0 0x7f27c80a4d60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f27d002c250 tx=0x7f27d002cf90 comp rx=0 tx=0).stop 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27e4107c20 msgr2=0x7f27e419de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27e4107c20 0x7f27e419de80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f27d400e9c0 tx=0x7f27d400edc0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 shutdown_connections 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27c80a28a0 0x7f27c80a4d60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27e41097d0 0x7f27e41a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27e4108e10 0x7f27e419e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 --2- 172.21.15.7:0/736240556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27e4107c20 0x7f27e419de80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:11.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 >> 172.21.15.7:0/736240556 conn(0x7f27e4076bf0 msgr2=0x7f27e410c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:11.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.369+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 shutdown_connections 2024-03-18T01:34:11.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:11.370+0000 7f27e9200640 1 -- 172.21.15.7:0/736240556 wait complete. 2024-03-18T01:34:12.064 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:11 smithi203 ceph-mon[36345]: Deploying daemon mds.l on smithi203 2024-03-18T01:34:12.064 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:11 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/736240556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:12.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:11 smithi046 ceph-mon[36108]: Deploying daemon mds.l on smithi203 2024-03-18T01:34:12.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:11 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/736240556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:12.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:11 smithi007 ceph-mon[31996]: Deploying daemon mds.l on smithi203 2024-03-18T01:34:12.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:11 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/736240556' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:12.391 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 systemd[1]: Starting Ceph mds.l for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:34:12.767 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 podman[67211]: 2024-03-18 01:34:12.367993176 +0000 UTC m=+0.017846199 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:12.767 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 podman[67211]: 2024-03-18 01:34:12.530669145 +0000 UTC m=+0.180522173 container create f292167f2cf18883133742c8b9858825766320ef89e754357bd9213f0a5d71a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T01:34:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:12 smithi203 ceph-mon[36345]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:12 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:boot 2024-03-18T01:34:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:12 smithi203 ceph-mon[36345]: fsmap 1 up:standby 2024-03-18T01:34:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:34:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:13.142 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 podman[67211]: 2024-03-18 01:34:12.767242049 +0000 UTC m=+0.417095052 container init f292167f2cf18883133742c8b9858825766320ef89e754357bd9213f0a5d71a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:34:13.142 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 podman[67211]: 2024-03-18 01:34:12.769995814 +0000 UTC m=+0.419848815 container start f292167f2cf18883133742c8b9858825766320ef89e754357bd9213f0a5d71a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-18T01:34:13.142 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 ceph-mds[67230]: -- 172.21.15.203:0/3436398859 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d13319a40 con 0x555d132ed000 2024-03-18T01:34:13.142 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l[67226]: starting mds.l at 2024-03-18T01:34:13.142 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 bash[67211]: f292167f2cf18883133742c8b9858825766320ef89e754357bd9213f0a5d71a5 2024-03-18T01:34:13.142 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:34:12 smithi203 systemd[1]: Started Ceph mds.l for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:13.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:12 smithi046 ceph-mon[36108]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:12 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:boot 2024-03-18T01:34:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:12 smithi046 ceph-mon[36108]: fsmap 1 up:standby 2024-03-18T01:34:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:34:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:13.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:12 smithi007 ceph-mon[31996]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:13.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:12 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:boot 2024-03-18T01:34:13.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:12 smithi007 ceph-mon[31996]: fsmap 1 up:standby 2024-03-18T01:34:13.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:34:13.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:13.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:13.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.622+0000 7f9cf5800640 1 -- 172.21.15.7:0/1048641737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00773e0 msgr2=0x7f9cf0077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:13.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.622+0000 7f9cf5800640 1 --2- 172.21.15.7:0/1048641737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00773e0 0x7f9cf0077840 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009900 tx=0x7f9cd802bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:13.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.622+0000 7f9cf5800640 1 -- 172.21.15.7:0/1048641737 shutdown_connections 2024-03-18T01:34:13.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.622+0000 7f9cf5800640 1 --2- 172.21.15.7:0/1048641737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cf0077d80 0x7f9cf0078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.622+0000 7f9cf5800640 1 --2- 172.21.15.7:0/1048641737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00773e0 0x7f9cf0077840 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.622+0000 7f9cf5800640 1 --2- 172.21.15.7:0/1048641737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf00761a0 0x7f9cf00765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.622+0000 7f9cf5800640 1 -- 172.21.15.7:0/1048641737 >> 172.21.15.7:0/1048641737 conn(0x7f9cf00fedb0 msgr2=0x7f9cf01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:13.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.623+0000 7f9cf5800640 1 -- 172.21.15.7:0/1048641737 shutdown_connections 2024-03-18T01:34:13.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.623+0000 7f9cf5800640 1 -- 172.21.15.7:0/1048641737 wait complete. 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.623+0000 7f9cf5800640 1 Processor -- start 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cf5800640 1 -- start start 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cf5800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00761a0 0x7f9cf019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cf5800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf00773e0 0x7f9cf019e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cf5800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cf0077d80 0x7f9cf01a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cf5800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf01138f0 con 0x7f9cf0077d80 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cf5800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf0113770 con 0x7f9cf00761a0 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cf5800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf0113a70 con 0x7f9cf00773e0 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cef400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf00773e0 0x7f9cf019e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:13.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cefe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00761a0 0x7f9cf019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cef400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf00773e0 0x7f9cf019e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36080/0 (socket says 172.21.15.7:36080) 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cef400640 1 -- 172.21.15.7:0/2276300666 learned_addr learned my addr 172.21.15.7:0/2276300666 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.624+0000 7f9cf4e00640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cf0077d80 0x7f9cf01a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cef400640 1 -- 172.21.15.7:0/2276300666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00761a0 msgr2=0x7f9cf019ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cef400640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00761a0 0x7f9cf019ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cef400640 1 -- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cf0077d80 msgr2=0x7f9cf01a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cef400640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cf0077d80 0x7f9cf01a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cef400640 1 -- 172.21.15.7:0/2276300666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf01a59f0 con 0x7f9cf00773e0 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cefe00640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00761a0 0x7f9cf019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:13.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cef400640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf00773e0 0x7f9cf019e310 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8000c00 tx=0x7f9cd802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:13.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cecc00640 1 -- 172.21.15.7:0/2276300666 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8043020 con 0x7f9cf00773e0 2024-03-18T01:34:13.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cecc00640 1 -- 172.21.15.7:0/2276300666 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cd80054e0 con 0x7f9cf00773e0 2024-03-18T01:34:13.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf01a5c80 con 0x7f9cf00773e0 2024-03-18T01:34:13.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.625+0000 7f9cecc00640 1 -- 172.21.15.7:0/2276300666 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8003c40 con 0x7f9cf00773e0 2024-03-18T01:34:13.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.626+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf01a61d0 con 0x7f9cf00773e0 2024-03-18T01:34:13.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.627+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cf0110190 con 0x7f9cf00773e0 2024-03-18T01:34:13.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.629+0000 7f9cecc00640 1 -- 172.21.15.7:0/2276300666 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f9cd8046020 con 0x7f9cf00773e0 2024-03-18T01:34:13.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.630+0000 7f9cecc00640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cd40a2920 0x7f9cd40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:13.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.630+0000 7f9cecc00640 1 -- 172.21.15.7:0/2276300666 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9cd80f0620 con 0x7f9cf00773e0 2024-03-18T01:34:13.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.630+0000 7f9cefe00640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cd40a2920 0x7f9cd40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:13.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.631+0000 7f9cefe00640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cd40a2920 0x7f9cd40a4de0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009fc0 tx=0x7f9ce0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:13.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.634+0000 7f9cecc00640 1 -- 172.21.15.7:0/2276300666 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9cd804d020 con 0x7f9cf00773e0 2024-03-18T01:34:13.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.880+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cf002d1b0 con 0x7f9cf00773e0 2024-03-18T01:34:13.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.881+0000 7f9cecc00640 1 -- 172.21.15.7:0/2276300666 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f9cd80bdac0 con 0x7f9cf00773e0 2024-03-18T01:34:13.881 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:13.881 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T01:34:13.882 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 2 2024-03-18T01:34:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.884+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cd40a2920 msgr2=0x7f9cd40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.884+0000 7f9cf5800640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cd40a2920 0x7f9cd40a4de0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009fc0 tx=0x7f9ce0005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.884+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf00773e0 msgr2=0x7f9cf019e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.884+0000 7f9cf5800640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf00773e0 0x7f9cf019e310 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8000c00 tx=0x7f9cd802c650 comp rx=0 tx=0).stop 2024-03-18T01:34:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.885+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 shutdown_connections 2024-03-18T01:34:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.885+0000 7f9cf5800640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cd40a2920 0x7f9cd40a4de0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.885+0000 7f9cf5800640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cf0077d80 0x7f9cf01a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.885+0000 7f9cf5800640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf00773e0 0x7f9cf019e310 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.885+0000 7f9cf5800640 1 --2- 172.21.15.7:0/2276300666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cf00761a0 0x7f9cf019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.885+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 >> 172.21.15.7:0/2276300666 conn(0x7f9cf00fedb0 msgr2=0x7f9cf01008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.885+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 shutdown_connections 2024-03-18T01:34:13.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:13.885+0000 7f9cf5800640 1 -- 172.21.15.7:0/2276300666 wait complete. 2024-03-18T01:34:14.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:14.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:14.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:14.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:14.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:13 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:14.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:13 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2276300666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:14.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:14.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:14.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:14.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:14.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:13 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:14.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:13 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2276300666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:14.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:14.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:14.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:14.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:14.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:13 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:14.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:13 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2276300666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:15.248 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:14 smithi046 ceph-mon[36108]: Deploying daemon mds.k on smithi046 2024-03-18T01:34:15.248 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:14 smithi046 ceph-mon[36108]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:15.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:14 smithi007 ceph-mon[31996]: Deploying daemon mds.k on smithi046 2024-03-18T01:34:15.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:14 smithi007 ceph-mon[31996]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:15.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:14 smithi203 ceph-mon[36345]: Deploying daemon mds.k on smithi046 2024-03-18T01:34:15.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:14 smithi203 ceph-mon[36345]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:15.519 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:34:15 smithi046 podman[65807]: 2024-03-18 01:34:15.369479264 +0000 UTC m=+0.018242792 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:15.801 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:34:15 smithi046 podman[65807]: 2024-03-18 01:34:15.531084846 +0000 UTC m=+0.179848413 container create e12538bc638e1defbc0a33966eaa58c1de6bce182d0f9cdeb6eafaf47e0d51de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-18T01:34:15.801 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:34:15 smithi046 podman[65807]: 2024-03-18 01:34:15.775126022 +0000 UTC m=+0.423889534 container init e12538bc638e1defbc0a33966eaa58c1de6bce182d0f9cdeb6eafaf47e0d51de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-18T01:34:15.802 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:34:15 smithi046 podman[65807]: 2024-03-18 01:34:15.778021268 +0000 UTC m=+0.426784787 container start e12538bc638e1defbc0a33966eaa58c1de6bce182d0f9cdeb6eafaf47e0d51de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T01:34:15.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:16.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 -- 172.21.15.7:0/3451193898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3581055b0 msgr2=0x7fd35810be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:16.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 --2- 172.21.15.7:0/3451193898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3581055b0 0x7fd35810be40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd354009ca0 tx=0x7fd35402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 -- 172.21.15.7:0/3451193898 shutdown_connections 2024-03-18T01:34:16.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 --2- 172.21.15.7:0/3451193898 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3581055b0 0x7fd35810be40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 --2- 172.21.15.7:0/3451193898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd358104bf0 0x7fd358105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 --2- 172.21.15.7:0/3451193898 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd358103a00 0x7fd358103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 -- 172.21.15.7:0/3451193898 >> 172.21.15.7:0/3451193898 conn(0x7fd3580fef90 msgr2=0x7fd3581013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:16.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 -- 172.21.15.7:0/3451193898 shutdown_connections 2024-03-18T01:34:16.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.134+0000 7fd360800640 1 -- 172.21.15.7:0/3451193898 wait complete. 2024-03-18T01:34:16.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.135+0000 7fd360800640 1 Processor -- start 2024-03-18T01:34:16.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd360800640 1 -- start start 2024-03-18T01:34:16.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd360800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd358103a00 0x7fd358199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:16.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd360800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd358104bf0 0x7fd35819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:16.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd360800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3581055b0 0x7fd3581a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:16.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd360800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd35810f680 con 0x7fd3581055b0 2024-03-18T01:34:16.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd360800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd35810f500 con 0x7fd358104bf0 2024-03-18T01:34:16.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd360800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd35810f800 con 0x7fd358103a00 2024-03-18T01:34:16.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd35d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd358104bf0 0x7fd35819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd35d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd358104bf0 0x7fd35819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34666/0 (socket says 172.21.15.7:34666) 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd35d400640 1 -- 172.21.15.7:0/4175060618 learned_addr learned my addr 172.21.15.7:0/4175060618 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd35de00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd358103a00 0x7fd358199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.136+0000 7fd35e800640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3581055b0 0x7fd3581a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd35d400640 1 -- 172.21.15.7:0/4175060618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd358103a00 msgr2=0x7fd358199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd35d400640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd358103a00 0x7fd358199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd35d400640 1 -- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3581055b0 msgr2=0x7fd3581a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd35d400640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3581055b0 0x7fd3581a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd35d400640 1 -- 172.21.15.7:0/4175060618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3581a1890 con 0x7fd358104bf0 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd35d400640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd358104bf0 0x7fd35819a1c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd34800e9f0 tx=0x7fd34800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd352a00640 1 -- 172.21.15.7:0/4175060618 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34800cd80 con 0x7fd358104bf0 2024-03-18T01:34:16.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd352a00640 1 -- 172.21.15.7:0/4175060618 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd348005500 con 0x7fd358104bf0 2024-03-18T01:34:16.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.137+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3581a1b80 con 0x7fd358104bf0 2024-03-18T01:34:16.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.138+0000 7fd352a00640 1 -- 172.21.15.7:0/4175060618 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348010690 con 0x7fd358104bf0 2024-03-18T01:34:16.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.138+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3581a20d0 con 0x7fd358104bf0 2024-03-18T01:34:16.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.139+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd35810ad20 con 0x7fd358104bf0 2024-03-18T01:34:16.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.141+0000 7fd352a00640 1 -- 172.21.15.7:0/4175060618 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fd3480046c0 con 0x7fd358104bf0 2024-03-18T01:34:16.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.143+0000 7fd352a00640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd3400a2880 0x7fd3400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:16.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.143+0000 7fd352a00640 1 -- 172.21.15.7:0/4175060618 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd3480cea70 con 0x7fd358104bf0 2024-03-18T01:34:16.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.143+0000 7fd35de00640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd3400a2880 0x7fd3400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:16.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.144+0000 7fd35de00640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd3400a2880 0x7fd3400a4d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd35819b150 tx=0x7fd34c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:16.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.147+0000 7fd352a00640 1 -- 172.21.15.7:0/4175060618 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd34809bf10 con 0x7fd358104bf0 2024-03-18T01:34:16.268 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:34:15 smithi046 ceph-mds[65826]: -- 172.21.15.46:0/966674595 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c64daafa40 con 0x55c64da82800 2024-03-18T01:34:16.269 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:34:15 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k[65822]: starting mds.k at 2024-03-18T01:34:16.269 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:34:15 smithi046 bash[65807]: e12538bc638e1defbc0a33966eaa58c1de6bce182d0f9cdeb6eafaf47e0d51de 2024-03-18T01:34:16.269 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:34:15 smithi046 systemd[1]: Started Ceph mds.k for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:16.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.398+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd35802d1b0 con 0x7fd358104bf0 2024-03-18T01:34:16.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.399+0000 7fd352a00640 1 -- 172.21.15.7:0/4175060618 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fd34809b810 con 0x7fd358104bf0 2024-03-18T01:34:16.399 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:16.399 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T01:34:16.400 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 3 2024-03-18T01:34:16.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd3400a2880 msgr2=0x7fd3400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:16.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd3400a2880 0x7fd3400a4d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd35819b150 tx=0x7fd34c005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:16.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd358104bf0 msgr2=0x7fd35819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd358104bf0 0x7fd35819a1c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd34800e9f0 tx=0x7fd34800edf0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 shutdown_connections 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd3400a2880 0x7fd3400a4d40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd3581055b0 0x7fd3581a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd358104bf0 0x7fd35819a1c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 --2- 172.21.15.7:0/4175060618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd358103a00 0x7fd358199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 >> 172.21.15.7:0/4175060618 conn(0x7fd3580fef90 msgr2=0x7fd358107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 shutdown_connections 2024-03-18T01:34:16.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:16.402+0000 7fd360800640 1 -- 172.21.15.7:0/4175060618 wait complete. 2024-03-18T01:34:17.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:17.170 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6832/1027889667,v1:172.21.15.46:6833/1027889667] up:boot 2024-03-18T01:34:17.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: fsmap 2 up:standby 2024-03-18T01:34:17.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:34:17.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:17.171 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:17.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:17.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:16 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4175060618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:17.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6832/1027889667,v1:172.21.15.46:6833/1027889667] up:boot 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: fsmap 2 up:standby 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:16 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/4175060618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6832/1027889667,v1:172.21.15.46:6833/1027889667] up:boot 2024-03-18T01:34:17.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: fsmap 2 up:standby 2024-03-18T01:34:17.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:34:17.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:17.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:17.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:17.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:17.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:16 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4175060618' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:18.137 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:17 smithi007 ceph-mon[31996]: Deploying daemon mds.j on smithi007 2024-03-18T01:34:18.187 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:17 smithi046 ceph-mon[36108]: Deploying daemon mds.j on smithi007 2024-03-18T01:34:18.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:17 smithi203 ceph-mon[36345]: Deploying daemon mds.j on smithi007 2024-03-18T01:34:18.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.719+0000 7f3efa800640 1 -- 172.21.15.7:0/3244541912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 msgr2=0x7f3ef410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.719+0000 7f3efa800640 1 --2- 172.21.15.7:0/3244541912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 0x7f3ef410df80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009860 tx=0x7f3ee002bb70 comp rx=0 tx=0).stop 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.719+0000 7f3efa800640 1 -- 172.21.15.7:0/3244541912 shutdown_connections 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.719+0000 7f3efa800640 1 --2- 172.21.15.7:0/3244541912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 0x7f3ef410df80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.719+0000 7f3efa800640 1 --2- 172.21.15.7:0/3244541912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef410b150 0x7f3ef410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.719+0000 7f3efa800640 1 --2- 172.21.15.7:0/3244541912 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ef41087e0 0x7f3ef410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.719+0000 7f3efa800640 1 -- 172.21.15.7:0/3244541912 >> 172.21.15.7:0/3244541912 conn(0x7f3ef406e820 msgr2=0x7f3ef4070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.719+0000 7f3efa800640 1 -- 172.21.15.7:0/3244541912 shutdown_connections 2024-03-18T01:34:18.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.720+0000 7f3efa800640 1 -- 172.21.15.7:0/3244541912 wait complete. 2024-03-18T01:34:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.721+0000 7f3efa800640 1 Processor -- start 2024-03-18T01:34:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.721+0000 7f3efa800640 1 -- start start 2024-03-18T01:34:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.721+0000 7f3efa800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ef41087e0 0x7f3ef41a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:18.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.721+0000 7f3efa800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef410b150 0x7f3ef41a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.721+0000 7f3efa800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 0x7f3ef41addb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.721+0000 7f3efa800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef411c260 con 0x7f3ef41087e0 2024-03-18T01:34:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.721+0000 7f3efa800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef411c0e0 con 0x7f3ef410b150 2024-03-18T01:34:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.721+0000 7f3efa800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef411c3e0 con 0x7f3ef410db00 2024-03-18T01:34:18.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef410b150 0x7f3ef41a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef410b150 0x7f3ef41a6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34688/0 (socket says 172.21.15.7:34688) 2024-03-18T01:34:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef2a00640 1 -- 172.21.15.7:0/1127929666 learned_addr learned my addr 172.21.15.7:0/1127929666 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 0x7f3ef41addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef3400640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ef41087e0 0x7f3ef41a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef3e00640 1 -- 172.21.15.7:0/1127929666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef410b150 msgr2=0x7f3ef41a6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef3e00640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef410b150 0x7f3ef41a6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef3e00640 1 -- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ef41087e0 msgr2=0x7f3ef41a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:18.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef3e00640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ef41087e0 0x7f3ef41a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.722+0000 7f3ef3e00640 1 -- 172.21.15.7:0/1127929666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef41ae420 con 0x7f3ef410db00 2024-03-18T01:34:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.723+0000 7f3ef3e00640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 0x7f3ef41addb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009780 tx=0x7f3ee0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.723+0000 7f3eebe00640 1 -- 172.21.15.7:0/1127929666 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0041020 con 0x7f3ef410db00 2024-03-18T01:34:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.723+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef41ae6b0 con 0x7f3ef410db00 2024-03-18T01:34:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.723+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef41aebc0 con 0x7f3ef410db00 2024-03-18T01:34:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.723+0000 7f3eebe00640 1 -- 172.21.15.7:0/1127929666 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee002c9f0 con 0x7f3ef410db00 2024-03-18T01:34:18.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.723+0000 7f3eebe00640 1 -- 172.21.15.7:0/1127929666 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0039040 con 0x7f3ef410db00 2024-03-18T01:34:18.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.725+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef4068660 con 0x7f3ef410db00 2024-03-18T01:34:18.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.727+0000 7f3eebe00640 1 -- 172.21.15.7:0/1127929666 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f3ee0044020 con 0x7f3ef410db00 2024-03-18T01:34:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.728+0000 7f3eebe00640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3ed80a2800 0x7f3ed80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.728+0000 7f3eebe00640 1 -- 172.21.15.7:0/1127929666 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3ee00efa00 con 0x7f3ef410db00 2024-03-18T01:34:18.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.728+0000 7f3ef3400640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3ed80a2800 0x7f3ed80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:18.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.729+0000 7f3ef3400640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3ed80a2800 0x7f3ed80a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3eec005c40 tx=0x7f3eec00a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:18.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.732+0000 7f3eebe00640 1 -- 172.21.15.7:0/1127929666 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ee004b030 con 0x7f3ef410db00 2024-03-18T01:34:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.985+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ef40623e0 con 0x7f3ef410db00 2024-03-18T01:34:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.986+0000 7f3eebe00640 1 -- 172.21.15.7:0/1127929666 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f3ee00bcea0 con 0x7f3ef410db00 2024-03-18T01:34:18.986 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:18.986 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-18T01:34:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 3 2024-03-18T01:34:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3ed80a2800 msgr2=0x7f3ed80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3ed80a2800 0x7f3ed80a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3eec005c40 tx=0x7f3eec00a590 comp rx=0 tx=0).stop 2024-03-18T01:34:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 msgr2=0x7f3ef41addb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 0x7f3ef41addb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0009780 tx=0x7f3ee0004270 comp rx=0 tx=0).stop 2024-03-18T01:34:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 shutdown_connections 2024-03-18T01:34:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef410db00 0x7f3ef41addb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ef410b150 0x7f3ef41a6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3ed80a2800 0x7f3ed80a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 --2- 172.21.15.7:0/1127929666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ef41087e0 0x7f3ef41a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 >> 172.21.15.7:0/1127929666 conn(0x7f3ef406e820 msgr2=0x7f3ef410a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 shutdown_connections 2024-03-18T01:34:18.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:18.990+0000 7f3efa800640 1 -- 172.21.15.7:0/1127929666 wait complete. 2024-03-18T01:34:19.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:18 smithi046 ceph-mon[36108]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:19.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:18 smithi007 ceph-mon[31996]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:19.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:18 smithi203 ceph-mon[36345]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:19.660 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:34:19 smithi007 podman[72474]: 2024-03-18 01:34:19.372187061 +0000 UTC m=+0.185665158 container create 6f3aae8dc00727f1a5157192dc6b879d0f247a6467adc3b3daa1ec35428b9156 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-18T01:34:19.661 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:34:19 smithi007 podman[72474]: 2024-03-18 01:34:19.65980594 +0000 UTC m=+0.473284041 container init 6f3aae8dc00727f1a5157192dc6b879d0f247a6467adc3b3daa1ec35428b9156 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T01:34:19.930 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:34:19 smithi007 podman[72474]: 2024-03-18 01:34:19.67097592 +0000 UTC m=+0.484453987 container start 6f3aae8dc00727f1a5157192dc6b879d0f247a6467adc3b3daa1ec35428b9156 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-18T01:34:19.930 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:34:19 smithi007 ceph-mds[72493]: -- 172.21.15.7:0/285350961 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56524a803a40 con 0x56524a7d6c00 2024-03-18T01:34:19.930 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:34:19 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j[72489]: starting mds.j at 2024-03-18T01:34:19.930 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:34:19 smithi007 bash[72474]: 6f3aae8dc00727f1a5157192dc6b879d0f247a6467adc3b3daa1ec35428b9156 2024-03-18T01:34:19.930 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:34:19 smithi007 systemd[1]: Started Ceph mds.j for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:19 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1127929666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:20.302 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:19 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1127929666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:20.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:19 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1127929666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:21.010 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6834/49989672,v1:172.21.15.7:6835/49989672] up:boot 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: fsmap 3 up:standby 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:21.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:20 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.244+0000 7f048bc00640 1 -- 172.21.15.7:0/1786489564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484074240 msgr2=0x7f04840746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.245+0000 7f048bc00640 1 --2- 172.21.15.7:0/1786489564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484074240 0x7f04840746c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0470009930 tx=0x7f047002bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.245+0000 7f048bc00640 1 -- 172.21.15.7:0/1786489564 shutdown_connections 2024-03-18T01:34:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.245+0000 7f048bc00640 1 --2- 172.21.15.7:0/1786489564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f048410ddd0 0x7f04841101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.245+0000 7f048bc00640 1 --2- 172.21.15.7:0/1786489564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484074240 0x7f04840746c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.245+0000 7f048bc00640 1 --2- 172.21.15.7:0/1786489564 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0484075820 0x7f0484073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.245+0000 7f048bc00640 1 -- 172.21.15.7:0/1786489564 >> 172.21.15.7:0/1786489564 conn(0x7f04840fcd70 msgr2=0x7f04840ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:21.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.245+0000 7f048bc00640 1 -- 172.21.15.7:0/1786489564 shutdown_connections 2024-03-18T01:34:21.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.245+0000 7f048bc00640 1 -- 172.21.15.7:0/1786489564 wait complete. 2024-03-18T01:34:21.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.246+0000 7f048bc00640 1 Processor -- start 2024-03-18T01:34:21.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f048bc00640 1 -- start start 2024-03-18T01:34:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f048bc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0484074240 0x7f048419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f048bc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0484075820 0x7f048419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f048bc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048410ddd0 0x7f04841a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f048bc00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0484113a70 con 0x7f0484075820 2024-03-18T01:34:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f048bc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f04841138f0 con 0x7f048410ddd0 2024-03-18T01:34:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f048bc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0484113bf0 con 0x7f0484074240 2024-03-18T01:34:21.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f0483e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0484075820 0x7f048419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.247+0000 7f0489200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0484074240 0x7f048419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0483e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0484075820 0x7f048419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51254/0 (socket says 172.21.15.7:51254) 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0483e00640 1 -- 172.21.15.7:0/199874449 learned_addr learned my addr 172.21.15.7:0/199874449 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0489c00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048410ddd0 0x7f04841a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0489200640 1 -- 172.21.15.7:0/199874449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048410ddd0 msgr2=0x7f04841a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0489200640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048410ddd0 0x7f04841a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0489200640 1 -- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0484075820 msgr2=0x7f048419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0489200640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0484075820 0x7f048419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0489200640 1 -- 172.21.15.7:0/199874449 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04841a5d20 con 0x7f0484074240 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0483e00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0484075820 0x7f048419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:21.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0489200640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0484074240 0x7f048419e100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f047800e9f0 tx=0x7f047800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.248+0000 7f0481600640 1 -- 172.21.15.7:0/199874449 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047800ccc0 con 0x7f0484074240 2024-03-18T01:34:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.249+0000 7f0481600640 1 -- 172.21.15.7:0/199874449 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0478010910 con 0x7f0484074240 2024-03-18T01:34:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.249+0000 7f0481600640 1 -- 172.21.15.7:0/199874449 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478018960 con 0x7f0484074240 2024-03-18T01:34:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.249+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04841a5fb0 con 0x7f0484074240 2024-03-18T01:34:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.249+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04841a64c0 con 0x7f0484074240 2024-03-18T01:34:21.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.250+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0484103fa0 con 0x7f0484074240 2024-03-18T01:34:21.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.252+0000 7f0481600640 1 -- 172.21.15.7:0/199874449 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f0478010ac0 con 0x7f0484074240 2024-03-18T01:34:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.253+0000 7f0481600640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f046c0a2880 0x7f046c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.253+0000 7f0481600640 1 -- 172.21.15.7:0/199874449 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f04780ce950 con 0x7f0484074240 2024-03-18T01:34:21.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.254+0000 7f0483e00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f046c0a2880 0x7f046c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:21.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.254+0000 7f0483e00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f046c0a2880 0x7f046c0a4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f047002c8e0 tx=0x7f0470005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:21.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.258+0000 7f0481600640 1 -- 172.21.15.7:0/199874449 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f047809b7a0 con 0x7f0484074240 2024-03-18T01:34:21.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:21.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6834/49989672,v1:172.21.15.7:6835/49989672] up:boot 2024-03-18T01:34:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: fsmap 3 up:standby 2024-03-18T01:34:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T01:34:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:20 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:21.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:21.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6834/49989672,v1:172.21.15.7:6835/49989672] up:boot 2024-03-18T01:34:21.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: fsmap 3 up:standby 2024-03-18T01:34:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T01:34:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:20 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:21.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.507+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0484062410 con 0x7f0484074240 2024-03-18T01:34:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.509+0000 7f0481600640 1 -- 172.21.15.7:0/199874449 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3037 (secure 0 0 0) 0x7f047809f840 con 0x7f0484074240 2024-03-18T01:34:21.510 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:21.510 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-18T01:34:21.510 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 4 2024-03-18T01:34:21.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.511+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f046c0a2880 msgr2=0x7f046c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:21.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.511+0000 7f048bc00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f046c0a2880 0x7f046c0a4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f047002c8e0 tx=0x7f0470005b50 comp rx=0 tx=0).stop 2024-03-18T01:34:21.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.511+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0484074240 msgr2=0x7f048419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:21.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.511+0000 7f048bc00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0484074240 0x7f048419e100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f047800e9f0 tx=0x7f047800eeb0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.512+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 shutdown_connections 2024-03-18T01:34:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.512+0000 7f048bc00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f048410ddd0 0x7f04841a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.512+0000 7f048bc00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f046c0a2880 0x7f046c0a4d40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.512+0000 7f048bc00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0484075820 0x7f048419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.512+0000 7f048bc00640 1 --2- 172.21.15.7:0/199874449 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0484074240 0x7f048419e100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:21.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.512+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 >> 172.21.15.7:0/199874449 conn(0x7f04840fcd70 msgr2=0x7f0484101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:21.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.512+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 shutdown_connections 2024-03-18T01:34:21.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:21.512+0000 7f048bc00640 1 -- 172.21.15.7:0/199874449 wait complete. 2024-03-18T01:34:22.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:21 smithi046 ceph-mon[36108]: Deploying daemon mds.d on smithi007 2024-03-18T01:34:22.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:21 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/199874449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:22.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:21 smithi007 ceph-mon[31996]: Deploying daemon mds.d on smithi007 2024-03-18T01:34:22.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:21 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/199874449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:22.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:21 smithi203 ceph-mon[36345]: Deploying daemon mds.d on smithi007 2024-03-18T01:34:22.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:21 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/199874449' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:23.136 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:22 smithi007 ceph-mon[31996]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:23.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:22 smithi046 ceph-mon[36108]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:23.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:22 smithi203 ceph-mon[36345]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:23.513 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 systemd[1]: Starting Ceph mds.d for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:34:23.534 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:23.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 -- 172.21.15.7:0/4123452546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1053c0 msgr2=0x7ff5dc10bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:23.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/4123452546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1053c0 0x7ff5dc10bc50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009be0 tx=0x7ff5d402bef0 comp rx=0 tx=0).stop 2024-03-18T01:34:23.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 -- 172.21.15.7:0/4123452546 shutdown_connections 2024-03-18T01:34:23.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/4123452546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1053c0 0x7ff5dc10bc50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:23.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/4123452546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc104a00 0x7ff5dc104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:23.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/4123452546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc103810 0x7ff5dc103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:23.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 -- 172.21.15.7:0/4123452546 >> 172.21.15.7:0/4123452546 conn(0x7ff5dc0fed80 msgr2=0x7ff5dc1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:23.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 -- 172.21.15.7:0/4123452546 shutdown_connections 2024-03-18T01:34:23.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.765+0000 7ff5e0c00640 1 -- 172.21.15.7:0/4123452546 wait complete. 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.766+0000 7ff5e0c00640 1 Processor -- start 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.766+0000 7ff5e0c00640 1 -- start start 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5e0c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc103810 0x7ff5dc1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5e0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc104a00 0x7ff5dc199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5e0c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc1053c0 0x7ff5dc1a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5e0c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc10f4c0 con 0x7ff5dc103810 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5e0c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc10f340 con 0x7ff5dc104a00 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5e0c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc10f640 con 0x7ff5dc1053c0 2024-03-18T01:34:23.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5db400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc103810 0x7ff5dc1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5db400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc103810 0x7ff5dc1999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51264/0 (socket says 172.21.15.7:51264) 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5db400640 1 -- 172.21.15.7:0/2728287192 learned_addr learned my addr 172.21.15.7:0/2728287192 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5dbe00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc1053c0 0x7ff5dc1a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5daa00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc104a00 0x7ff5dc199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5db400640 1 -- 172.21.15.7:0/2728287192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc1053c0 msgr2=0x7ff5dc1a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5db400640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc1053c0 0x7ff5dc1a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5db400640 1 -- 172.21.15.7:0/2728287192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc104a00 msgr2=0x7ff5dc199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5db400640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc104a00 0x7ff5dc199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.767+0000 7ff5db400640 1 -- 172.21.15.7:0/2728287192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5dc1a15e0 con 0x7ff5dc103810 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.768+0000 7ff5db400640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc103810 0x7ff5dc1999c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00e9a0 tx=0x7ff5cc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.768+0000 7ff5d3e00640 1 -- 172.21.15.7:0/2728287192 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc00cd80 con 0x7ff5dc103810 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.768+0000 7ff5d3e00640 1 -- 172.21.15.7:0/2728287192 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5cc010910 con 0x7ff5dc103810 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.768+0000 7ff5d3e00640 1 -- 172.21.15.7:0/2728287192 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc018a30 con 0x7ff5dc103810 2024-03-18T01:34:23.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.768+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5dc1a18d0 con 0x7ff5dc103810 2024-03-18T01:34:23.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.768+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5dc1a1dd0 con 0x7ff5dc103810 2024-03-18T01:34:23.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.769+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5dc10aa60 con 0x7ff5dc103810 2024-03-18T01:34:23.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.771+0000 7ff5d3e00640 1 -- 172.21.15.7:0/2728287192 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7ff5cc010ac0 con 0x7ff5dc103810 2024-03-18T01:34:23.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.773+0000 7ff5d3e00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff5c00a28a0 0x7ff5c00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:23.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.774+0000 7ff5d3e00640 1 -- 172.21.15.7:0/2728287192 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff5cc0ceaf0 con 0x7ff5dc103810 2024-03-18T01:34:23.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.774+0000 7ff5daa00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff5c00a28a0 0x7ff5c00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:23.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.774+0000 7ff5daa00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff5c00a28a0 0x7ff5c00a4d60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc19aed0 tx=0x7ff5c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:23.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:23.778+0000 7ff5d3e00640 1 -- 172.21.15.7:0/2728287192 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff5cc053740 con 0x7ff5dc103810 2024-03-18T01:34:23.802 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 podman[72822]: 2024-03-18 01:34:23.418724874 +0000 UTC m=+0.039993338 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:23.802 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 podman[72822]: 2024-03-18 01:34:23.519808946 +0000 UTC m=+0.141077435 container create 0387ab145024b00de49efe003416e571d97b4b76265cdc599975d0dec8165f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-18T01:34:24.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.036+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5dc02d1b0 con 0x7ff5dc103810 2024-03-18T01:34:24.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.037+0000 7ff5d3e00640 1 -- 172.21.15.7:0/2728287192 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3859 (secure 0 0 0) 0x7ff5cc084e20 con 0x7ff5dc103810 2024-03-18T01:34:24.039 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:24.039 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-18T01:34:24.039 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 5 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff5c00a28a0 msgr2=0x7ff5c00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff5c00a28a0 0x7ff5c00a4d60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc19aed0 tx=0x7ff5c8005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc103810 msgr2=0x7ff5dc1999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc103810 0x7ff5dc1999c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00e9a0 tx=0x7ff5cc00ee60 comp rx=0 tx=0).stop 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 shutdown_connections 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5dc1053c0 0x7ff5dc1a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc104a00 0x7ff5dc199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff5c00a28a0 0x7ff5c00a4d60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 --2- 172.21.15.7:0/2728287192 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5dc103810 0x7ff5dc1999c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:24.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.040+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 >> 172.21.15.7:0/2728287192 conn(0x7ff5dc0fed80 msgr2=0x7ff5dc107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:24.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.041+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 shutdown_connections 2024-03-18T01:34:24.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:24.041+0000 7ff5e0c00640 1 -- 172.21.15.7:0/2728287192 wait complete. 2024-03-18T01:34:24.061 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 podman[72822]: 2024-03-18 01:34:23.806526542 +0000 UTC m=+0.427795011 container init 0387ab145024b00de49efe003416e571d97b4b76265cdc599975d0dec8165f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:34:24.061 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 podman[72822]: 2024-03-18 01:34:23.817118904 +0000 UTC m=+0.438387361 container start 0387ab145024b00de49efe003416e571d97b4b76265cdc599975d0dec8165f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:34:24.061 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 ceph-mds[72880]: -- 172.21.15.7:0/2522122358 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ecca0ba40 con 0x555ecc9de800 2024-03-18T01:34:24.061 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d[72857]: starting mds.d at 2024-03-18T01:34:24.061 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 bash[72822]: 0387ab145024b00de49efe003416e571d97b4b76265cdc599975d0dec8165f02 2024-03-18T01:34:24.061 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:34:23 smithi007 systemd[1]: Started Ceph mds.d for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:25.012 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:25.012 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6836/3527563344,v1:172.21.15.7:6837/3527563344] up:boot 2024-03-18T01:34:25.012 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: fsmap 4 up:standby 2024-03-18T01:34:25.012 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T01:34:25.012 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.012 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.013 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2728287192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:25.013 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.013 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:25.013 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:25.013 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6836/3527563344,v1:172.21.15.7:6837/3527563344] up:boot 2024-03-18T01:34:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: fsmap 4 up:standby 2024-03-18T01:34:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T01:34:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2728287192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:25.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:25.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:25.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:25.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6836/3527563344,v1:172.21.15.7:6837/3527563344] up:boot 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: fsmap 4 up:standby 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2728287192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:25.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:26.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:26.217 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:25 smithi046 ceph-mon[36108]: Deploying daemon mds.e on smithi046 2024-03-18T01:34:26.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:25 smithi007 ceph-mon[31996]: Deploying daemon mds.e on smithi046 2024-03-18T01:34:26.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 -- 172.21.15.7:0/3856583762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdb4109950 msgr2=0x7fcdb4110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:26.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3856583762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdb4109950 0x7fcdb4110270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009ca0 tx=0x7fcdac02c0f0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 -- 172.21.15.7:0/3856583762 shutdown_connections 2024-03-18T01:34:26.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3856583762 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdb4109950 0x7fcdb4110270 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3856583762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdb4108f90 0x7fcdb4109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3856583762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdb4107e40 0x7fcdb4108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 -- 172.21.15.7:0/3856583762 >> 172.21.15.7:0/3856583762 conn(0x7fcdb4076dd0 msgr2=0x7fcdb4079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:26.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 -- 172.21.15.7:0/3856583762 shutdown_connections 2024-03-18T01:34:26.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.304+0000 7fcdbb800640 1 -- 172.21.15.7:0/3856583762 wait complete. 2024-03-18T01:34:26.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.305+0000 7fcdbb800640 1 Processor -- start 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdbb800640 1 -- start start 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdbb800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdb4107e40 0x7fcdb419e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdbb800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdb4108f90 0x7fcdb419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdbb800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdb4109950 0x7fcdb41a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdbb800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb4113af0 con 0x7fcdb4109950 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdbb800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb4113970 con 0x7fcdb4108f90 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdbb800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb4113c70 con 0x7fcdb4107e40 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdb3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdb4108f90 0x7fcdb419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdb3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdb4108f90 0x7fcdb419e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:33984/0 (socket says 172.21.15.7:33984) 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdb3e00640 1 -- 172.21.15.7:0/3846846254 learned_addr learned my addr 172.21.15.7:0/3846846254 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdb9800640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdb4109950 0x7fcdb41a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.306+0000 7fcdb8e00640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdb4107e40 0x7fcdb419e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:26.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb3e00640 1 -- 172.21.15.7:0/3846846254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdb4107e40 msgr2=0x7fcdb419e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:26.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb3e00640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdb4107e40 0x7fcdb419e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb3e00640 1 -- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdb4109950 msgr2=0x7fcdb41a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:26.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb3e00640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdb4109950 0x7fcdb41a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb3e00640 1 -- 172.21.15.7:0/3846846254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb41a5ca0 con 0x7fcdb4108f90 2024-03-18T01:34:26.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb3e00640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdb4108f90 0x7fcdb419e5c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcda000e9c0 tx=0x7fcda000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:26.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb1600640 1 -- 172.21.15.7:0/3846846254 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda000cd80 con 0x7fcdb4108f90 2024-03-18T01:34:26.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb1600640 1 -- 172.21.15.7:0/3846846254 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcda00035f0 con 0x7fcdb4108f90 2024-03-18T01:34:26.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb41a5f90 con 0x7fcdb4108f90 2024-03-18T01:34:26.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdb1600640 1 -- 172.21.15.7:0/3846846254 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0010690 con 0x7fcdb4108f90 2024-03-18T01:34:26.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.307+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb41a6400 con 0x7fcdb4108f90 2024-03-18T01:34:26.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.309+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb410f0f0 con 0x7fcdb4108f90 2024-03-18T01:34:26.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.311+0000 7fcdb1600640 1 -- 172.21.15.7:0/3846846254 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fcda00037a0 con 0x7fcdb4108f90 2024-03-18T01:34:26.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.312+0000 7fcdb1600640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:26.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.313+0000 7fcdb1600640 1 -- 172.21.15.7:0/3846846254 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcda00ceb40 con 0x7fcdb4108f90 2024-03-18T01:34:26.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.313+0000 7fcdb8e00640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:26.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.314+0000 7fcdb8e00640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcda80095f0 tx=0x7fcda8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:26.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.317+0000 7fcdb1600640 1 -- 172.21.15.7:0/3846846254 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcda009bfe0 con 0x7fcdb4108f90 2024-03-18T01:34:26.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:25 smithi203 ceph-mon[36345]: Deploying daemon mds.e on smithi046 2024-03-18T01:34:26.519 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 systemd[1]: Starting Ceph mds.e for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:34:26.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.559+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdb4062480 con 0x7fcdb4108f90 2024-03-18T01:34:26.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.560+0000 7fcdb1600640 1 -- 172.21.15.7:0/3846846254 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3859 (secure 0 0 0) 0x7fcda009b8e0 con 0x7fcdb4108f90 2024-03-18T01:34:26.562 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:26.562 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-03-18T01:34:26.562 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 5 2024-03-18T01:34:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcd9c0a2800 msgr2=0x7fcd9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:26.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcda80095f0 tx=0x7fcda8005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdb4108f90 msgr2=0x7fcdb419e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdb4108f90 0x7fcdb419e5c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcda000e9c0 tx=0x7fcda000ee80 comp rx=0 tx=0).stop 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 shutdown_connections 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcd9c0a2800 0x7fcd9c0a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcdb4109950 0x7fcdb41a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcdb4108f90 0x7fcdb419e5c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 --2- 172.21.15.7:0/3846846254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdb4107e40 0x7fcdb419e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 >> 172.21.15.7:0/3846846254 conn(0x7fcdb4076dd0 msgr2=0x7fcdb410c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.563+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 shutdown_connections 2024-03-18T01:34:26.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:26.564+0000 7fcdbb800640 1 -- 172.21.15.7:0/3846846254 wait complete. 2024-03-18T01:34:26.888 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 podman[66127]: 2024-03-18 01:34:26.526151052 +0000 UTC m=+0.018471284 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:26.888 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 podman[66127]: 2024-03-18 01:34:26.652641067 +0000 UTC m=+0.144961285 container create f2113119d359b53f57681741274160f3c89783b0bce2cc8d086da67c92d8b238 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T01:34:26.888 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 podman[66127]: 2024-03-18 01:34:26.888644627 +0000 UTC m=+0.380964838 container init f2113119d359b53f57681741274160f3c89783b0bce2cc8d086da67c92d8b238 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-18T01:34:27.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:26 smithi046 ceph-mon[36108]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:27.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:26 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3846846254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:27.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 podman[66127]: 2024-03-18 01:34:26.891374499 +0000 UTC m=+0.383694710 container start f2113119d359b53f57681741274160f3c89783b0bce2cc8d086da67c92d8b238 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-18T01:34:27.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 ceph-mds[66147]: -- 172.21.15.46:0/3074714486 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e634e7ba40 con 0x55e634e4e800 2024-03-18T01:34:27.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e[66143]: starting mds.e at 2024-03-18T01:34:27.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 bash[66127]: f2113119d359b53f57681741274160f3c89783b0bce2cc8d086da67c92d8b238 2024-03-18T01:34:27.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:34:26 smithi046 systemd[1]: Started Ceph mds.e for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:27.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:26 smithi007 ceph-mon[31996]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:27.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:26 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3846846254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:27.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:26 smithi203 ceph-mon[36345]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:27.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:26 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3846846254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:boot 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: fsmap 5 up:standby 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:28.062 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:28.063 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: Deploying daemon mds.f on smithi203 2024-03-18T01:34:28.063 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:27 smithi203 ceph-mon[36345]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:28.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:boot 2024-03-18T01:34:28.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: fsmap 5 up:standby 2024-03-18T01:34:28.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:34:28.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:28.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:28.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:28.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: Deploying daemon mds.f on smithi203 2024-03-18T01:34:28.235 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:27 smithi007 ceph-mon[31996]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:boot 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: fsmap 5 up:standby 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: Deploying daemon mds.f on smithi203 2024-03-18T01:34:28.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:27 smithi046 ceph-mon[36108]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:28.585 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:28.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.806+0000 7fa2f9600640 1 -- 172.21.15.7:0/764130175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2f40761a0 msgr2=0x7fa2f40765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:28.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.806+0000 7fa2f9600640 1 --2- 172.21.15.7:0/764130175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2f40761a0 0x7fa2f40765c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009900 tx=0x7fa2e402bc40 comp rx=0 tx=0).stop 2024-03-18T01:34:28.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.807+0000 7fa2f9600640 1 -- 172.21.15.7:0/764130175 shutdown_connections 2024-03-18T01:34:28.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.807+0000 7fa2f9600640 1 --2- 172.21.15.7:0/764130175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f4077d80 0x7fa2f4078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:28.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.807+0000 7fa2f9600640 1 --2- 172.21.15.7:0/764130175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40773e0 0x7fa2f4077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:28.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.807+0000 7fa2f9600640 1 --2- 172.21.15.7:0/764130175 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2f40761a0 0x7fa2f40765c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:28.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.807+0000 7fa2f9600640 1 -- 172.21.15.7:0/764130175 >> 172.21.15.7:0/764130175 conn(0x7fa2f40fef90 msgr2=0x7fa2f41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:28.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.807+0000 7fa2f9600640 1 -- 172.21.15.7:0/764130175 shutdown_connections 2024-03-18T01:34:28.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.807+0000 7fa2f9600640 1 -- 172.21.15.7:0/764130175 wait complete. 2024-03-18T01:34:28.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.809+0000 7fa2f9600640 1 Processor -- start 2024-03-18T01:34:28.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.809+0000 7fa2f9600640 1 -- start start 2024-03-18T01:34:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f9600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2f40761a0 0x7fa2f419e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f9600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40773e0 0x7fa2f419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f9600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f4077d80 0x7fa2f41a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f9600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f4113ad0 con 0x7fa2f40761a0 2024-03-18T01:34:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f9600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f4113950 con 0x7fa2f40773e0 2024-03-18T01:34:28.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f9600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f4113c50 con 0x7fa2f4077d80 2024-03-18T01:34:28.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40773e0 0x7fa2f419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2f40761a0 0x7fa2f419e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40773e0 0x7fa2f419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34000/0 (socket says 172.21.15.7:34000) 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f2000640 1 -- 172.21.15.7:0/2495716376 learned_addr learned my addr 172.21.15.7:0/2495716376 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.810+0000 7fa2f3400640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f4077d80 0x7fa2f41a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2f2000640 1 -- 172.21.15.7:0/2495716376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f4077d80 msgr2=0x7fa2f41a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2f2000640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f4077d80 0x7fa2f41a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2f2000640 1 -- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2f40761a0 msgr2=0x7fa2f419e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2f2000640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2f40761a0 0x7fa2f419e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2f2000640 1 -- 172.21.15.7:0/2495716376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f41a5ce0 con 0x7fa2f40773e0 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2f2000640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40773e0 0x7fa2f419e600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000e9f0 tx=0x7fa2e000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2eb400640 1 -- 172.21.15.7:0/2495716376 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e000cd80 con 0x7fa2f40773e0 2024-03-18T01:34:28.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2eb400640 1 -- 172.21.15.7:0/2495716376 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e0010910 con 0x7fa2f40773e0 2024-03-18T01:34:28.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f41a5fd0 con 0x7fa2f40773e0 2024-03-18T01:34:28.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2eb400640 1 -- 172.21.15.7:0/2495716376 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0018990 con 0x7fa2f40773e0 2024-03-18T01:34:28.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.811+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f41a6490 con 0x7fa2f40773e0 2024-03-18T01:34:28.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.813+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f4110370 con 0x7fa2f40773e0 2024-03-18T01:34:28.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.815+0000 7fa2eb400640 1 -- 172.21.15.7:0/2495716376 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fa2e0010ac0 con 0x7fa2f40773e0 2024-03-18T01:34:28.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.816+0000 7fa2eb400640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa2d80a2800 0x7fa2d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:28.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.817+0000 7fa2eb400640 1 -- 172.21.15.7:0/2495716376 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa2e00ce990 con 0x7fa2f40773e0 2024-03-18T01:34:28.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.817+0000 7fa2f2a00640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa2d80a2800 0x7fa2d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:28.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.818+0000 7fa2f2a00640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa2d80a2800 0x7fa2d80a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa2e402c250 tx=0x7fa2e402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:28.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:28.821+0000 7fa2eb400640 1 -- 172.21.15.7:0/2495716376 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa2e009b7e0 con 0x7fa2f40773e0 2024-03-18T01:34:29.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.081+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2f40623e0 con 0x7fa2f40773e0 2024-03-18T01:34:29.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.082+0000 7fa2eb400640 1 -- 172.21.15.7:0/2495716376 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4684 (secure 0 0 0) 0x7fa2e009f880 con 0x7fa2f40773e0 2024-03-18T01:34:29.083 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:29.084 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-18T01:34:29.084 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 6 2024-03-18T01:34:29.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa2d80a2800 msgr2=0x7fa2d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:29.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa2d80a2800 0x7fa2d80a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa2e402c250 tx=0x7fa2e402cf90 comp rx=0 tx=0).stop 2024-03-18T01:34:29.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40773e0 msgr2=0x7fa2f419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:29.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40773e0 0x7fa2f419e600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2e000e9f0 tx=0x7fa2e000eeb0 comp rx=0 tx=0).stop 2024-03-18T01:34:29.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 shutdown_connections 2024-03-18T01:34:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2f4077d80 0x7fa2f41a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2f40773e0 0x7fa2f419e600 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa2d80a2800 0x7fa2d80a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 --2- 172.21.15.7:0/2495716376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa2f40761a0 0x7fa2f419e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 >> 172.21.15.7:0/2495716376 conn(0x7fa2f40fef90 msgr2=0x7fa2f4100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 shutdown_connections 2024-03-18T01:34:29.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:29.085+0000 7fa2f9600640 1 -- 172.21.15.7:0/2495716376 wait complete. 2024-03-18T01:34:29.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:29 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2495716376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:29.382 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:29 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2495716376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:29.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:29 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2495716376' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:29.641 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:34:29 smithi203 podman[67520]: 2024-03-18 01:34:29.502105815 +0000 UTC m=+0.017815338 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:30.119 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:34:29 smithi203 podman[67520]: 2024-03-18 01:34:29.817006643 +0000 UTC m=+0.332716162 container init 66bc46eea89d2ab821becba72b98940068d04c7d6bf2284d791f99ed3aeb3348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:34:30.120 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:34:29 smithi203 podman[67520]: 2024-03-18 01:34:29.819522827 +0000 UTC m=+0.335232333 container start 66bc46eea89d2ab821becba72b98940068d04c7d6bf2284d791f99ed3aeb3348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:34:30.120 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:34:29 smithi203 ceph-mds[67540]: -- 172.21.15.203:0/2625216793 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fead54da40 con 0x55fead520c00 2024-03-18T01:34:30.120 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:34:29 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f[67536]: starting mds.f at 2024-03-18T01:34:30.120 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:34:29 smithi203 bash[67520]: 66bc46eea89d2ab821becba72b98940068d04c7d6bf2284d791f99ed3aeb3348 2024-03-18T01:34:30.120 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:34:29 smithi203 systemd[1]: Started Ceph mds.f for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:30.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:30 smithi203 ceph-mon[36345]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:30.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:30.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:30.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:30.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:30 smithi046 ceph-mon[36108]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:30.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:30.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:30.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:30.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:30 smithi007 ceph-mon[31996]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:30.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:30.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:30.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:30.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:31.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.339+0000 7f50b6a00640 1 -- 172.21.15.7:0/718928443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b01027d0 msgr2=0x7f50b0102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.339+0000 7f50b6a00640 1 --2- 172.21.15.7:0/718928443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b01027d0 0x7f50b0102c50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f509c009860 tx=0x7f509c02bb70 comp rx=0 tx=0).stop 2024-03-18T01:34:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.340+0000 7f50b6a00640 1 -- 172.21.15.7:0/718928443 shutdown_connections 2024-03-18T01:34:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.340+0000 7f50b6a00640 1 --2- 172.21.15.7:0/718928443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b0103190 0x7f50b0109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.340+0000 7f50b6a00640 1 --2- 172.21.15.7:0/718928443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b01027d0 0x7f50b0102c50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.340+0000 7f50b6a00640 1 --2- 172.21.15.7:0/718928443 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b01015e0 0x7f50b0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.340+0000 7f50b6a00640 1 -- 172.21.15.7:0/718928443 >> 172.21.15.7:0/718928443 conn(0x7f50b00fcb70 msgr2=0x7f50b00fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.340+0000 7f50b6a00640 1 -- 172.21.15.7:0/718928443 shutdown_connections 2024-03-18T01:34:31.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.340+0000 7f50b6a00640 1 -- 172.21.15.7:0/718928443 wait complete. 2024-03-18T01:34:31.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.341+0000 7f50b6a00640 1 Processor -- start 2024-03-18T01:34:31.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.341+0000 7f50b6a00640 1 -- start start 2024-03-18T01:34:31.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.341+0000 7f50b6a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b01015e0 0x7f50b0075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.341+0000 7f50b6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b01027d0 0x7f50b0073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.341+0000 7f50b6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b0103190 0x7f50b0074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.341+0000 7f50b6a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f50b010d240 con 0x7f50b01015e0 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.341+0000 7f50b6a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f50b010d0c0 con 0x7f50b0103190 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.341+0000 7f50b6a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50b010d3c0 con 0x7f50b01027d0 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b01015e0 0x7f50b0075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b01015e0 0x7f50b0075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43550/0 (socket says 172.21.15.7:43550) 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 -- 172.21.15.7:0/33073064 learned_addr learned my addr 172.21.15.7:0/33073064 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b4c00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b01027d0 0x7f50b0073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b6000640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b0103190 0x7f50b0074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 -- 172.21.15.7:0/33073064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b01027d0 msgr2=0x7f50b0073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:31.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b01027d0 0x7f50b0073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 -- 172.21.15.7:0/33073064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b0103190 msgr2=0x7f50b0074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b0103190 0x7f50b0074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 -- 172.21.15.7:0/33073064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b00749a0 con 0x7f50b01015e0 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b4c00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b01027d0 0x7f50b0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.342+0000 7f50b5600640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b01015e0 0x7f50b0075590 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f50a400e9c0 tx=0x7f50a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.343+0000 7f50ae000640 1 -- 172.21.15.7:0/33073064 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a400cd80 con 0x7f50b01015e0 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.343+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b01a1920 con 0x7f50b01015e0 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.343+0000 7f50ae000640 1 -- 172.21.15.7:0/33073064 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50a40035f0 con 0x7f50b01015e0 2024-03-18T01:34:31.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.343+0000 7f50ae000640 1 -- 172.21.15.7:0/33073064 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a4010690 con 0x7f50b01015e0 2024-03-18T01:34:31.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.343+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b01a1e40 con 0x7f50b01015e0 2024-03-18T01:34:31.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.344+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b0108900 con 0x7f50b01015e0 2024-03-18T01:34:31.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.346+0000 7f50ae000640 1 -- 172.21.15.7:0/33073064 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f50a40037a0 con 0x7f50b01015e0 2024-03-18T01:34:31.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.348+0000 7f50ae000640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50980a28a0 0x7f50980a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:31.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.348+0000 7f50ae000640 1 -- 172.21.15.7:0/33073064 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f50a40ceaa0 con 0x7f50b01015e0 2024-03-18T01:34:31.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.348+0000 7f50b4c00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50980a28a0 0x7f50980a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:31.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.349+0000 7f50b4c00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50980a28a0 0x7f50980a4d60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f509c02c180 tx=0x7f509c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:31.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.353+0000 7f50ae000640 1 -- 172.21.15.7:0/33073064 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f50a409bf40 con 0x7f50b01015e0 2024-03-18T01:34:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:31 smithi203 ceph-mon[36345]: Deploying daemon mds.c on smithi203 2024-03-18T01:34:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:31 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6834/848080613,v1:172.21.15.203:6835/848080613] up:boot 2024-03-18T01:34:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:31 smithi203 ceph-mon[36345]: fsmap 6 up:standby 2024-03-18T01:34:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:34:31.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:31 smithi046 ceph-mon[36108]: Deploying daemon mds.c on smithi203 2024-03-18T01:34:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:31 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6834/848080613,v1:172.21.15.203:6835/848080613] up:boot 2024-03-18T01:34:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:31 smithi046 ceph-mon[36108]: fsmap 6 up:standby 2024-03-18T01:34:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:34:31.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:31 smithi007 ceph-mon[31996]: Deploying daemon mds.c on smithi203 2024-03-18T01:34:31.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:31 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6834/848080613,v1:172.21.15.203:6835/848080613] up:boot 2024-03-18T01:34:31.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:31 smithi007 ceph-mon[31996]: fsmap 6 up:standby 2024-03-18T01:34:31.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:34:31.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.598+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50b002d1b0 con 0x7f50b01015e0 2024-03-18T01:34:31.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.599+0000 7f50ae000640 1 -- 172.21.15.7:0/33073064 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5509 (secure 0 0 0) 0x7f50a409b840 con 0x7f50b01015e0 2024-03-18T01:34:31.601 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:31.601 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-18T01:34:31.601 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 7 2024-03-18T01:34:31.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50980a28a0 msgr2=0x7f50980a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:31.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50980a28a0 0x7f50980a4d60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f509c02c180 tx=0x7f509c02e040 comp rx=0 tx=0).stop 2024-03-18T01:34:31.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b01015e0 msgr2=0x7f50b0075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:31.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b01015e0 0x7f50b0075590 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f50a400e9c0 tx=0x7f50a400ee80 comp rx=0 tx=0).stop 2024-03-18T01:34:31.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 shutdown_connections 2024-03-18T01:34:31.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b0103190 0x7f50b0074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b01027d0 0x7f50b0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50980a28a0 0x7f50980a4d60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.602+0000 7f50b6a00640 1 --2- 172.21.15.7:0/33073064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f50b01015e0 0x7f50b0075590 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:31.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.603+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 >> 172.21.15.7:0/33073064 conn(0x7f50b00fcb70 msgr2=0x7f50b0105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:31.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.603+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 shutdown_connections 2024-03-18T01:34:31.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:31.603+0000 7f50b6a00640 1 -- 172.21.15.7:0/33073064 wait complete. 2024-03-18T01:34:32.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:32 smithi203 ceph-mon[36345]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:32.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:32 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/33073064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:32.392 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 systemd[1]: Starting Ceph mds.c for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:34:32.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:32 smithi046 ceph-mon[36108]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:32.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:32 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/33073064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:32.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:32 smithi007 ceph-mon[31996]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:32.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:32 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/33073064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:32.823 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 podman[67827]: 2024-03-18 01:34:32.429527461 +0000 UTC m=+0.017590046 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:32.823 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 podman[67827]: 2024-03-18 01:34:32.553706009 +0000 UTC m=+0.141768622 container create d0bd257a2babf70a56e47979d228b72ce390ef86d4e8fe2a8176e09f7155a8ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:34:32.823 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 podman[67827]: 2024-03-18 01:34:32.823419228 +0000 UTC m=+0.411481816 container init d0bd257a2babf70a56e47979d228b72ce390ef86d4e8fe2a8176e09f7155a8ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T01:34:33.141 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 podman[67827]: 2024-03-18 01:34:32.826184985 +0000 UTC m=+0.414247549 container start d0bd257a2babf70a56e47979d228b72ce390ef86d4e8fe2a8176e09f7155a8ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:34:33.141 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 ceph-mds[67849]: -- 172.21.15.203:0/3498498520 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602a55afa40 con 0x5602a5582800 2024-03-18T01:34:33.141 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c[67845]: starting mds.c at 2024-03-18T01:34:33.141 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 bash[67827]: d0bd257a2babf70a56e47979d228b72ce390ef86d4e8fe2a8176e09f7155a8ec 2024-03-18T01:34:33.141 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:34:32 smithi203 systemd[1]: Started Ceph mds.c for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:33.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 -- 172.21.15.7:0/2013011701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501027d0 msgr2=0x7f5350102c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:33.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 --2- 172.21.15.7:0/2013011701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501027d0 0x7f5350102c50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5340009900 tx=0x7f534002bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 -- 172.21.15.7:0/2013011701 shutdown_connections 2024-03-18T01:34:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 --2- 172.21.15.7:0/2013011701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103190 0x7f5350109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 --2- 172.21.15.7:0/2013011701 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501027d0 0x7f5350102c50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 --2- 172.21.15.7:0/2013011701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501015e0 0x7f5350101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 -- 172.21.15.7:0/2013011701 >> 172.21.15.7:0/2013011701 conn(0x7f53500fcb90 msgr2=0x7f53500fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 -- 172.21.15.7:0/2013011701 shutdown_connections 2024-03-18T01:34:33.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.856+0000 7f5356200640 1 -- 172.21.15.7:0/2013011701 wait complete. 2024-03-18T01:34:33.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.857+0000 7f5356200640 1 Processor -- start 2024-03-18T01:34:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.857+0000 7f5356200640 1 -- start start 2024-03-18T01:34:33.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f5356200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501015e0 0x7f53501999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f5356200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501027d0 0x7f5350199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f5356200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103190 0x7f53501a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f5356200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f535010d2c0 con 0x7f53501015e0 2024-03-18T01:34:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f5356200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f535010d140 con 0x7f53501027d0 2024-03-18T01:34:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f5356200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f535010d440 con 0x7f5350103190 2024-03-18T01:34:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f534fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501027d0 0x7f5350199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f5355800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103190 0x7f53501a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:33.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f534fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501027d0 0x7f5350199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34020/0 (socket says 172.21.15.7:34020) 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f534fe00640 1 -- 172.21.15.7:0/3542619868 learned_addr learned my addr 172.21.15.7:0/3542619868 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.858+0000 7f5354e00640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501015e0 0x7f53501999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534fe00640 1 -- 172.21.15.7:0/3542619868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103190 msgr2=0x7f53501a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534fe00640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103190 0x7f53501a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534fe00640 1 -- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501015e0 msgr2=0x7f53501999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534fe00640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501015e0 0x7f53501999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534fe00640 1 -- 172.21.15.7:0/3542619868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53501a1600 con 0x7f53501027d0 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f5355800640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103190 0x7f53501a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f5354e00640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501015e0 0x7f53501999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:34:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534fe00640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501027d0 0x7f5350199f20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5340000c00 tx=0x7f534002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534d600640 1 -- 172.21.15.7:0/3542619868 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5340043020 con 0x7f53501027d0 2024-03-18T01:34:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534d600640 1 -- 172.21.15.7:0/3542619868 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53400054e0 con 0x7f53501027d0 2024-03-18T01:34:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53501a1890 con 0x7f53501027d0 2024-03-18T01:34:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f534d600640 1 -- 172.21.15.7:0/3542619868 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5340003c40 con 0x7f53501027d0 2024-03-18T01:34:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.859+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53501a1de0 con 0x7f53501027d0 2024-03-18T01:34:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.861+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53501a1fe0 con 0x7f53501027d0 2024-03-18T01:34:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.862+0000 7f534d600640 1 -- 172.21.15.7:0/3542619868 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f5340046020 con 0x7f53501027d0 2024-03-18T01:34:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.863+0000 7f534d600640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53380a28a0 0x7f53380a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.864+0000 7f534d600640 1 -- 172.21.15.7:0/3542619868 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f53400f0750 con 0x7f53501027d0 2024-03-18T01:34:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.864+0000 7f5354e00640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53380a28a0 0x7f53380a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.864+0000 7f5354e00640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53380a28a0 0x7f53380a4d60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5350102630 tx=0x7f5344005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:33.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:33.868+0000 7f534d600640 1 -- 172.21.15.7:0/3542619868 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f534004d020 con 0x7f53501027d0 2024-03-18T01:34:34.117 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.116+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53501a2370 con 0x7f53501027d0 2024-03-18T01:34:34.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.117+0000 7f534d600640 1 -- 172.21.15.7:0/3542619868 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6337 (secure 0 0 0) 0x7f53400bdbf0 con 0x7f53501027d0 2024-03-18T01:34:34.119 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:34.119 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T01:34:34.119 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 8 2024-03-18T01:34:34.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53380a28a0 msgr2=0x7f53380a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:34.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53380a28a0 0x7f53380a4d60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5350102630 tx=0x7f5344005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:34.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501027d0 msgr2=0x7f5350199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:34.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501027d0 0x7f5350199f20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5340000c00 tx=0x7f534002c650 comp rx=0 tx=0).stop 2024-03-18T01:34:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 shutdown_connections 2024-03-18T01:34:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103190 0x7f53501a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501027d0 0x7f5350199f20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53380a28a0 0x7f53380a4d60 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 --2- 172.21.15.7:0/3542619868 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501015e0 0x7f53501999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.120+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 >> 172.21.15.7:0/3542619868 conn(0x7f53500fcb90 msgr2=0x7f5350073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.121+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 shutdown_connections 2024-03-18T01:34:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:34.121+0000 7f5356200640 1 -- 172.21.15.7:0/3542619868 wait complete. 2024-03-18T01:34:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: Deploying daemon mds.h on smithi046 2024-03-18T01:34:34.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:boot 2024-03-18T01:34:34.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: fsmap 7 up:standby 2024-03-18T01:34:34.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:34:34.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:33 smithi046 ceph-mon[36108]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:34.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:34.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:34.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:34.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: Deploying daemon mds.h on smithi046 2024-03-18T01:34:34.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:boot 2024-03-18T01:34:34.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: fsmap 7 up:standby 2024-03-18T01:34:34.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:34:34.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:33 smithi007 ceph-mon[31996]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: Deploying daemon mds.h on smithi046 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:boot 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: fsmap 7 up:standby 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:34:34.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:33 smithi203 ceph-mon[36345]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:35.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:34 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3542619868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:35.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:34 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3542619868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:35.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:34 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3542619868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:35.494 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 systemd[1]: Starting Ceph mds.h for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:34:35.759 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 podman[66441]: 2024-03-18 01:34:35.395823096 +0000 UTC m=+0.018087652 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:35.760 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 podman[66441]: 2024-03-18 01:34:35.51520557 +0000 UTC m=+0.137470110 container create 43371fca873109d18a91d9d1e2f368de963242a7121e10f0a914b7e1082b7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T01:34:36.019 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 podman[66441]: 2024-03-18 01:34:35.759845754 +0000 UTC m=+0.382110301 container init 43371fca873109d18a91d9d1e2f368de963242a7121e10f0a914b7e1082b7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:34:36.019 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 podman[66441]: 2024-03-18 01:34:35.762624172 +0000 UTC m=+0.384888715 container start 43371fca873109d18a91d9d1e2f368de963242a7121e10f0a914b7e1082b7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:34:36.019 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 ceph-mds[66460]: -- 172.21.15.46:0/3642618043 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5567723c1a40 con 0x556772395000 2024-03-18T01:34:36.019 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h[66456]: starting mds.h at 2024-03-18T01:34:36.019 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 bash[66441]: 43371fca873109d18a91d9d1e2f368de963242a7121e10f0a914b7e1082b7edb 2024-03-18T01:34:36.019 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:34:35 smithi046 systemd[1]: Started Ceph mds.h for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:36.145 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 -- 172.21.15.7:0/239920637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534107e40 msgr2=0x7f0534108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 --2- 172.21.15.7:0/239920637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534107e40 0x7f0534108260 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0524009900 tx=0x7f052402bc40 comp rx=0 tx=0).stop 2024-03-18T01:34:36.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 -- 172.21.15.7:0/239920637 shutdown_connections 2024-03-18T01:34:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 --2- 172.21.15.7:0/239920637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0534109950 0x7f0534110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 --2- 172.21.15.7:0/239920637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0534108f90 0x7f0534109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 --2- 172.21.15.7:0/239920637 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534107e40 0x7f0534108260 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 -- 172.21.15.7:0/239920637 >> 172.21.15.7:0/239920637 conn(0x7f0534076dd0 msgr2=0x7f0534079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 -- 172.21.15.7:0/239920637 shutdown_connections 2024-03-18T01:34:36.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.379+0000 7f0538a00640 1 -- 172.21.15.7:0/239920637 wait complete. 2024-03-18T01:34:36.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.380+0000 7f0538a00640 1 Processor -- start 2024-03-18T01:34:36.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.380+0000 7f0538a00640 1 -- start start 2024-03-18T01:34:36.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.380+0000 7f0538a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0534107e40 0x7f053419e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:36.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.380+0000 7f0538a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534108f90 0x7f053419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:36.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.380+0000 7f0538a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0534109950 0x7f05341a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.380+0000 7f0538a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0534113ac0 con 0x7f0534108f90 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.380+0000 7f0538a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0534113940 con 0x7f0534107e40 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.380+0000 7f0538a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0534113c40 con 0x7f0534109950 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0531400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534108f90 0x7f053419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0532800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0534109950 0x7f05341a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0531400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534108f90 0x7f053419e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56212/0 (socket says 172.21.15.7:56212) 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0532800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0534109950 0x7f05341a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:60116/0 (socket says 172.21.15.7:60116) 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0531400640 1 -- 172.21.15.7:0/3100098811 learned_addr learned my addr 172.21.15.7:0/3100098811 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0531e00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0534107e40 0x7f053419e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0532800640 1 -- 172.21.15.7:0/3100098811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0534107e40 msgr2=0x7f053419e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0532800640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0534107e40 0x7f053419e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0532800640 1 -- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534108f90 msgr2=0x7f053419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0532800640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534108f90 0x7f053419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0532800640 1 -- 172.21.15.7:0/3100098811 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05341a5d00 con 0x7f0534109950 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0531400640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534108f90 0x7f053419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.381+0000 7f0532800640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0534109950 0x7f05341a5690 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f052c009fd0 tx=0x7f052c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.382+0000 7f052aa00640 1 -- 172.21.15.7:0/3100098811 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c00ce20 con 0x7f0534109950 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.382+0000 7f052aa00640 1 -- 172.21.15.7:0/3100098811 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f052c005020 con 0x7f0534109950 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.382+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05341a5ff0 con 0x7f0534109950 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.382+0000 7f052aa00640 1 -- 172.21.15.7:0/3100098811 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f052c010720 con 0x7f0534109950 2024-03-18T01:34:36.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.382+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05341a6460 con 0x7f0534109950 2024-03-18T01:34:36.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.383+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f053410f0f0 con 0x7f0534109950 2024-03-18T01:34:36.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.385+0000 7f052aa00640 1 -- 172.21.15.7:0/3100098811 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f052c005540 con 0x7f0534109950 2024-03-18T01:34:36.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.386+0000 7f052aa00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f05180a2800 0x7f05180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:36.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.386+0000 7f052aa00640 1 -- 172.21.15.7:0/3100098811 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f052c0ced40 con 0x7f0534109950 2024-03-18T01:34:36.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.387+0000 7f0531e00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f05180a2800 0x7f05180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:36.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.387+0000 7f0531e00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f05180a2800 0x7f05180a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f052402c7c0 tx=0x7f0524005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:36.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.391+0000 7f052aa00640 1 -- 172.21.15.7:0/3100098811 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f052c09c1e0 con 0x7f0534109950 2024-03-18T01:34:36.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.635+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f053402d1b0 con 0x7f0534109950 2024-03-18T01:34:36.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.636+0000 7f052aa00640 1 -- 172.21.15.7:0/3100098811 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7162 (secure 0 0 0) 0x7f052c09bae0 con 0x7f0534109950 2024-03-18T01:34:36.637 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:36.637 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-18T01:34:36.638 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 9 2024-03-18T01:34:36.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f05180a2800 msgr2=0x7f05180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:36.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f05180a2800 0x7f05180a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f052402c7c0 tx=0x7f0524005fd0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0534109950 msgr2=0x7f05341a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:36.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0534109950 0x7f05341a5690 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f052c009fd0 tx=0x7f052c00ef10 comp rx=0 tx=0).stop 2024-03-18T01:34:36.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 shutdown_connections 2024-03-18T01:34:36.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0534109950 0x7f05341a5690 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f05180a2800 0x7f05180a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0534108f90 0x7f053419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 --2- 172.21.15.7:0/3100098811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0534107e40 0x7f053419e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:36.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 >> 172.21.15.7:0/3100098811 conn(0x7f0534076dd0 msgr2=0x7f053410c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:36.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 shutdown_connections 2024-03-18T01:34:36.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:36.639+0000 7f0538a00640 1 -- 172.21.15.7:0/3100098811 wait complete. 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: Deploying daemon mds.i on smithi203 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:boot 2024-03-18T01:34:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: fsmap 8 up:standby 2024-03-18T01:34:37.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:34:37.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:34:37.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:36 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3100098811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: Deploying daemon mds.i on smithi203 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:boot 2024-03-18T01:34:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: fsmap 8 up:standby 2024-03-18T01:34:37.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:34:37.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:34:37.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:36 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3100098811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: Deploying daemon mds.i on smithi203 2024-03-18T01:34:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:boot 2024-03-18T01:34:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: fsmap 8 up:standby 2024-03-18T01:34:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:34:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:34:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:36 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3100098811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:38.295 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 systemd[1]: Starting Ceph mds.i for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:34:38.561 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 podman[68138]: 2024-03-18 01:34:38.197141594 +0000 UTC m=+0.017676329 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:38.561 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 podman[68138]: 2024-03-18 01:34:38.317150389 +0000 UTC m=+0.137685169 container create b94be326eadc665ac4f60567c3632b9779ca1bb1f1ac5252e7f4510682ce532d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:34:38.561 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 podman[68138]: 2024-03-18 01:34:38.561327418 +0000 UTC m=+0.381862135 container init b94be326eadc665ac4f60567c3632b9779ca1bb1f1ac5252e7f4510682ce532d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T01:34:38.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.884+0000 7f6998c00640 1 -- 172.21.15.7:0/498640710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 msgr2=0x7f6994103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.884+0000 7f6998c00640 1 --2- 172.21.15.7:0/498640710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 0x7f6994103e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6984009860 tx=0x7f698402bb70 comp rx=0 tx=0).stop 2024-03-18T01:34:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.885+0000 7f6998c00640 1 -- 172.21.15.7:0/498640710 shutdown_connections 2024-03-18T01:34:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.885+0000 7f6998c00640 1 --2- 172.21.15.7:0/498640710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69941055b0 0x7f699410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.885+0000 7f6998c00640 1 --2- 172.21.15.7:0/498640710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994104bf0 0x7f6994105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:38.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.885+0000 7f6998c00640 1 --2- 172.21.15.7:0/498640710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 0x7f6994103e20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.885+0000 7f6998c00640 1 -- 172.21.15.7:0/498640710 >> 172.21.15.7:0/498640710 conn(0x7f69940fef90 msgr2=0x7f69941013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.885+0000 7f6998c00640 1 -- 172.21.15.7:0/498640710 shutdown_connections 2024-03-18T01:34:38.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.885+0000 7f6998c00640 1 -- 172.21.15.7:0/498640710 wait complete. 2024-03-18T01:34:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.886+0000 7f6998c00640 1 Processor -- start 2024-03-18T01:34:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.886+0000 7f6998c00640 1 -- start start 2024-03-18T01:34:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.886+0000 7f6998c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 0x7f6994199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.886+0000 7f6998c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6994104bf0 0x7f699419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.886+0000 7f6998c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69941055b0 0x7f69941a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.886+0000 7f6998c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f699410f670 con 0x7f6994104bf0 2024-03-18T01:34:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.886+0000 7f6998c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f699410f4f0 con 0x7f69941055b0 2024-03-18T01:34:38.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.886+0000 7f6998c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f699410f7f0 con 0x7f6994103a00 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6994104bf0 0x7f699419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 0x7f6994199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6994104bf0 0x7f699419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56228/0 (socket says 172.21.15.7:56228) 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991400640 1 -- 172.21.15.7:0/3780936202 learned_addr learned my addr 172.21.15.7:0/3780936202 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6992800640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69941055b0 0x7f69941a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991e00640 1 -- 172.21.15.7:0/3780936202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69941055b0 msgr2=0x7f69941a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991e00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69941055b0 0x7f69941a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991e00640 1 -- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6994104bf0 msgr2=0x7f699419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991e00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6994104bf0 0x7f699419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991e00640 1 -- 172.21.15.7:0/3780936202 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69941a1920 con 0x7f6994103a00 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.887+0000 7f6991e00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 0x7f6994199d10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6984009780 tx=0x7f6984032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:38.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.888+0000 7f698aa00640 1 -- 172.21.15.7:0/3780936202 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984041020 con 0x7f6994103a00 2024-03-18T01:34:38.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.888+0000 7f698aa00640 1 -- 172.21.15.7:0/3780936202 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f698402c9f0 con 0x7f6994103a00 2024-03-18T01:34:38.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.888+0000 7f698aa00640 1 -- 172.21.15.7:0/3780936202 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984039040 con 0x7f6994103a00 2024-03-18T01:34:38.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.888+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69941a1bb0 con 0x7f6994103a00 2024-03-18T01:34:38.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.888+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69941a2020 con 0x7f6994103a00 2024-03-18T01:34:38.892 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 podman[68138]: 2024-03-18 01:34:38.564216528 +0000 UTC m=+0.384751241 container start b94be326eadc665ac4f60567c3632b9779ca1bb1f1ac5252e7f4510682ce532d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:34:38.892 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-mds[68157]: -- 172.21.15.203:0/2690032581 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556f4bb3a40 con 0x5556f4b87000 2024-03-18T01:34:38.892 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i[68153]: starting mds.i at 2024-03-18T01:34:38.892 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 bash[68138]: b94be326eadc665ac4f60567c3632b9779ca1bb1f1ac5252e7f4510682ce532d 2024-03-18T01:34:38.892 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:34:38 smithi203 systemd[1]: Started Ceph mds.i for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:38.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.890+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f699410adf0 con 0x7f6994103a00 2024-03-18T01:34:38.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.891+0000 7f698aa00640 1 -- 172.21.15.7:0/3780936202 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f6984044020 con 0x7f6994103a00 2024-03-18T01:34:38.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.892+0000 7f698aa00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69780a2950 0x7f69780a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.893+0000 7f698aa00640 1 -- 172.21.15.7:0/3780936202 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f69840efe50 con 0x7f6994103a00 2024-03-18T01:34:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.893+0000 7f6991400640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69780a2950 0x7f69780a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:38.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.893+0000 7f6991400640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69780a2950 0x7f69780a4e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f697c009cd0 tx=0x7f697c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:38.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:38.896+0000 7f698aa00640 1 -- 172.21.15.7:0/3780936202 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f698404b020 con 0x7f6994103a00 2024-03-18T01:34:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:38 smithi007 ceph-mon[31996]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:38 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:39.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.145+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69940623e0 con 0x7f6994103a00 2024-03-18T01:34:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.146+0000 7f698aa00640 1 -- 172.21.15.7:0/3780936202 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7992 (secure 0 0 0) 0x7f69840bd2f0 con 0x7f6994103a00 2024-03-18T01:34:39.147 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:39.147 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T01:34:39.147 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 10 2024-03-18T01:34:39.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69780a2950 msgr2=0x7f69780a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69780a2950 0x7f69780a4e10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f697c009cd0 tx=0x7f697c005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 msgr2=0x7f6994199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:39.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 0x7f6994199d10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6984009780 tx=0x7f6984032430 comp rx=0 tx=0).stop 2024-03-18T01:34:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 shutdown_connections 2024-03-18T01:34:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69941055b0 0x7f69941a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69780a2950 0x7f69780a4e10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6994104bf0 0x7f699419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 --2- 172.21.15.7:0/3780936202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6994103a00 0x7f6994199d10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 >> 172.21.15.7:0/3780936202 conn(0x7f69940fef90 msgr2=0x7f6994107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 shutdown_connections 2024-03-18T01:34:39.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:39.149+0000 7f6998c00640 1 -- 172.21.15.7:0/3780936202 wait complete. 2024-03-18T01:34:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:38 smithi046 ceph-mon[36108]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:38 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-mon[36345]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:39.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:39.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:39.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:38 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:40.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:39 smithi007 ceph-mon[31996]: Deploying daemon mds.g on smithi007 2024-03-18T01:34:40.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:39 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:boot 2024-03-18T01:34:40.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:39 smithi007 ceph-mon[31996]: fsmap 9 up:standby 2024-03-18T01:34:40.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:39 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:34:40.046 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:39 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3780936202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:40.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:39 smithi046 ceph-mon[36108]: Deploying daemon mds.g on smithi007 2024-03-18T01:34:40.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:39 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:boot 2024-03-18T01:34:40.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:39 smithi046 ceph-mon[36108]: fsmap 9 up:standby 2024-03-18T01:34:40.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:39 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:34:40.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:39 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3780936202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:40.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:39 smithi203 ceph-mon[36345]: Deploying daemon mds.g on smithi007 2024-03-18T01:34:40.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:39 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:boot 2024-03-18T01:34:40.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:39 smithi203 ceph-mon[36345]: fsmap 9 up:standby 2024-03-18T01:34:40.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:39 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:34:40.293 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:39 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3780936202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:40.977 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:40 smithi203 ceph-mon[36345]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:41.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:40 smithi007 ceph-mon[31996]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:41.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:41.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:40 smithi046 ceph-mon[36108]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:41.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 -- 172.21.15.7:0/269019142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02f8075820 msgr2=0x7f02f8073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:41.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 --2- 172.21.15.7:0/269019142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02f8075820 0x7f02f8073c70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009930 tx=0x7f02ec02bc20 comp rx=0 tx=0).stop 2024-03-18T01:34:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 -- 172.21.15.7:0/269019142 shutdown_connections 2024-03-18T01:34:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 --2- 172.21.15.7:0/269019142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02f810ddb0 0x7f02f81101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 --2- 172.21.15.7:0/269019142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02f8074240 0x7f02f80746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 --2- 172.21.15.7:0/269019142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02f8075820 0x7f02f8073c70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 -- 172.21.15.7:0/269019142 >> 172.21.15.7:0/269019142 conn(0x7f02f80fcd50 msgr2=0x7f02f80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 -- 172.21.15.7:0/269019142 shutdown_connections 2024-03-18T01:34:41.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.407+0000 7f02ff600640 1 -- 172.21.15.7:0/269019142 wait complete. 2024-03-18T01:34:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.408+0000 7f02ff600640 1 Processor -- start 2024-03-18T01:34:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.408+0000 7f02ff600640 1 -- start start 2024-03-18T01:34:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02ff600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02f8074240 0x7f02f819e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:41.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02ff600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02f8075820 0x7f02f819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:41.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02ff600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02f810ddb0 0x7f02f81a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:41.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02ff600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8113a50 con 0x7f02f8075820 2024-03-18T01:34:41.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02ff600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81138d0 con 0x7f02f8074240 2024-03-18T01:34:41.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02ff600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8113bd0 con 0x7f02f810ddb0 2024-03-18T01:34:41.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02f8074240 0x7f02f819e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:41.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02fcc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02f8074240 0x7f02f819e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55660/0 (socket says 172.21.15.7:55660) 2024-03-18T01:34:41.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02fcc00640 1 -- 172.21.15.7:0/61842361 learned_addr learned my addr 172.21.15.7:0/61842361 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:41.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02f7e00640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02f8075820 0x7f02f819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.409+0000 7f02fd600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02f810ddb0 0x7f02f81a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.410+0000 7f02fd600640 1 -- 172.21.15.7:0/61842361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02f8074240 msgr2=0x7f02f819e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.410+0000 7f02fd600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02f8074240 0x7f02f819e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.410+0000 7f02fd600640 1 -- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02f8075820 msgr2=0x7f02f819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.410+0000 7f02fd600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02f8075820 0x7f02f819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.410+0000 7f02fd600640 1 -- 172.21.15.7:0/61842361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f81a5d20 con 0x7f02f810ddb0 2024-03-18T01:34:41.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.410+0000 7f02fd600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02f810ddb0 0x7f02f81a56b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009fd0 tx=0x7f02f000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.411+0000 7f02f5600640 1 -- 172.21.15.7:0/61842361 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f000ce20 con 0x7f02f810ddb0 2024-03-18T01:34:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.411+0000 7f02f5600640 1 -- 172.21.15.7:0/61842361 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02f0005020 con 0x7f02f810ddb0 2024-03-18T01:34:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.411+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f81a5fb0 con 0x7f02f810ddb0 2024-03-18T01:34:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.411+0000 7f02f5600640 1 -- 172.21.15.7:0/61842361 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f0010720 con 0x7f02f810ddb0 2024-03-18T01:34:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.411+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f81a6520 con 0x7f02f810ddb0 2024-03-18T01:34:41.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.412+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f8103f80 con 0x7f02f810ddb0 2024-03-18T01:34:41.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.416+0000 7f02f5600640 1 -- 172.21.15.7:0/61842361 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f02f0005540 con 0x7f02f810ddb0 2024-03-18T01:34:41.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.419+0000 7f02f5600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f02e00a2900 0x7f02e00a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:41.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.419+0000 7f02f5600640 1 -- 172.21.15.7:0/61842361 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f02f00ceec0 con 0x7f02f810ddb0 2024-03-18T01:34:41.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.420+0000 7f02fcc00640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f02e00a2900 0x7f02e00a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:41.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.420+0000 7f02f5600640 1 -- 172.21.15.7:0/61842361 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f02f00cf2e0 con 0x7f02f810ddb0 2024-03-18T01:34:41.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.421+0000 7f02fcc00640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f02e00a2900 0x7f02e00a4dc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0096c0 tx=0x7f02ec02c600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:41.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.680+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02f802d1b0 con 0x7f02f810ddb0 2024-03-18T01:34:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.681+0000 7f02f5600640 1 -- 172.21.15.7:0/61842361 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7992 (secure 0 0 0) 0x7f02f009c3f0 con 0x7f02f810ddb0 2024-03-18T01:34:41.683 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:41.683 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T01:34:41.683 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 10 2024-03-18T01:34:41.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f02e00a2900 msgr2=0x7f02e00a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f02e00a2900 0x7f02e00a4dc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02ec0096c0 tx=0x7f02ec02c600 comp rx=0 tx=0).stop 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02f810ddb0 msgr2=0x7f02f81a56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02f810ddb0 0x7f02f81a56b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009fd0 tx=0x7f02f000ef10 comp rx=0 tx=0).stop 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 shutdown_connections 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02f810ddb0 0x7f02f81a56b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f02e00a2900 0x7f02e00a4dc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f02f8075820 0x7f02f819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 --2- 172.21.15.7:0/61842361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02f8074240 0x7f02f819e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.685+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 >> 172.21.15.7:0/61842361 conn(0x7f02f80fcd50 msgr2=0x7f02f8101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.686+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 shutdown_connections 2024-03-18T01:34:41.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:41.686+0000 7f02ff600640 1 -- 172.21.15.7:0/61842361 wait complete. 2024-03-18T01:34:42.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:41 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/61842361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:42.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:41 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/61842361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:42.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:41 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/61842361' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:42.802 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:34:42 smithi007 podman[73468]: 2024-03-18 01:34:42.57429586 +0000 UTC m=+0.166571513 container create 62fd0322f37c82259e6953f61ad164b0aa4e3013d8086d2d9d1ce4cc98aabc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T01:34:43.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:42 smithi046 ceph-mon[36108]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:43.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:42 smithi007 ceph-mon[31996]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:43.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:34:42 smithi007 podman[73468]: 2024-03-18 01:34:42.89433174 +0000 UTC m=+0.486607373 container init 62fd0322f37c82259e6953f61ad164b0aa4e3013d8086d2d9d1ce4cc98aabc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:34:43.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:34:42 smithi007 podman[73468]: 2024-03-18 01:34:42.90416902 +0000 UTC m=+0.496444642 container start 62fd0322f37c82259e6953f61ad164b0aa4e3013d8086d2d9d1ce4cc98aabc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T01:34:43.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:34:42 smithi007 ceph-mds[73494]: -- 172.21.15.7:0/627631229 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5610368c64e0 con 0x56103689d000 2024-03-18T01:34:43.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:34:42 smithi007 ceph-mds[73494]: -- 172.21.15.7:0/627631229 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610368c9a40 con 0x56103689d000 2024-03-18T01:34:43.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:34:42 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g[73485]: starting mds.g at 2024-03-18T01:34:43.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:34:43 smithi007 bash[73468]: 62fd0322f37c82259e6953f61ad164b0aa4e3013d8086d2d9d1ce4cc98aabc89 2024-03-18T01:34:43.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:34:43 smithi007 systemd[1]: Started Ceph mds.g for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:43.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:42 smithi203 ceph-mon[36345]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:43.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 -- 172.21.15.7:0/3375827984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77900761a0 msgr2=0x7f77900765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 --2- 172.21.15.7:0/3375827984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77900761a0 0x7f77900765c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7780009860 tx=0x7f778002bb70 comp rx=0 tx=0).stop 2024-03-18T01:34:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 -- 172.21.15.7:0/3375827984 shutdown_connections 2024-03-18T01:34:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 --2- 172.21.15.7:0/3375827984 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7790077d80 0x7f7790078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 --2- 172.21.15.7:0/3375827984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77900773e0 0x7f7790077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 --2- 172.21.15.7:0/3375827984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77900761a0 0x7f77900765c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 -- 172.21.15.7:0/3375827984 >> 172.21.15.7:0/3375827984 conn(0x7f77900fef90 msgr2=0x7f77901013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:43.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 -- 172.21.15.7:0/3375827984 shutdown_connections 2024-03-18T01:34:43.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.916+0000 7f7796400640 1 -- 172.21.15.7:0/3375827984 wait complete. 2024-03-18T01:34:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.917+0000 7f7796400640 1 Processor -- start 2024-03-18T01:34:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.917+0000 7f7796400640 1 -- start start 2024-03-18T01:34:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.917+0000 7f7796400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77900761a0 0x7f779019e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.917+0000 7f7796400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77900773e0 0x7f779019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.917+0000 7f7796400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7790077d80 0x7f77901a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.917+0000 7f7796400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7790113a80 con 0x7f77900761a0 2024-03-18T01:34:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.917+0000 7f7796400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7790113900 con 0x7f77900773e0 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.917+0000 7f7796400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7790113c00 con 0x7f7790077d80 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77900761a0 0x7f779019e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77900761a0 0x7f779019e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56258/0 (socket says 172.21.15.7:56258) 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778f400640 1 -- 172.21.15.7:0/2654487841 learned_addr learned my addr 172.21.15.7:0/2654487841 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f7787e00640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77900773e0 0x7f779019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778fe00640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7790077d80 0x7f77901a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778f400640 1 -- 172.21.15.7:0/2654487841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7790077d80 msgr2=0x7f77901a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778f400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7790077d80 0x7f77901a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778f400640 1 -- 172.21.15.7:0/2654487841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77900773e0 msgr2=0x7f779019e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778f400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77900773e0 0x7f779019e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.918+0000 7f778f400640 1 -- 172.21.15.7:0/2654487841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77901a5cd0 con 0x7f77900761a0 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.919+0000 7f778f400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77900761a0 0x7f779019e0b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f77800349e0 tx=0x7f7780032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.919+0000 7f778cc00640 1 -- 172.21.15.7:0/2654487841 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780041030 con 0x7f77900761a0 2024-03-18T01:34:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.919+0000 7f778cc00640 1 -- 172.21.15.7:0/2654487841 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f778002c510 con 0x7f77900761a0 2024-03-18T01:34:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.919+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77901a5f60 con 0x7f77900761a0 2024-03-18T01:34:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.919+0000 7f778cc00640 1 -- 172.21.15.7:0/2654487841 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780039040 con 0x7f77900761a0 2024-03-18T01:34:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.919+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77901a6360 con 0x7f77900761a0 2024-03-18T01:34:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.920+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7790110370 con 0x7f77900761a0 2024-03-18T01:34:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.923+0000 7f778cc00640 1 -- 172.21.15.7:0/2654487841 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f7780044070 con 0x7f77900761a0 2024-03-18T01:34:43.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.925+0000 7f778cc00640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77740a2800 0x7f77740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:43.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.925+0000 7f778cc00640 1 -- 172.21.15.7:0/2654487841 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f77800efb90 con 0x7f77900761a0 2024-03-18T01:34:43.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.925+0000 7f7787e00640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77740a2800 0x7f77740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:43.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.926+0000 7f7787e00640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77740a2800 0x7f77740a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f779019f5c0 tx=0x7f7778005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:43.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:43.929+0000 7f778cc00640 1 -- 172.21.15.7:0/2654487841 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f77800bd030 con 0x7f77900761a0 2024-03-18T01:34:44.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.212+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f779002d1b0 con 0x7f77900761a0 2024-03-18T01:34:44.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.214+0000 7f778cc00640 1 -- 172.21.15.7:0/2654487841 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8815 (secure 0 0 0) 0x7f77800bc930 con 0x7f77900761a0 2024-03-18T01:34:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:44.215 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-03-18T01:34:44.215 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 11 2024-03-18T01:34:44.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77740a2800 msgr2=0x7f77740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:44.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77740a2800 0x7f77740a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f779019f5c0 tx=0x7f7778005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:44.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77900761a0 msgr2=0x7f779019e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:44.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77900761a0 0x7f779019e0b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f77800349e0 tx=0x7f7780032450 comp rx=0 tx=0).stop 2024-03-18T01:34:44.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 shutdown_connections 2024-03-18T01:34:44.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7790077d80 0x7f77901a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:44.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77900773e0 0x7f779019e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77740a2800 0x7f77740a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 --2- 172.21.15.7:0/2654487841 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77900761a0 0x7f779019e0b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 >> 172.21.15.7:0/2654487841 conn(0x7f77900fef90 msgr2=0x7f7790100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 shutdown_connections 2024-03-18T01:34:44.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:44.217+0000 7f7796400640 1 -- 172.21.15.7:0/2654487841 wait complete. 2024-03-18T01:34:44.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:44.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:44.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:44.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:44.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: Deploying daemon mds.a on smithi007 2024-03-18T01:34:44.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:boot 2024-03-18T01:34:44.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: fsmap 10 up:standby 2024-03-18T01:34:44.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: Deploying daemon mds.a on smithi007 2024-03-18T01:34:44.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:boot 2024-03-18T01:34:44.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: fsmap 10 up:standby 2024-03-18T01:34:44.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:34:44.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: Deploying daemon mds.a on smithi007 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:boot 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: fsmap 10 up:standby 2024-03-18T01:34:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:34:45.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:45 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2654487841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:45.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:45 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2654487841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:45 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2654487841' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:46.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:46.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:46 smithi007 ceph-mon[31996]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:46.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.464+0000 7f3accc00640 1 -- 172.21.15.7:0/2418395516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac8102320 msgr2=0x7f3ac8104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:46.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.464+0000 7f3accc00640 1 --2- 172.21.15.7:0/2418395516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac8102320 0x7f3ac8104710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8009900 tx=0x7f3ab802bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:46.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.464+0000 7f3accc00640 1 -- 172.21.15.7:0/2418395516 shutdown_connections 2024-03-18T01:34:46.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.464+0000 7f3accc00640 1 --2- 172.21.15.7:0/2418395516 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac8107580 0x7f3ac8109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.464+0000 7f3accc00640 1 --2- 172.21.15.7:0/2418395516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac8104c50 0x7f3ac8107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.464+0000 7f3accc00640 1 --2- 172.21.15.7:0/2418395516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac8102320 0x7f3ac8104710 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.464+0000 7f3accc00640 1 -- 172.21.15.7:0/2418395516 >> 172.21.15.7:0/2418395516 conn(0x7f3ac80fbc70 msgr2=0x7f3ac80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:46.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.465+0000 7f3accc00640 1 -- 172.21.15.7:0/2418395516 shutdown_connections 2024-03-18T01:34:46.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.465+0000 7f3accc00640 1 -- 172.21.15.7:0/2418395516 wait complete. 2024-03-18T01:34:46.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3accc00640 1 Processor -- start 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3accc00640 1 -- start start 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3accc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac8102320 0x7f3ac8101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3accc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac8104c50 0x7f3ac81002d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3accc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac8107580 0x7f3ac8100810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3accc00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac810d4c0 con 0x7f3ac8102320 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3accc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac810d340 con 0x7f3ac8104c50 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3accc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac810d640 con 0x7f3ac8107580 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.466+0000 7f3ac5e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac8102320 0x7f3ac8101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac8102320 0x7f3ac8101c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39472/0 (socket says 172.21.15.7:39472) 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5e00640 1 -- 172.21.15.7:0/3855565662 learned_addr learned my addr 172.21.15.7:0/3855565662 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5400640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac8104c50 0x7f3ac81002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:46.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac6800640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac8107580 0x7f3ac8100810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:46.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5400640 1 -- 172.21.15.7:0/3855565662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac8107580 msgr2=0x7f3ac8100810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:46.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5400640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac8107580 0x7f3ac8100810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5400640 1 -- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac8102320 msgr2=0x7f3ac8101c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5400640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac8102320 0x7f3ac8101c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5400640 1 -- 172.21.15.7:0/3855565662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac81010a0 con 0x7f3ac8104c50 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5e00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac8102320 0x7f3ac8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.467+0000 7f3ac5400640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac8104c50 0x7f3ac81002d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400e9c0 tx=0x7f3ab400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.468+0000 7f3abea00640 1 -- 172.21.15.7:0/3855565662 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab400cd80 con 0x7f3ac8104c50 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.468+0000 7f3abea00640 1 -- 172.21.15.7:0/3855565662 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab40035f0 con 0x7f3ac8104c50 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.468+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac81a1bc0 con 0x7f3ac8104c50 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.468+0000 7f3abea00640 1 -- 172.21.15.7:0/3855565662 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4010690 con 0x7f3ac8104c50 2024-03-18T01:34:46.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.468+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac81a2110 con 0x7f3ac8104c50 2024-03-18T01:34:46.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.469+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac80fd760 con 0x7f3ac8104c50 2024-03-18T01:34:46.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.471+0000 7f3abea00640 1 -- 172.21.15.7:0/3855565662 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f3ab40037a0 con 0x7f3ac8104c50 2024-03-18T01:34:46.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.473+0000 7f3abea00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3aac0a2800 0x7f3aac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:46.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.473+0000 7f3abea00640 1 -- 172.21.15.7:0/3855565662 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3ab40ce8c0 con 0x7f3ac8104c50 2024-03-18T01:34:46.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.473+0000 7f3ac5e00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3aac0a2800 0x7f3aac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:46.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.474+0000 7f3ac5e00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3aac0a2800 0x7f3aac0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3ab802c8e0 tx=0x7f3ab802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:46.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.477+0000 7f3abea00640 1 -- 172.21.15.7:0/3855565662 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ab409bd60 con 0x7f3ac8104c50 2024-03-18T01:34:46.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:46 smithi046 ceph-mon[36108]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:46.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:46 smithi203 ceph-mon[36345]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:46.712 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:46 smithi007 systemd[1]: Starting Ceph mds.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:34:46.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.737+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ac80623e0 con 0x7f3ac8104c50 2024-03-18T01:34:46.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.739+0000 7f3abea00640 1 -- 172.21.15.7:0/3855565662 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8815 (secure 0 0 0) 0x7f3ab409b660 con 0x7f3ac8104c50 2024-03-18T01:34:46.740 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:46.740 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-03-18T01:34:46.740 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 11 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3aac0a2800 msgr2=0x7f3aac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3aac0a2800 0x7f3aac0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3ab802c8e0 tx=0x7f3ab802df90 comp rx=0 tx=0).stop 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac8104c50 msgr2=0x7f3ac81002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac8104c50 0x7f3ac81002d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400e9c0 tx=0x7f3ab400ee80 comp rx=0 tx=0).stop 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 shutdown_connections 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ac8107580 0x7f3ac8100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ac8104c50 0x7f3ac81002d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3aac0a2800 0x7f3aac0a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 --2- 172.21.15.7:0/3855565662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ac8102320 0x7f3ac8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 >> 172.21.15.7:0/3855565662 conn(0x7f3ac80fbc70 msgr2=0x7f3ac8102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 shutdown_connections 2024-03-18T01:34:46.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:46.741+0000 7f3accc00640 1 -- 172.21.15.7:0/3855565662 wait complete. 2024-03-18T01:34:47.052 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:46 smithi007 podman[73858]: 2024-03-18 01:34:46.617915956 +0000 UTC m=+0.036061845 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:34:47.052 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:46 smithi007 podman[73858]: 2024-03-18 01:34:46.745644173 +0000 UTC m=+0.163790067 container create a8aabe0db2a02e37f24642a9de6bfcec43b490ae48d3c943346bffcb0402d9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T01:34:47.522 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:47 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3855565662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:47.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:47 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3855565662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:47.552 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:47 smithi007 podman[73858]: 2024-03-18 01:34:47.124782266 +0000 UTC m=+0.542928198 container init a8aabe0db2a02e37f24642a9de6bfcec43b490ae48d3c943346bffcb0402d9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T01:34:47.553 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:47 smithi007 podman[73858]: 2024-03-18 01:34:47.135673151 +0000 UTC m=+0.553819049 container start a8aabe0db2a02e37f24642a9de6bfcec43b490ae48d3c943346bffcb0402d9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True) 2024-03-18T01:34:47.553 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:47 smithi007 ceph-mds[73879]: -- 172.21.15.7:0/876282002 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55940dca9a40 con 0x55940dc7c800 2024-03-18T01:34:47.553 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:47 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a[73874]: starting mds.a at 2024-03-18T01:34:47.553 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:47 smithi007 bash[73858]: a8aabe0db2a02e37f24642a9de6bfcec43b490ae48d3c943346bffcb0402d9ef 2024-03-18T01:34:47.553 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:34:47 smithi007 systemd[1]: Started Ceph mds.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:47.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:47 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3855565662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:48.764 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:48.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:48.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: Deploying daemon mds.b on smithi046 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6840/1978448378,v1:172.21.15.7:6841/1978448378] up:boot 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: fsmap 11 up:standby 2024-03-18T01:34:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T01:34:48.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: Deploying daemon mds.b on smithi046 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6840/1978448378,v1:172.21.15.7:6841/1978448378] up:boot 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: fsmap 11 up:standby 2024-03-18T01:34:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: Deploying daemon mds.b on smithi046 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6840/1978448378,v1:172.21.15.7:6841/1978448378] up:boot 2024-03-18T01:34:48.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: fsmap 11 up:standby 2024-03-18T01:34:48.895 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T01:34:48.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 -- 172.21.15.7:0/317726180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8308104a00 msgr2=0x7f8308104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 --2- 172.21.15.7:0/317726180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8308104a00 0x7f8308104e80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009900 tx=0x7f82f402bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 -- 172.21.15.7:0/317726180 shutdown_connections 2024-03-18T01:34:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 --2- 172.21.15.7:0/317726180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83081053c0 0x7f830810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 --2- 172.21.15.7:0/317726180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8308104a00 0x7f8308104e80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 --2- 172.21.15.7:0/317726180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8308103810 0x7f8308103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:48.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 -- 172.21.15.7:0/317726180 >> 172.21.15.7:0/317726180 conn(0x7f83080fed80 msgr2=0x7f83081011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:48.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 -- 172.21.15.7:0/317726180 shutdown_connections 2024-03-18T01:34:48.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.997+0000 7f830ec00640 1 -- 172.21.15.7:0/317726180 wait complete. 2024-03-18T01:34:49.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.998+0000 7f830ec00640 1 Processor -- start 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.998+0000 7f830ec00640 1 -- start start 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ec00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8308103810 0x7f8308199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ec00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8308104a00 0x7f8308199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ec00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83081053c0 0x7f83081a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ec00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f830810f4e0 con 0x7f8308104a00 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ec00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f830810f360 con 0x7f8308103810 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ec00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f830810f660 con 0x7f83081053c0 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ce00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8308104a00 0x7f8308199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:49.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ce00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8308104a00 0x7f8308199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39514/0 (socket says 172.21.15.7:39514) 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830ce00640 1 -- 172.21.15.7:0/3835036299 learned_addr learned my addr 172.21.15.7:0/3835036299 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830e200640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83081053c0 0x7f83081a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:48.999+0000 7f830d800640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8308103810 0x7f8308199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830d800640 1 -- 172.21.15.7:0/3835036299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83081053c0 msgr2=0x7f83081a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830d800640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83081053c0 0x7f83081a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830d800640 1 -- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8308104a00 msgr2=0x7f8308199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830d800640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8308104a00 0x7f8308199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830d800640 1 -- 172.21.15.7:0/3835036299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83081a1630 con 0x7f8308103810 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830ce00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8308104a00 0x7f8308199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830e200640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83081053c0 0x7f83081a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:49.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830d800640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8308103810 0x7f8308199a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00e9c0 tx=0x7f82fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:49.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f8306000640 1 -- 172.21.15.7:0/3835036299 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00ccc0 con 0x7f8308103810 2024-03-18T01:34:49.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f8306000640 1 -- 172.21.15.7:0/3835036299 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82fc0035f0 con 0x7f8308103810 2024-03-18T01:34:49.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83081a1920 con 0x7f8308103810 2024-03-18T01:34:49.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.000+0000 7f8306000640 1 -- 172.21.15.7:0/3835036299 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00ce60 con 0x7f8308103810 2024-03-18T01:34:49.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.001+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83081a1df0 con 0x7f8308103810 2024-03-18T01:34:49.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.002+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f830810aad0 con 0x7f8308103810 2024-03-18T01:34:49.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.004+0000 7f8306000640 1 -- 172.21.15.7:0/3835036299 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f82fc0037a0 con 0x7f8308103810 2024-03-18T01:34:49.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.005+0000 7f8306000640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82f00a28a0 0x7f82f00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:49.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.005+0000 7f8306000640 1 -- 172.21.15.7:0/3835036299 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f82fc0ceb70 con 0x7f8308103810 2024-03-18T01:34:49.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.005+0000 7f830ce00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82f00a28a0 0x7f82f00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:49.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.006+0000 7f830ce00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82f00a28a0 0x7f82f00a4d60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f82f402c8e0 tx=0x7f82f402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:49.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.009+0000 7f8306000640 1 -- 172.21.15.7:0/3835036299 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f82fc09c010 con 0x7f8308103810 2024-03-18T01:34:49.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.254+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f830802d1b0 con 0x7f8308103810 2024-03-18T01:34:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.255+0000 7f8306000640 1 -- 172.21.15.7:0/3835036299 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9638 (secure 0 0 0) 0x7f82fc09b910 con 0x7f8308103810 2024-03-18T01:34:49.255 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:49.255 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-03-18T01:34:49.255 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 12 2024-03-18T01:34:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82f00a28a0 msgr2=0x7f82f00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82f00a28a0 0x7f82f00a4d60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f82f402c8e0 tx=0x7f82f402df90 comp rx=0 tx=0).stop 2024-03-18T01:34:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8308103810 msgr2=0x7f8308199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8308103810 0x7f8308199a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00e9c0 tx=0x7f82fc00ee80 comp rx=0 tx=0).stop 2024-03-18T01:34:49.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 shutdown_connections 2024-03-18T01:34:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83081053c0 0x7f83081a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82f00a28a0 0x7f82f00a4d60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8308104a00 0x7f8308199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 --2- 172.21.15.7:0/3835036299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8308103810 0x7f8308199a10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 >> 172.21.15.7:0/3835036299 conn(0x7f83080fed80 msgr2=0x7f8308107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 shutdown_connections 2024-03-18T01:34:49.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:49.258+0000 7f830ec00640 1 -- 172.21.15.7:0/3835036299 wait complete. 2024-03-18T01:34:49.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:49 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3835036299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:49.738 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:49 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3835036299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:49.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:49 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3835036299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:50.019 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:34:49 smithi046 podman[66748]: 2024-03-18 01:34:49.961745231 +0000 UTC m=+0.122056653 container create efeff0e0d0eb1cc1bc175bc69973aabab95d78394acf7f4a634783ae4fd47911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T01:34:50.348 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:34:50 smithi046 podman[66748]: 2024-03-18 01:34:50.205893487 +0000 UTC m=+0.366204884 container init efeff0e0d0eb1cc1bc175bc69973aabab95d78394acf7f4a634783ae4fd47911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T01:34:50.348 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:34:50 smithi046 podman[66748]: 2024-03-18 01:34:50.209846886 +0000 UTC m=+0.370158283 container start efeff0e0d0eb1cc1bc175bc69973aabab95d78394acf7f4a634783ae4fd47911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-18T01:34:50.348 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:34:50 smithi046 ceph-mds[66768]: -- 172.21.15.46:0/1619800686 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e028211a40 con 0x55e0281e4c00 2024-03-18T01:34:50.348 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:34:50 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: starting mds.b at 2024-03-18T01:34:50.348 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:34:50 smithi046 bash[66748]: efeff0e0d0eb1cc1bc175bc69973aabab95d78394acf7f4a634783ae4fd47911 2024-03-18T01:34:50.348 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:34:50 smithi046 systemd[1]: Started Ceph mds.b for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:34:50.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:50 smithi046 ceph-mon[36108]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:50 smithi007 ceph-mon[31996]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:50 smithi203 ceph-mon[36345]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:51.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:34:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 -- 172.21.15.7:0/1233876237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc101800 msgr2=0x7f9cdc101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 --2- 172.21.15.7:0/1233876237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc101800 0x7f9cdc101c20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc009900 tx=0x7f9ccc02bc40 comp rx=0 tx=0).stop 2024-03-18T01:34:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 -- 172.21.15.7:0/1233876237 shutdown_connections 2024-03-18T01:34:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 --2- 172.21.15.7:0/1233876237 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc1033b0 0x7f9cdc109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 --2- 172.21.15.7:0/1233876237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc1029f0 0x7f9cdc102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 --2- 172.21.15.7:0/1233876237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc101800 0x7f9cdc101c20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 -- 172.21.15.7:0/1233876237 >> 172.21.15.7:0/1233876237 conn(0x7f9cdc0fcd70 msgr2=0x7f9cdc0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:51.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 -- 172.21.15.7:0/1233876237 shutdown_connections 2024-03-18T01:34:51.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.500+0000 7f9ce1600640 1 -- 172.21.15.7:0/1233876237 wait complete. 2024-03-18T01:34:51.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9ce1600640 1 Processor -- start 2024-03-18T01:34:51.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9ce1600640 1 -- start start 2024-03-18T01:34:51.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9ce1600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc101800 0x7f9cdc107580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:51.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9ce1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc1029f0 0x7f9cdc107ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:51.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9ce1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc1033b0 0x7f9cdc1040f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:51.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9ce1600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc10d4c0 con 0x7f9cdc101800 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9ce1600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc10d340 con 0x7f9cdc1029f0 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9ce1600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9cdc10d640 con 0x7f9cdc1033b0 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.501+0000 7f9cdb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc1033b0 0x7f9cdc1040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdaa00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc101800 0x7f9cdc107580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc1033b0 0x7f9cdc1040f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:47426/0 (socket says 172.21.15.7:47426) 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdb400640 1 -- 172.21.15.7:0/3928077385 learned_addr learned my addr 172.21.15.7:0/3928077385 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cda000640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc1029f0 0x7f9cdc107ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdb400640 1 -- 172.21.15.7:0/3928077385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc1029f0 msgr2=0x7f9cdc107ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdb400640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc1029f0 0x7f9cdc107ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdb400640 1 -- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc101800 msgr2=0x7f9cdc107580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdb400640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc101800 0x7f9cdc107580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdb400640 1 -- 172.21.15.7:0/3928077385 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cdc1049b0 con 0x7f9cdc1033b0 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cda000640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc1029f0 0x7f9cdc107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:34:51.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cdb400640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc1033b0 0x7f9cdc1040f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400eaa0 tx=0x7f9cd400ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cd3400640 1 -- 172.21.15.7:0/3928077385 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400cdb0 con 0x7f9cdc1033b0 2024-03-18T01:34:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cd3400640 1 -- 172.21.15.7:0/3928077385 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cd4005020 con 0x7f9cdc1033b0 2024-03-18T01:34:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.502+0000 7f9cd3400640 1 -- 172.21.15.7:0/3928077385 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd40116b0 con 0x7f9cdc1033b0 2024-03-18T01:34:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.503+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cdc1a6180 con 0x7f9cdc1033b0 2024-03-18T01:34:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.503+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cdc1a6590 con 0x7f9cdc1033b0 2024-03-18T01:34:51.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.504+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cdc1a6820 con 0x7f9cdc1033b0 2024-03-18T01:34:51.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.506+0000 7f9cd3400640 1 -- 172.21.15.7:0/3928077385 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f9cd4005560 con 0x7f9cdc1033b0 2024-03-18T01:34:51.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.507+0000 7f9cd3400640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc00a2800 0x7f9cc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:51.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.507+0000 7f9cd3400640 1 -- 172.21.15.7:0/3928077385 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9cd40ce8c0 con 0x7f9cdc1033b0 2024-03-18T01:34:51.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.508+0000 7f9cdaa00640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc00a2800 0x7f9cc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:51.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.508+0000 7f9cdaa00640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc00a2800 0x7f9cc00a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc02c180 tx=0x7f9ccc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:51.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.511+0000 7f9cd3400640 1 -- 172.21.15.7:0/3928077385 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9cd4016030 con 0x7f9cdc1033b0 2024-03-18T01:34:51.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:51 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:boot 2024-03-18T01:34:51.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:51 smithi046 ceph-mon[36108]: fsmap 12 up:standby 2024-03-18T01:34:51.671 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:51 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:34:51.671 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:51 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:51.709 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:51 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:boot 2024-03-18T01:34:51.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:51 smithi203 ceph-mon[36345]: fsmap 12 up:standby 2024-03-18T01:34:51.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:51 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:34:51.710 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:51 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:51.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.786+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cdc1a69f0 con 0x7f9cdc1033b0 2024-03-18T01:34:51.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.787+0000 7f9cd3400640 1 -- 172.21.15.7:0/3928077385 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10464 (secure 0 0 0) 0x7f9cd409bd60 con 0x7f9cdc1033b0 2024-03-18T01:34:51.787 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:51.787 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24596,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-18T01:34:51.788 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 13 2024-03-18T01:34:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc00a2800 msgr2=0x7f9cc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc00a2800 0x7f9cc00a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc02c180 tx=0x7f9ccc02cf90 comp rx=0 tx=0).stop 2024-03-18T01:34:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc1033b0 msgr2=0x7f9cdc1040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc1033b0 0x7f9cdc1040f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400eaa0 tx=0x7f9cd400ef60 comp rx=0 tx=0).stop 2024-03-18T01:34:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 shutdown_connections 2024-03-18T01:34:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc1033b0 0x7f9cdc1040f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cdc1029f0 0x7f9cdc107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc00a2800 0x7f9cc00a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 --2- 172.21.15.7:0/3928077385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9cdc101800 0x7f9cdc107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 >> 172.21.15.7:0/3928077385 conn(0x7f9cdc0fcd70 msgr2=0x7f9cdc073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 shutdown_connections 2024-03-18T01:34:51.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:51.790+0000 7f9ce1600640 1 -- 172.21.15.7:0/3928077385 wait complete. 2024-03-18T01:34:51.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:51 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:boot 2024-03-18T01:34:51.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:51 smithi007 ceph-mon[31996]: fsmap 12 up:standby 2024-03-18T01:34:51.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:51 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:34:51.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:51 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:51.809 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:34:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 -- 172.21.15.7:0/3541945822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048109950 msgr2=0x7f6048110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 --2- 172.21.15.7:0/3541945822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048109950 0x7f6048110270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6040009ca0 tx=0x7f604002c100 comp rx=0 tx=0).stop 2024-03-18T01:34:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 -- 172.21.15.7:0/3541945822 shutdown_connections 2024-03-18T01:34:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 --2- 172.21.15.7:0/3541945822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048109950 0x7f6048110270 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 --2- 172.21.15.7:0/3541945822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6048108f90 0x7f6048109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 --2- 172.21.15.7:0/3541945822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6048107e40 0x7f6048108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 -- 172.21.15.7:0/3541945822 >> 172.21.15.7:0/3541945822 conn(0x7f6048076dd0 msgr2=0x7f6048079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 -- 172.21.15.7:0/3541945822 shutdown_connections 2024-03-18T01:34:52.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.011+0000 7f604d000640 1 -- 172.21.15.7:0/3541945822 wait complete. 2024-03-18T01:34:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.012+0000 7f604d000640 1 Processor -- start 2024-03-18T01:34:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.012+0000 7f604d000640 1 -- start start 2024-03-18T01:34:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.012+0000 7f604d000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6048107e40 0x7f604819e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.012+0000 7f604d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048108f90 0x7f604819e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.012+0000 7f604d000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6048109950 0x7f60481a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.012+0000 7f604d000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6048113ac0 con 0x7f6048107e40 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.012+0000 7f604d000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6048113940 con 0x7f6048109950 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.012+0000 7f604d000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6048113c40 con 0x7f6048108f90 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6048107e40 0x7f604819e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6048107e40 0x7f604819e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39554/0 (socket says 172.21.15.7:39554) 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 -- 172.21.15.7:0/3641234579 learned_addr learned my addr 172.21.15.7:0/3641234579 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045400640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048108f90 0x7f604819e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6046800640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6048109950 0x7f60481a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 -- 172.21.15.7:0/3641234579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048108f90 msgr2=0x7f604819e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048108f90 0x7f604819e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 -- 172.21.15.7:0/3641234579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6048109950 msgr2=0x7f60481a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6048109950 0x7f60481a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 -- 172.21.15.7:0/3641234579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60481a5c60 con 0x7f6048107e40 2024-03-18T01:34:52.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045400640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048108f90 0x7f604819e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6046800640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6048109950 0x7f60481a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.013+0000 7f6045e00640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6048107e40 0x7f604819e040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f603800ea60 tx=0x7f603800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.014+0000 7f603ea00640 1 -- 172.21.15.7:0/3641234579 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603800cde0 con 0x7f6048107e40 2024-03-18T01:34:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.014+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60481a5f50 con 0x7f6048107e40 2024-03-18T01:34:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.014+0000 7f603ea00640 1 -- 172.21.15.7:0/3641234579 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60380035f0 con 0x7f6048107e40 2024-03-18T01:34:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.014+0000 7f603ea00640 1 -- 172.21.15.7:0/3641234579 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6038010670 con 0x7f6048107e40 2024-03-18T01:34:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.014+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60481a6410 con 0x7f6048107e40 2024-03-18T01:34:52.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.015+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f604810f080 con 0x7f6048107e40 2024-03-18T01:34:52.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.017+0000 7f603ea00640 1 -- 172.21.15.7:0/3641234579 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f60380037a0 con 0x7f6048107e40 2024-03-18T01:34:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.019+0000 7f603ea00640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f602c0a2800 0x7f602c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.020+0000 7f603ea00640 1 -- 172.21.15.7:0/3641234579 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f60380ce950 con 0x7f6048107e40 2024-03-18T01:34:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.020+0000 7f6045400640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f602c0a2800 0x7f602c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.021+0000 7f6045400640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f602c0a2800 0x7f602c0a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f604819f550 tx=0x7f6034005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.025+0000 7f603ea00640 1 -- 172.21.15.7:0/3641234579 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f603809bdf0 con 0x7f6048107e40 2024-03-18T01:34:52.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.237+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6048062480 con 0x7f6048107e40 2024-03-18T01:34:52.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.238+0000 7f603ea00640 1 -- 172.21.15.7:0/3641234579 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18340 (secure 0 0 0) 0x7f603809b6f0 con 0x7f6048107e40 2024-03-18T01:34:52.239 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:52.239 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":81,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:33:58.842081+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:34:52.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f602c0a2800 msgr2=0x7f602c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f602c0a2800 0x7f602c0a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f604819f550 tx=0x7f6034005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6048107e40 msgr2=0x7f604819e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6048107e40 0x7f604819e040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f603800ea60 tx=0x7f603800ef20 comp rx=0 tx=0).stop 2024-03-18T01:34:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 shutdown_connections 2024-03-18T01:34:52.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6048109950 0x7f60481a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6048108f90 0x7f604819e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f602c0a2800 0x7f602c0a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 --2- 172.21.15.7:0/3641234579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6048107e40 0x7f604819e040 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 >> 172.21.15.7:0/3641234579 conn(0x7f6048076dd0 msgr2=0x7f604810c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 shutdown_connections 2024-03-18T01:34:52.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.241+0000 7f604d000640 1 -- 172.21.15.7:0/3641234579 wait complete. 2024-03-18T01:34:52.260 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T01:34:52.261 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:34:52.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.474+0000 7f854ac00640 1 -- 172.21.15.7:0/4034540688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544107e20 msgr2=0x7f8544108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.474+0000 7f854ac00640 1 --2- 172.21.15.7:0/4034540688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544107e20 0x7f8544108240 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8534009900 tx=0x7f853402bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:52.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.475+0000 7f854ac00640 1 -- 172.21.15.7:0/4034540688 shutdown_connections 2024-03-18T01:34:52.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.475+0000 7f854ac00640 1 --2- 172.21.15.7:0/4034540688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544109930 0x7f8544110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.475+0000 7f854ac00640 1 --2- 172.21.15.7:0/4034540688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8544108f70 0x7f85441093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.475+0000 7f854ac00640 1 --2- 172.21.15.7:0/4034540688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544107e20 0x7f8544108240 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.475+0000 7f854ac00640 1 -- 172.21.15.7:0/4034540688 >> 172.21.15.7:0/4034540688 conn(0x7f8544076dd0 msgr2=0x7f85440791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:52.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.475+0000 7f854ac00640 1 -- 172.21.15.7:0/4034540688 shutdown_connections 2024-03-18T01:34:52.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.475+0000 7f854ac00640 1 -- 172.21.15.7:0/4034540688 wait complete. 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.476+0000 7f854ac00640 1 Processor -- start 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.476+0000 7f854ac00640 1 -- start start 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.476+0000 7f854ac00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8544107e20 0x7f854419e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.476+0000 7f854ac00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544108f70 0x7f854419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.476+0000 7f854ac00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544109930 0x7f85441a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.476+0000 7f854ac00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8544113b10 con 0x7f8544108f70 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.476+0000 7f854ac00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8544113990 con 0x7f8544107e20 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.476+0000 7f854ac00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8544113c90 con 0x7f8544109930 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8544107e20 0x7f854419e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8544107e20 0x7f854419e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:51694/0 (socket says 172.21.15.7:51694) 2024-03-18T01:34:52.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8548c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544109930 0x7f85441a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 -- 172.21.15.7:0/329751458 learned_addr learned my addr 172.21.15.7:0/329751458 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543400640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544108f70 0x7f854419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 -- 172.21.15.7:0/329751458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544109930 msgr2=0x7f85441a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544109930 0x7f85441a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 -- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544108f70 msgr2=0x7f854419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544108f70 0x7f854419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 -- 172.21.15.7:0/329751458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85441a5d50 con 0x7f8544107e20 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8548c00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544109930 0x7f85441a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543400640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544108f70 0x7f854419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.477+0000 7f8543e00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8544107e20 0x7f854419e130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8534000c00 tx=0x7f853402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.478+0000 7f8540c00640 1 -- 172.21.15.7:0/329751458 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534043020 con 0x7f8544107e20 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.478+0000 7f8540c00640 1 -- 172.21.15.7:0/329751458 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85340054e0 con 0x7f8544107e20 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.478+0000 7f8540c00640 1 -- 172.21.15.7:0/329751458 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534003c40 con 0x7f8544107e20 2024-03-18T01:34:52.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.478+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85441a5fe0 con 0x7f8544107e20 2024-03-18T01:34:52.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.478+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85441a63f0 con 0x7f8544107e20 2024-03-18T01:34:52.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.479+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f854410f130 con 0x7f8544107e20 2024-03-18T01:34:52.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.481+0000 7f8540c00640 1 -- 172.21.15.7:0/329751458 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f8534046020 con 0x7f8544107e20 2024-03-18T01:34:52.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.482+0000 7f8540c00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f85280a2880 0x7f85280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.482+0000 7f8540c00640 1 -- 172.21.15.7:0/329751458 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f85340f06b0 con 0x7f8544107e20 2024-03-18T01:34:52.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.482+0000 7f8543400640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f85280a2880 0x7f85280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.483+0000 7f8543400640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f85280a2880 0x7f85280a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f854419f640 tx=0x7f8530005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:52.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.486+0000 7f8540c00640 1 -- 172.21.15.7:0/329751458 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f853404d020 con 0x7f8544107e20 2024-03-18T01:34:52.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.702+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8544062480 con 0x7f8544107e20 2024-03-18T01:34:52.703 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.703+0000 7f8540c00640 1 -- 172.21.15.7:0/329751458 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f85340bdb50 con 0x7f8544107e20 2024-03-18T01:34:52.703 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:34:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f85280a2880 msgr2=0x7f85280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f85280a2880 0x7f85280a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f854419f640 tx=0x7f8530005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:52.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8544107e20 msgr2=0x7f854419e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8544107e20 0x7f854419e130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8534000c00 tx=0x7f853402c650 comp rx=0 tx=0).stop 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 shutdown_connections 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8544109930 0x7f85441a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f85280a2880 0x7f85280a4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8544108f70 0x7f854419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 --2- 172.21.15.7:0/329751458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8544107e20 0x7f854419e130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.705+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 >> 172.21.15.7:0/329751458 conn(0x7f8544076dd0 msgr2=0x7f854410c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.706+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 shutdown_connections 2024-03-18T01:34:52.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.706+0000 7f854ac00640 1 -- 172.21.15.7:0/329751458 wait complete. 2024-03-18T01:34:52.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:52 smithi007 ceph-mon[31996]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:52.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:52 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3928077385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:52.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:52.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:52 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3641234579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:52.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T01:34:52.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:52 smithi046 ceph-mon[36108]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:52 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3928077385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:52 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3641234579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:52.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:52 smithi203 ceph-mon[36345]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:52.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:52 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3928077385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:34:52.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:52.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:52 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3641234579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:34:52.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 -- 172.21.15.7:0/2421772047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 msgr2=0x7f1068103eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 --2- 172.21.15.7:0/2421772047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 0x7f1068103eb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1058009900 tx=0x7f105802bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:52.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 -- 172.21.15.7:0/2421772047 shutdown_connections 2024-03-18T01:34:52.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 --2- 172.21.15.7:0/2421772047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1068105640 0x7f106810bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 --2- 172.21.15.7:0/2421772047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104c80 0x7f1068105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 --2- 172.21.15.7:0/2421772047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 0x7f1068103eb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 -- 172.21.15.7:0/2421772047 >> 172.21.15.7:0/2421772047 conn(0x7f10680ff040 msgr2=0x7f1068101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:52.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 -- 172.21.15.7:0/2421772047 shutdown_connections 2024-03-18T01:34:52.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.927+0000 7f106da00640 1 -- 172.21.15.7:0/2421772047 wait complete. 2024-03-18T01:34:52.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.928+0000 7f106da00640 1 Processor -- start 2024-03-18T01:34:52.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.928+0000 7f106da00640 1 -- start start 2024-03-18T01:34:52.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.928+0000 7f106da00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 0x7f1068199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.928+0000 7f106da00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104c80 0x7f106819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f106da00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1068105640 0x7f10681a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f106da00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f106810f6c0 con 0x7f1068103a90 2024-03-18T01:34:52.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f106da00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f106810f540 con 0x7f1068105640 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f106da00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f106810f840 con 0x7f1068104c80 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 0x7f1068199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 0x7f1068199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39592/0 (socket says 172.21.15.7:39592) 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066a00640 1 -- 172.21.15.7:0/2991086910 learned_addr learned my addr 172.21.15.7:0/2991086910 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1067400640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1068105640 0x7f10681a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066000640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104c80 0x7f106819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066a00640 1 -- 172.21.15.7:0/2991086910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104c80 msgr2=0x7f106819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066a00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104c80 0x7f106819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066a00640 1 -- 172.21.15.7:0/2991086910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1068105640 msgr2=0x7f10681a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066a00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1068105640 0x7f10681a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.929+0000 7f1066a00640 1 -- 172.21.15.7:0/2991086910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10681a19c0 con 0x7f1068103a90 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.930+0000 7f1066a00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 0x7f1068199db0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1058000c00 tx=0x7f105802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.930+0000 7f105f400640 1 -- 172.21.15.7:0/2991086910 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058043020 con 0x7f1068103a90 2024-03-18T01:34:52.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.930+0000 7f105f400640 1 -- 172.21.15.7:0/2991086910 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10580054e0 con 0x7f1068103a90 2024-03-18T01:34:52.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.930+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10681a1c50 con 0x7f1068103a90 2024-03-18T01:34:52.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.930+0000 7f105f400640 1 -- 172.21.15.7:0/2991086910 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058003bd0 con 0x7f1068103a90 2024-03-18T01:34:52.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.930+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10681a21a0 con 0x7f1068103a90 2024-03-18T01:34:52.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.932+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f106810ae00 con 0x7f1068103a90 2024-03-18T01:34:52.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.934+0000 7f105f400640 1 -- 172.21.15.7:0/2991086910 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f1058046020 con 0x7f1068103a90 2024-03-18T01:34:52.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.936+0000 7f105f400640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f104c0a2880 0x7f104c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:52.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.937+0000 7f105f400640 1 -- 172.21.15.7:0/2991086910 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f10580f09e0 con 0x7f1068103a90 2024-03-18T01:34:52.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.937+0000 7f1066000640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f104c0a2880 0x7f104c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:52.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.937+0000 7f1066000640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f104c0a2880 0x7f104c0a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1054009bd0 tx=0x7f1054005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:52.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:52.940+0000 7f105f400640 1 -- 172.21.15.7:0/2991086910 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f105804d020 con 0x7f1068103a90 2024-03-18T01:34:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.172+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f106802d330 con 0x7f1068103a90 2024-03-18T01:34:53.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.173+0000 7f105f400640 1 -- 172.21.15.7:0/2991086910 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f10580bdf20 con 0x7f1068103a90 2024-03-18T01:34:53.173 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:34:53.174 INFO:teuthology.orchestra.run.smithi007.stdout:[ 2024-03-18T01:34:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:34:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "poolnum": 1, 2024-03-18T01:34:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: "poolname": ".mgr" 2024-03-18T01:34:53.174 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:34:53.174 INFO:teuthology.orchestra.run.smithi007.stdout:] 2024-03-18T01:34:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f104c0a2880 msgr2=0x7f104c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f104c0a2880 0x7f104c0a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1054009bd0 tx=0x7f1054005f90 comp rx=0 tx=0).stop 2024-03-18T01:34:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 msgr2=0x7f1068199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 0x7f1068199db0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1058000c00 tx=0x7f105802c650 comp rx=0 tx=0).stop 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 shutdown_connections 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1068105640 0x7f10681a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1068104c80 0x7f106819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f104c0a2880 0x7f104c0a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 --2- 172.21.15.7:0/2991086910 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1068103a90 0x7f1068199db0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 >> 172.21.15.7:0/2991086910 conn(0x7f10680ff040 msgr2=0x7f1068107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 shutdown_connections 2024-03-18T01:34:53.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.177+0000 7f106da00640 1 -- 172.21.15.7:0/2991086910 wait complete. 2024-03-18T01:34:53.205 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T01:34:53.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.404+0000 7f191ca00640 1 -- 172.21.15.7:0/1903962183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918104c00 msgr2=0x7f1918105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.404+0000 7f191ca00640 1 --2- 172.21.15.7:0/1903962183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918104c00 0x7f1918105080 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1904009900 tx=0x7f190402bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.404+0000 7f191ca00640 1 -- 172.21.15.7:0/1903962183 shutdown_connections 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.404+0000 7f191ca00640 1 --2- 172.21.15.7:0/1903962183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19181055c0 0x7f191810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.404+0000 7f191ca00640 1 --2- 172.21.15.7:0/1903962183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918104c00 0x7f1918105080 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.404+0000 7f191ca00640 1 --2- 172.21.15.7:0/1903962183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1918103a10 0x7f1918103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.404+0000 7f191ca00640 1 -- 172.21.15.7:0/1903962183 >> 172.21.15.7:0/1903962183 conn(0x7f19180fefa0 msgr2=0x7f19181013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.404+0000 7f191ca00640 1 -- 172.21.15.7:0/1903962183 shutdown_connections 2024-03-18T01:34:53.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.405+0000 7f191ca00640 1 -- 172.21.15.7:0/1903962183 wait complete. 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.405+0000 7f191ca00640 1 Processor -- start 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f191ca00640 1 -- start start 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f191ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918103a10 0x7f1918199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f191ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1918104c00 0x7f191819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f191ca00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19181055c0 0x7f19181a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f191ca00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f191810f6c0 con 0x7f19181055c0 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f191ca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f191810f540 con 0x7f1918103a10 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f191ca00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f191810f840 con 0x7f1918104c00 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f1916800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19181055c0 0x7f19181a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f1915e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918103a10 0x7f1918199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:53.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f1915400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1918104c00 0x7f191819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f1915e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918103a10 0x7f1918199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:51738/0 (socket says 172.21.15.7:51738) 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f1916800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19181055c0 0x7f19181a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39612/0 (socket says 172.21.15.7:39612) 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.406+0000 7f1915e00640 1 -- 172.21.15.7:0/3100185911 learned_addr learned my addr 172.21.15.7:0/3100185911 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f1915e00640 1 -- 172.21.15.7:0/3100185911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1918104c00 msgr2=0x7f191819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f1915e00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1918104c00 0x7f191819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f1915e00640 1 -- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19181055c0 msgr2=0x7f19181a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f1915e00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19181055c0 0x7f19181a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f1915e00640 1 -- 172.21.15.7:0/3100185911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19181a1980 con 0x7f1918103a10 2024-03-18T01:34:53.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f1915e00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918103a10 0x7f1918199d60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f190800e9c0 tx=0x7f190800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:53.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f190ea00640 1 -- 172.21.15.7:0/3100185911 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190800cd80 con 0x7f1918103a10 2024-03-18T01:34:53.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19181a1c70 con 0x7f1918103a10 2024-03-18T01:34:53.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f190ea00640 1 -- 172.21.15.7:0/3100185911 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19080035f0 con 0x7f1918103a10 2024-03-18T01:34:53.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.407+0000 7f190ea00640 1 -- 172.21.15.7:0/3100185911 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908010690 con 0x7f1918103a10 2024-03-18T01:34:53.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.408+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19181a2190 con 0x7f1918103a10 2024-03-18T01:34:53.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.409+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f191810ad30 con 0x7f1918103a10 2024-03-18T01:34:53.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.411+0000 7f190ea00640 1 -- 172.21.15.7:0/3100185911 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f19080037a0 con 0x7f1918103a10 2024-03-18T01:34:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.412+0000 7f190ea00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f18fc0a2880 0x7f18fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.412+0000 7f190ea00640 1 -- 172.21.15.7:0/3100185911 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f19080ce990 con 0x7f1918103a10 2024-03-18T01:34:53.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.413+0000 7f1915400640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f18fc0a2880 0x7f18fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:53.413 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.413+0000 7f1915400640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f18fc0a2880 0x7f18fc0a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f190402c240 tx=0x7f190402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:53.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.417+0000 7f190ea00640 1 -- 172.21.15.7:0/3100185911 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f190809be30 con 0x7f1918103a10 2024-03-18T01:34:53.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:53.646+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f191802d1b0 con 0x7f1918103a10 2024-03-18T01:34:53.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:53 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:53 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:53 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:53 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:53 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/329751458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:53.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:53 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2991086910' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T01:34:53.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:53 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:53 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:53 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:53 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:53 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/329751458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:53.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:53 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2991086910' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T01:34:53.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:53 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:53 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:53 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:53 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:53.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:53 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/329751458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:34:53.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:53 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2991086910' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T01:34:54.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.598+0000 7f190ea00640 1 -- 172.21.15.7:0/3100185911 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f190809b730 con 0x7f1918103a10 2024-03-18T01:34:54.599 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' created 2024-03-18T01:34:54.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f18fc0a2880 msgr2=0x7f18fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f18fc0a2880 0x7f18fc0a4d40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f190402c240 tx=0x7f190402df90 comp rx=0 tx=0).stop 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918103a10 msgr2=0x7f1918199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918103a10 0x7f1918199d60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f190800e9c0 tx=0x7f190800ee80 comp rx=0 tx=0).stop 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 shutdown_connections 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f18fc0a2880 0x7f18fc0a4d40 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f190402c240 tx=0x7f190402df90 comp rx=0 tx=0).stop 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f19181055c0 0x7f19181a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1918104c00 0x7f191819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 --2- 172.21.15.7:0/3100185911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1918103a10 0x7f1918199d60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:54.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.605+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 >> 172.21.15.7:0/3100185911 conn(0x7f19180fefa0 msgr2=0x7f1918107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.608+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 shutdown_connections 2024-03-18T01:34:54.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.608+0000 7f191ca00640 1 -- 172.21.15.7:0/3100185911 wait complete. 2024-03-18T01:34:54.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-18T01:34:54.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 -- 172.21.15.7:0/2578921918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304102340 msgr2=0x7f4304104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:54.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 --2- 172.21.15.7:0/2578921918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304102340 0x7f4304104730 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009900 tx=0x7f42f402bc50 comp rx=0 tx=0).stop 2024-03-18T01:34:54.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 -- 172.21.15.7:0/2578921918 shutdown_connections 2024-03-18T01:34:54.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 --2- 172.21.15.7:0/2578921918 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43041075a0 0x7f4304109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:54.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 --2- 172.21.15.7:0/2578921918 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4304104c70 0x7f4304107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:54.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 --2- 172.21.15.7:0/2578921918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304102340 0x7f4304104730 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:54.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 -- 172.21.15.7:0/2578921918 >> 172.21.15.7:0/2578921918 conn(0x7f43040fbc90 msgr2=0x7f43040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:54.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 -- 172.21.15.7:0/2578921918 shutdown_connections 2024-03-18T01:34:54.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.829+0000 7f430ac00640 1 -- 172.21.15.7:0/2578921918 wait complete. 2024-03-18T01:34:54.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.830+0000 7f430ac00640 1 Processor -- start 2024-03-18T01:34:54.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.830+0000 7f430ac00640 1 -- start start 2024-03-18T01:34:54.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.830+0000 7f430ac00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4304102340 0x7f4304199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:54.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.830+0000 7f430ac00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304104c70 0x7f430419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:54.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.830+0000 7f430ac00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43041075a0 0x7f43041a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:54.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f430ac00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f430410d500 con 0x7f4304102340 2024-03-18T01:34:54.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f430ac00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f430410d380 con 0x7f4304104c70 2024-03-18T01:34:54.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f430ac00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f430410d680 con 0x7f43041075a0 2024-03-18T01:34:54.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304104c70 0x7f430419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:54.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304104c70 0x7f430419a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:51752/0 (socket says 172.21.15.7:51752) 2024-03-18T01:34:54.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4304102340 0x7f4304199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:54.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 -- 172.21.15.7:0/3123653520 learned_addr learned my addr 172.21.15.7:0/3123653520 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:54.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4308c00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43041075a0 0x7f43041a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:54.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 -- 172.21.15.7:0/3123653520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43041075a0 msgr2=0x7f43041a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:54.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43041075a0 0x7f43041a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:54.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 -- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4304102340 msgr2=0x7f4304199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:54.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4304102340 0x7f4304199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:54.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 -- 172.21.15.7:0/3123653520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43041a1950 con 0x7f4304104c70 2024-03-18T01:34:54.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.831+0000 7f4303400640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304104c70 0x7f430419a2f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f42f000e9c0 tx=0x7f42f000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:54.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.832+0000 7f4300c00640 1 -- 172.21.15.7:0/3123653520 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f000cd80 con 0x7f4304104c70 2024-03-18T01:34:54.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.832+0000 7f4300c00640 1 -- 172.21.15.7:0/3123653520 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42f0005500 con 0x7f4304104c70 2024-03-18T01:34:54.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.832+0000 7f4300c00640 1 -- 172.21.15.7:0/3123653520 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f0010690 con 0x7f4304104c70 2024-03-18T01:34:54.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.832+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43041a1c40 con 0x7f4304104c70 2024-03-18T01:34:54.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.832+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43041a21b0 con 0x7f4304104c70 2024-03-18T01:34:54.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.833+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4304109090 con 0x7f4304104c70 2024-03-18T01:34:54.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.835+0000 7f4300c00640 1 -- 172.21.15.7:0/3123653520 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f42f00046c0 con 0x7f4304104c70 2024-03-18T01:34:54.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.837+0000 7f4300c00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f42e80a2880 0x7f42e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:54.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.837+0000 7f4300c00640 1 -- 172.21.15.7:0/3123653520 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f42f00ceb40 con 0x7f4304104c70 2024-03-18T01:34:54.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.837+0000 7f4303e00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f42e80a2880 0x7f42e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:54.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.838+0000 7f4303e00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f42e80a2880 0x7f42e80a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f42f402c260 tx=0x7f42f4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:54.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:54.841+0000 7f4300c00640 1 -- 172.21.15.7:0/3123653520 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f42f009be70 con 0x7f4304104c70 2024-03-18T01:34:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:54.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3100185911' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T01:34:54.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T01:34:54.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:55.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3100185911' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T01:34:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T01:34:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:55.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.047+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f430402d1b0 con 0x7f4304104c70 2024-03-18T01:34:55.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3100185911' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T01:34:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T01:34:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:34:55.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.633+0000 7f4300c00640 1 -- 172.21.15.7:0/3123653520 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f42f009b770 con 0x7f4304104c70 2024-03-18T01:34:55.635 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' created 2024-03-18T01:34:55.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.639+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f42e80a2880 msgr2=0x7f42e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:55.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.639+0000 7f430ac00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f42e80a2880 0x7f42e80a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f42f402c260 tx=0x7f42f4000bc0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.639+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304104c70 msgr2=0x7f430419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:55.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.639+0000 7f430ac00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304104c70 0x7f430419a2f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f42f000e9c0 tx=0x7f42f000ee80 comp rx=0 tx=0).stop 2024-03-18T01:34:55.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.640+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 shutdown_connections 2024-03-18T01:34:55.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.640+0000 7f430ac00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43041075a0 0x7f43041a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.640+0000 7f430ac00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4304104c70 0x7f430419a2f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.640+0000 7f430ac00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f42e80a2880 0x7f42e80a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.640+0000 7f430ac00640 1 --2- 172.21.15.7:0/3123653520 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4304102340 0x7f4304199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.640+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 >> 172.21.15.7:0/3123653520 conn(0x7f43040fbc90 msgr2=0x7f4304101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:55.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.644+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 shutdown_connections 2024-03-18T01:34:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.644+0000 7f430ac00640 1 -- 172.21.15.7:0/3123653520 wait complete. 2024-03-18T01:34:55.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T01:34:55.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:55 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T01:34:55.734 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:55 smithi007 ceph-mon[31996]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.859+0000 7effff000640 1 -- 172.21.15.7:0/1359289101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff8104c00 msgr2=0x7efff8105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.859+0000 7effff000640 1 --2- 172.21.15.7:0/1359289101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff8104c00 0x7efff8105080 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7effe4009900 tx=0x7effe402bc30 comp rx=0 tx=0).stop 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.859+0000 7effff000640 1 -- 172.21.15.7:0/1359289101 shutdown_connections 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.859+0000 7effff000640 1 --2- 172.21.15.7:0/1359289101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81055c0 0x7efff810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.859+0000 7effff000640 1 --2- 172.21.15.7:0/1359289101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff8104c00 0x7efff8105080 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.859+0000 7effff000640 1 --2- 172.21.15.7:0/1359289101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efff8103a10 0x7efff8103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.859+0000 7effff000640 1 -- 172.21.15.7:0/1359289101 >> 172.21.15.7:0/1359289101 conn(0x7efff80fefa0 msgr2=0x7efff81013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.859+0000 7effff000640 1 -- 172.21.15.7:0/1359289101 shutdown_connections 2024-03-18T01:34:55.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.860+0000 7effff000640 1 -- 172.21.15.7:0/1359289101 wait complete. 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.860+0000 7effff000640 1 Processor -- start 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7effff000640 1 -- start start 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7effff000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efff8103a10 0x7efff8199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7effff000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81055c0 0x7efff819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7effff000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff819f4c0 0x7efff81a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7effff000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efff810f6c0 con 0x7efff8103a10 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7effff000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efff810f540 con 0x7efff819f4c0 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7effff000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efff810f840 con 0x7efff81055c0 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7efff7e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efff8103a10 0x7efff8199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7efff7e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efff8103a10 0x7efff8199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44208/0 (socket says 172.21.15.7:44208) 2024-03-18T01:34:55.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7efff7e00640 1 -- 172.21.15.7:0/1826404988 learned_addr learned my addr 172.21.15.7:0/1826404988 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7efffd000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff819f4c0 0x7efff81a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.861+0000 7efff7400640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81055c0 0x7efff819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff7e00640 1 -- 172.21.15.7:0/1826404988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81055c0 msgr2=0x7efff819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff7e00640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81055c0 0x7efff819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff7e00640 1 -- 172.21.15.7:0/1826404988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff819f4c0 msgr2=0x7efff81a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff7e00640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff819f4c0 0x7efff81a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:55.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff7e00640 1 -- 172.21.15.7:0/1826404988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff81a1f20 con 0x7efff8103a10 2024-03-18T01:34:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efffd000640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff819f4c0 0x7efff81a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff7e00640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efff8103a10 0x7efff8199f20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7effe800d770 tx=0x7effe800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff4c00640 1 -- 172.21.15.7:0/1826404988 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8017070 con 0x7efff8103a10 2024-03-18T01:34:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff81a2210 con 0x7efff8103a10 2024-03-18T01:34:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff4c00640 1 -- 172.21.15.7:0/1826404988 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effe8009c30 con 0x7efff8103a10 2024-03-18T01:34:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.862+0000 7efff4c00640 1 -- 172.21.15.7:0/1826404988 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8003dd0 con 0x7efff8103a10 2024-03-18T01:34:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.863+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff81a2780 con 0x7efff8103a10 2024-03-18T01:34:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.864+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff810ad80 con 0x7efff8103a10 2024-03-18T01:34:55.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.866+0000 7efff4c00640 1 -- 172.21.15.7:0/1826404988 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7effe8003f70 con 0x7efff8103a10 2024-03-18T01:34:55.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.867+0000 7efff4c00640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7effdc0a2800 0x7effdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:55.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.867+0000 7efff4c00640 1 -- 172.21.15.7:0/1826404988 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7effe80ce5c0 con 0x7efff8103a10 2024-03-18T01:34:55.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.867+0000 7efff7400640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7effdc0a2800 0x7effdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:55.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.868+0000 7efff7400640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7effdc0a2800 0x7effdc0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7effe4009230 tx=0x7effe402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:55.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:55.871+0000 7efff4c00640 1 -- 172.21.15.7:0/1826404988 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7effe8009de0 con 0x7efff8103a10 2024-03-18T01:34:55.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:55 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T01:34:55.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:55 smithi203 ceph-mon[36345]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T01:34:56.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:55 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T01:34:56.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:55 smithi046 ceph-mon[36108]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T01:34:56.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:56.124+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7efff802d370 con 0x7efff8103a10 2024-03-18T01:34:56.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3123653520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T01:34:56.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T01:34:56.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: pgmap v311: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1826404988' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T01:34:56.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:56 smithi203 ceph-mon[36345]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3123653520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: pgmap v311: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1826404988' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T01:34:57.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:56 smithi046 ceph-mon[36108]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T01:34:57.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.050+0000 7efff4c00640 1 -- 172.21.15.7:0/1826404988 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7effe809b1c0 con 0x7efff8103a10 2024-03-18T01:34:57.051 INFO:teuthology.orchestra.run.smithi007.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T01:34:57.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3123653520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T01:34:57.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: pgmap v311: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1826404988' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T01:34:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:56 smithi007 ceph-mon[31996]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T01:34:57.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7effdc0a2800 msgr2=0x7effdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7effdc0a2800 0x7effdc0a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7effe4009230 tx=0x7effe402df90 comp rx=0 tx=0).stop 2024-03-18T01:34:57.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efff8103a10 msgr2=0x7efff8199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efff8103a10 0x7efff8199f20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7effe800d770 tx=0x7effe800dc30 comp rx=0 tx=0).stop 2024-03-18T01:34:57.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 shutdown_connections 2024-03-18T01:34:57.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efff819f4c0 0x7efff81a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efff81055c0 0x7efff819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7effdc0a2800 0x7effdc0a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 --2- 172.21.15.7:0/1826404988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efff8103a10 0x7efff8199f20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 >> 172.21.15.7:0/1826404988 conn(0x7efff80fefa0 msgr2=0x7efff8107f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:57.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 shutdown_connections 2024-03-18T01:34:57.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.054+0000 7effff000640 1 -- 172.21.15.7:0/1826404988 wait complete. 2024-03-18T01:34:57.081 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-18T01:34:57.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.287+0000 7f35cfe00640 1 -- 172.21.15.7:0/434422721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d01049e0 msgr2=0x7f35d0106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.287+0000 7f35cfe00640 1 --2- 172.21.15.7:0/434422721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d01049e0 0x7f35d0106dd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f35b8009900 tx=0x7f35b802bc40 comp rx=0 tx=0).stop 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.288+0000 7f35cfe00640 1 -- 172.21.15.7:0/434422721 shutdown_connections 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.288+0000 7f35cfe00640 1 --2- 172.21.15.7:0/434422721 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35d0107310 0x7f35d0109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.288+0000 7f35cfe00640 1 --2- 172.21.15.7:0/434422721 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d01049e0 0x7f35d0106dd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.288+0000 7f35cfe00640 1 --2- 172.21.15.7:0/434422721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35d01020b0 0x7f35d01044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.288+0000 7f35cfe00640 1 -- 172.21.15.7:0/434422721 >> 172.21.15.7:0/434422721 conn(0x7f35d00fbac0 msgr2=0x7f35d00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.288+0000 7f35cfe00640 1 -- 172.21.15.7:0/434422721 shutdown_connections 2024-03-18T01:34:57.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.288+0000 7f35cfe00640 1 -- 172.21.15.7:0/434422721 wait complete. 2024-03-18T01:34:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.289+0000 7f35cfe00640 1 Processor -- start 2024-03-18T01:34:57.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.289+0000 7f35cfe00640 1 -- start start 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35d01020b0 0x7f35d0101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35d01049e0 0x7f35d0100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d0107310 0x7f35d01005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35cfe00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f35d010d2e0 con 0x7f35d0107310 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35cfe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35d010d160 con 0x7f35d01020b0 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35cfe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35d010d460 con 0x7f35d01049e0 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35c6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35d01049e0 0x7f35d0100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35cf400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d0107310 0x7f35d01005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:57.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35c6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35d01049e0 0x7f35d0100060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:37466/0 (socket says 172.21.15.7:37466) 2024-03-18T01:34:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35c6000640 1 -- 172.21.15.7:0/1025436817 learned_addr learned my addr 172.21.15.7:0/1025436817 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:57.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35cea00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35d01020b0 0x7f35d0101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.290+0000 7f35c6000640 1 -- 172.21.15.7:0/1025436817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35d01020b0 msgr2=0x7f35d0101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35c6000640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35d01020b0 0x7f35d0101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35c6000640 1 -- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d0107310 msgr2=0x7f35d01005a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35c6000640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d0107310 0x7f35d01005a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35c6000640 1 -- 172.21.15.7:0/1025436817 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d0100e30 con 0x7f35d01049e0 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35cea00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35d01020b0 0x7f35d0101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35c6000640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35d01049e0 0x7f35d0100060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35b802efd0 tx=0x7f35b802c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35c7e00640 1 -- 172.21.15.7:0/1025436817 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8041070 con 0x7f35d01049e0 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d01a1a20 con 0x7f35d01049e0 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35c7e00640 1 -- 172.21.15.7:0/1025436817 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35b8004450 con 0x7f35d01049e0 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.291+0000 7f35c7e00640 1 -- 172.21.15.7:0/1025436817 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8003e50 con 0x7f35d01049e0 2024-03-18T01:34:57.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.292+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d01a1f40 con 0x7f35d01049e0 2024-03-18T01:34:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.293+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35d00fd5b0 con 0x7f35d01049e0 2024-03-18T01:34:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.295+0000 7f35c7e00640 1 -- 172.21.15.7:0/1025436817 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f35b8044070 con 0x7f35d01049e0 2024-03-18T01:34:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.297+0000 7f35c7e00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35b40a29a0 0x7f35b40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.297+0000 7f35c7e00640 1 -- 172.21.15.7:0/1025436817 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f35b80f04a0 con 0x7f35d01049e0 2024-03-18T01:34:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.297+0000 7f35cea00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35b40a29a0 0x7f35b40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:57.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.298+0000 7f35cea00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35b40a29a0 0x7f35b40a4e60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005fd0 tx=0x7f35c0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:57.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.301+0000 7f35c7e00640 1 -- 172.21.15.7:0/1025436817 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f35b80bcfa0 con 0x7f35d01049e0 2024-03-18T01:34:57.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.517+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f35d00623e0 con 0x7f35d01049e0 2024-03-18T01:34:57.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.642+0000 7f35c7e00640 1 -- 172.21.15.7:0/1025436817 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f35b80c1040 con 0x7f35d01049e0 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35b40a29a0 msgr2=0x7f35b40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35b40a29a0 0x7f35b40a4e60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f35c0005fd0 tx=0x7f35c0009500 comp rx=0 tx=0).stop 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35d01049e0 msgr2=0x7f35d0100060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35d01049e0 0x7f35d0100060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35b802efd0 tx=0x7f35b802c660 comp rx=0 tx=0).stop 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 shutdown_connections 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35b40a29a0 0x7f35b40a4e60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35d0107310 0x7f35d01005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35d01049e0 0x7f35d0100060 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 --2- 172.21.15.7:0/1025436817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35d01020b0 0x7f35d0101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 >> 172.21.15.7:0/1025436817 conn(0x7f35d00fbac0 msgr2=0x7f35d0102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 shutdown_connections 2024-03-18T01:34:57.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.648+0000 7f35cfe00640 1 -- 172.21.15.7:0/1025436817 wait complete. 2024-03-18T01:34:57.670 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T01:34:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T01:34:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1826404988' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T01:34:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T01:34:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T01:34:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1025436817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T01:34:57.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:57 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T01:34:57.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 -- 172.21.15.7:0/3535846606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a0108ef0 msgr2=0x7f56a0109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 --2- 172.21.15.7:0/3535846606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a0108ef0 0x7f56a0109370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f568c009900 tx=0x7f568c02bc50 comp rx=0 tx=0).stop 2024-03-18T01:34:57.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 -- 172.21.15.7:0/3535846606 shutdown_connections 2024-03-18T01:34:57.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 --2- 172.21.15.7:0/3535846606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56a01098b0 0x7f56a0110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 --2- 172.21.15.7:0/3535846606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a0108ef0 0x7f56a0109370 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 --2- 172.21.15.7:0/3535846606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56a0107d00 0x7f56a0108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 -- 172.21.15.7:0/3535846606 >> 172.21.15.7:0/3535846606 conn(0x7f56a0076ca0 msgr2=0x7f56a00790e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:34:57.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 -- 172.21.15.7:0/3535846606 shutdown_connections 2024-03-18T01:34:57.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.904+0000 7f56a4e00640 1 -- 172.21.15.7:0/3535846606 wait complete. 2024-03-18T01:34:57.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.905+0000 7f56a4e00640 1 Processor -- start 2024-03-18T01:34:57.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.905+0000 7f56a4e00640 1 -- start start 2024-03-18T01:34:57.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f56a4e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a0107d00 0x7f56a019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:57.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f56a4e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56a0108ef0 0x7f56a019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:57.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f56a4e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56a01098b0 0x7f56a01a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:57.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f56a4e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f56a01139d0 con 0x7f56a0108ef0 2024-03-18T01:34:57.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f56a4e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56a0113850 con 0x7f56a01098b0 2024-03-18T01:34:57.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f56a4e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f56a0113b50 con 0x7f56a0107d00 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f569ea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56a0108ef0 0x7f56a019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f569fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56a01098b0 0x7f56a01a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f569ea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56a0108ef0 0x7f56a019e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44246/0 (socket says 172.21.15.7:44246) 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f569ea00640 1 -- 172.21.15.7:0/4110288279 learned_addr learned my addr 172.21.15.7:0/4110288279 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.906+0000 7f569f400640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a0107d00 0x7f56a019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f569fe00640 1 -- 172.21.15.7:0/4110288279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a0107d00 msgr2=0x7f56a019dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f569fe00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a0107d00 0x7f56a019dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f569fe00640 1 -- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56a0108ef0 msgr2=0x7f56a019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f569fe00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56a0108ef0 0x7f56a019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f569fe00640 1 -- 172.21.15.7:0/4110288279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56a01a5bd0 con 0x7f56a01098b0 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f569fe00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56a01098b0 0x7f56a01a5560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5698009fd0 tx=0x7f569800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f5697e00640 1 -- 172.21.15.7:0/4110288279 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569800ce20 con 0x7f56a01098b0 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56a01a5ec0 con 0x7f56a01098b0 2024-03-18T01:34:57.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f5697e00640 1 -- 172.21.15.7:0/4110288279 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5698005020 con 0x7f56a01098b0 2024-03-18T01:34:57.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f5697e00640 1 -- 172.21.15.7:0/4110288279 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5698010720 con 0x7f56a01098b0 2024-03-18T01:34:57.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.907+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56a01a6380 con 0x7f56a01098b0 2024-03-18T01:34:57.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.908+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56a010efc0 con 0x7f56a01098b0 2024-03-18T01:34:57.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.910+0000 7f5697e00640 1 -- 172.21.15.7:0/4110288279 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f5698005540 con 0x7f56a01098b0 2024-03-18T01:34:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.912+0000 7f5697e00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56840a27f0 0x7f56840a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:34:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.912+0000 7f5697e00640 1 -- 172.21.15.7:0/4110288279 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f56980cf110 con 0x7f56a01098b0 2024-03-18T01:34:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.912+0000 7f569f400640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56840a27f0 0x7f56840a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:34:57.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.913+0000 7f569f400640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56840a27f0 0x7f56840a4cb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f56a0108d50 tx=0x7f5690005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:34:57.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:57.916+0000 7f5697e00640 1 -- 172.21.15.7:0/4110288279 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f569809c120 con 0x7f56a01098b0 2024-03-18T01:34:58.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1826404988' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1025436817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T01:34:58.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:57 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T01:34:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T01:34:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1826404988' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T01:34:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T01:34:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:34:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T01:34:58.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:34:58.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:34:58.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1025436817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T01:34:58.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:57 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T01:34:58.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:34:58.125+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f56a00623e0 con 0x7f56a01098b0 2024-03-18T01:34:59.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: pgmap v314: 97 pgs: 10 active+clean, 23 creating+peering, 64 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: Cluster is now healthy 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/4110288279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T01:34:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:34:58 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T01:34:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: pgmap v314: 97 pgs: 10 active+clean, 23 creating+peering, 64 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T01:34:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T01:34:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:34:59.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:34:59.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:34:59.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T01:34:59.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: Cluster is now healthy 2024-03-18T01:34:59.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4110288279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T01:34:59.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:34:58 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T01:34:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: pgmap v314: 97 pgs: 10 active+clean, 23 creating+peering, 64 unknown; 577 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:34:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T01:34:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T01:34:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:34:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:34:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:34:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T01:34:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: Cluster is now healthy 2024-03-18T01:34:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4110288279' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T01:34:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:34:58 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T01:35:00.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.103+0000 7f5697e00640 1 -- 172.21.15.7:0/4110288279 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f569809ba20 con 0x7f56a01098b0 2024-03-18T01:35:00.104 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data_ec' created 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.108+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56840a27f0 msgr2=0x7f56840a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.108+0000 7f56a4e00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56840a27f0 0x7f56840a4cb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f56a0108d50 tx=0x7f5690005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.108+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56a01098b0 msgr2=0x7f56a01a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.108+0000 7f56a4e00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56a01098b0 0x7f56a01a5560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5698009fd0 tx=0x7f569800ef10 comp rx=0 tx=0).stop 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.109+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 shutdown_connections 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.109+0000 7f56a4e00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56a01098b0 0x7f56a01a5560 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.109+0000 7f56a4e00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56840a27f0 0x7f56840a4cb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.109+0000 7f56a4e00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56a0108ef0 0x7f56a019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.109+0000 7f56a4e00640 1 --2- 172.21.15.7:0/4110288279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56a0107d00 0x7f56a019dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.109+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 >> 172.21.15.7:0/4110288279 conn(0x7f56a0076ca0 msgr2=0x7f56a010c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:00.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.109+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 shutdown_connections 2024-03-18T01:35:00.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.109+0000 7f56a4e00640 1 -- 172.21.15.7:0/4110288279 wait complete. 2024-03-18T01:35:00.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-18T01:35:00.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.324+0000 7f57b2400640 1 -- 172.21.15.7:0/1503444484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 msgr2=0x7f57ac073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:00.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.324+0000 7f57b2400640 1 --2- 172.21.15.7:0/1503444484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 0x7f57ac073c70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009900 tx=0x7f57a002bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:00.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.325+0000 7f57b2400640 1 -- 172.21.15.7:0/1503444484 shutdown_connections 2024-03-18T01:35:00.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.325+0000 7f57b2400640 1 --2- 172.21.15.7:0/1503444484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57ac1094e0 0x7f57ac10b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.325+0000 7f57b2400640 1 --2- 172.21.15.7:0/1503444484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57ac074240 0x7f57ac0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.325+0000 7f57b2400640 1 --2- 172.21.15.7:0/1503444484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 0x7f57ac073c70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.325+0000 7f57b2400640 1 -- 172.21.15.7:0/1503444484 >> 172.21.15.7:0/1503444484 conn(0x7f57ac0fcb90 msgr2=0x7f57ac0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:00.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.325+0000 7f57b2400640 1 -- 172.21.15.7:0/1503444484 shutdown_connections 2024-03-18T01:35:00.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.325+0000 7f57b2400640 1 -- 172.21.15.7:0/1503444484 wait complete. 2024-03-18T01:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b2400640 1 Processor -- start 2024-03-18T01:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b2400640 1 -- start start 2024-03-18T01:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b2400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57ac074240 0x7f57ac199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b2400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 0x7f57ac19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b2400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57ac1094e0 0x7f57ac1a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b2400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac10f460 con 0x7f57ac074240 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b2400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac10f2e0 con 0x7f57ac075820 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b2400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f57ac10f5e0 con 0x7f57ac1094e0 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b1000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57ac074240 0x7f57ac199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57abe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 0x7f57ac19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b1000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57ac074240 0x7f57ac199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44268/0 (socket says 172.21.15.7:44268) 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.326+0000 7f57b1000640 1 -- 172.21.15.7:0/957623567 learned_addr learned my addr 172.21.15.7:0/957623567 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57b1a00640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57ac1094e0 0x7f57ac1a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57b1000640 1 -- 172.21.15.7:0/957623567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57ac1094e0 msgr2=0x7f57ac1a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57b1000640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57ac1094e0 0x7f57ac1a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57b1000640 1 -- 172.21.15.7:0/957623567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 msgr2=0x7f57ac19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57b1000640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 0x7f57ac19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57b1000640 1 -- 172.21.15.7:0/957623567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ac1a16b0 con 0x7f57ac074240 2024-03-18T01:35:00.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57abe00640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 0x7f57ac19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:00.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57b1000640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57ac074240 0x7f57ac199b30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009230 tx=0x7f57a0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:00.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57a9600640 1 -- 172.21.15.7:0/957623567 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0041070 con 0x7f57ac074240 2024-03-18T01:35:00.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57a9600640 1 -- 172.21.15.7:0/957623567 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57a00044d0 con 0x7f57ac074240 2024-03-18T01:35:00.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.327+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ac1a1940 con 0x7f57ac074240 2024-03-18T01:35:00.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.328+0000 7f57a9600640 1 -- 172.21.15.7:0/957623567 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a0034440 con 0x7f57ac074240 2024-03-18T01:35:00.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.328+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ac1a1e30 con 0x7f57ac074240 2024-03-18T01:35:00.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.329+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57ac108210 con 0x7f57ac074240 2024-03-18T01:35:00.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.331+0000 7f57a9600640 1 -- 172.21.15.7:0/957623567 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f57a0044070 con 0x7f57ac074240 2024-03-18T01:35:00.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.332+0000 7f57a9600640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57940a27f0 0x7f57940a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:00.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.333+0000 7f57a9600640 1 -- 172.21.15.7:0/957623567 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f57a00f0c60 con 0x7f57ac074240 2024-03-18T01:35:00.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.333+0000 7f57abe00640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57940a27f0 0x7f57940a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:00.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.333+0000 7f57abe00640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57940a27f0 0x7f57940a4cb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f57ac19afa0 tx=0x7f5798005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:00.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.340+0000 7f57a9600640 1 -- 172.21.15.7:0/957623567 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f57a00bdae0 con 0x7f57ac074240 2024-03-18T01:35:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:00 smithi203 ceph-mon[36345]: pgmap v317: 97 pgs: 59 active+clean, 23 creating+peering, 15 unknown; 578 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-18T01:35:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:00 smithi203 ceph-mon[36345]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T01:35:00.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:00 smithi046 ceph-mon[36108]: pgmap v317: 97 pgs: 59 active+clean, 23 creating+peering, 15 unknown; 578 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-18T01:35:00.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:00 smithi046 ceph-mon[36108]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T01:35:00.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:00.533+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f57ac02d1b0 con 0x7f57ac074240 2024-03-18T01:35:00.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:00 smithi007 ceph-mon[31996]: pgmap v317: 97 pgs: 59 active+clean, 23 creating+peering, 15 unknown; 578 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-18T01:35:00.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:00 smithi007 ceph-mon[31996]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T01:35:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.114+0000 7f57a9600640 1 -- 172.21.15.7:0/957623567 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f57a00bd3e0 con 0x7f57ac074240 2024-03-18T01:35:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:set pool 4 allow_ec_overwrites to true 2024-03-18T01:35:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.119+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57940a27f0 msgr2=0x7f57940a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.119+0000 7f57b2400640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57940a27f0 0x7f57940a4cb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f57ac19afa0 tx=0x7f5798005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.119+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57ac074240 msgr2=0x7f57ac199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:01.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.119+0000 7f57b2400640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57ac074240 0x7f57ac199b30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009230 tx=0x7f57a0005d50 comp rx=0 tx=0).stop 2024-03-18T01:35:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.120+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 shutdown_connections 2024-03-18T01:35:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.120+0000 7f57b2400640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57ac1094e0 0x7f57ac1a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.120+0000 7f57b2400640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f57ac075820 0x7f57ac19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.120+0000 7f57b2400640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57940a27f0 0x7f57940a4cb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.120+0000 7f57b2400640 1 --2- 172.21.15.7:0/957623567 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f57ac074240 0x7f57ac199b30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.120+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 >> 172.21.15.7:0/957623567 conn(0x7f57ac0fcb90 msgr2=0x7f57ac1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.121+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 shutdown_connections 2024-03-18T01:35:01.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.121+0000 7f57b2400640 1 -- 172.21.15.7:0/957623567 wait complete. 2024-03-18T01:35:01.159 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.337+0000 7fe718600640 1 -- 172.21.15.7:0/731559771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe710109fe0 msgr2=0x7fe71010a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.337+0000 7fe718600640 1 --2- 172.21.15.7:0/731559771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe710109fe0 0x7fe71010a400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe704009900 tx=0x7fe70402bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.337+0000 7fe718600640 1 -- 172.21.15.7:0/731559771 shutdown_connections 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.337+0000 7fe718600640 1 --2- 172.21.15.7:0/731559771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71010bb90 0x7fe710112420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.337+0000 7fe718600640 1 --2- 172.21.15.7:0/731559771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe71010b1d0 0x7fe71010b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.337+0000 7fe718600640 1 --2- 172.21.15.7:0/731559771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe710109fe0 0x7fe71010a400 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.337+0000 7fe718600640 1 -- 172.21.15.7:0/731559771 >> 172.21.15.7:0/731559771 conn(0x7fe710070740 msgr2=0x7fe710072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.338+0000 7fe718600640 1 -- 172.21.15.7:0/731559771 shutdown_connections 2024-03-18T01:35:01.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.338+0000 7fe718600640 1 -- 172.21.15.7:0/731559771 wait complete. 2024-03-18T01:35:01.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe718600640 1 Processor -- start 2024-03-18T01:35:01.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe718600640 1 -- start start 2024-03-18T01:35:01.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe718600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe710109fe0 0x7fe7101a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe718600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe71010b1d0 0x7fe7101a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe718600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe71010bb90 0x7fe7101a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe718600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe710115cc0 con 0x7fe71010b1d0 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe718600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe710115b40 con 0x7fe71010bb90 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe718600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe710115e40 con 0x7fe710109fe0 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe715200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe71010b1d0 0x7fe7101a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.339+0000 7fe715c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe710109fe0 0x7fe7101a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe715c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe710109fe0 0x7fe7101a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:37526/0 (socket says 172.21.15.7:37526) 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe715c00640 1 -- 172.21.15.7:0/1392665774 learned_addr learned my addr 172.21.15.7:0/1392665774 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe715200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe71010b1d0 0x7fe7101a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44288/0 (socket says 172.21.15.7:44288) 2024-03-18T01:35:01.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe716600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe71010bb90 0x7fe7101a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe716600640 1 -- 172.21.15.7:0/1392665774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe710109fe0 msgr2=0x7fe7101a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe716600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe710109fe0 0x7fe7101a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe716600640 1 -- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe71010b1d0 msgr2=0x7fe7101a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe716600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe71010b1d0 0x7fe7101a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe716600640 1 -- 172.21.15.7:0/1392665774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7101a7f50 con 0x7fe71010bb90 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe715c00640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe710109fe0 0x7fe7101a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe715200640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe71010b1d0 0x7fe7101a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe716600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe71010bb90 0x7fe7101a78e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe70800da10 tx=0x7fe70800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe70e000640 1 -- 172.21.15.7:0/1392665774 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70800bd70 con 0x7fe71010bb90 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe70e000640 1 -- 172.21.15.7:0/1392665774 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe708003e50 con 0x7fe71010bb90 2024-03-18T01:35:01.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.340+0000 7fe70e000640 1 -- 172.21.15.7:0/1392665774 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7080106a0 con 0x7fe71010bb90 2024-03-18T01:35:01.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.341+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7101a8240 con 0x7fe71010bb90 2024-03-18T01:35:01.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.341+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7101a8740 con 0x7fe71010bb90 2024-03-18T01:35:01.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.342+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe710111300 con 0x7fe71010bb90 2024-03-18T01:35:01.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.344+0000 7fe70e000640 1 -- 172.21.15.7:0/1392665774 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fe708003970 con 0x7fe71010bb90 2024-03-18T01:35:01.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.345+0000 7fe70e000640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6f80a2880 0x7fe6f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:01.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.345+0000 7fe70e000640 1 -- 172.21.15.7:0/1392665774 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fe7080cefe0 con 0x7fe71010bb90 2024-03-18T01:35:01.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.345+0000 7fe715c00640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6f80a2880 0x7fe6f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:01.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.346+0000 7fe715c00640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6f80a2880 0x7fe6f80a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe704006010 tx=0x7fe70402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:01.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.349+0000 7fe70e000640 1 -- 172.21.15.7:0/1392665774 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe70809be60 con 0x7fe71010bb90 2024-03-18T01:35:01.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:01 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T01:35:01.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:01 smithi203 ceph-mon[36345]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T01:35:01.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:01 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/957623567' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T01:35:01.439 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:01 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T01:35:01.439 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:01 smithi046 ceph-mon[36108]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T01:35:01.439 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:01 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/957623567' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T01:35:01.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:01 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T01:35:01.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:01 smithi007 ceph-mon[31996]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T01:35:01.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:01 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/957623567' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T01:35:01.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:01.585+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fe71002d1b0 con 0x7fe71010bb90 2024-03-18T01:35:01.768 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:35:01 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: 2024-03-18T01:35:01.437+0000 7f8f22afa700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T01:35:02.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.130+0000 7fe70e000640 1 -- 172.21.15.7:0/1392665774 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fe70809b760 con 0x7fe71010bb90 2024-03-18T01:35:02.131 INFO:teuthology.orchestra.run.smithi007.stderr:added data pool 4 to fsmap 2024-03-18T01:35:02.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6f80a2880 msgr2=0x7fe6f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6f80a2880 0x7fe6f80a4d40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe704006010 tx=0x7fe70402df90 comp rx=0 tx=0).stop 2024-03-18T01:35:02.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe71010bb90 msgr2=0x7fe7101a78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe71010bb90 0x7fe7101a78e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe70800da10 tx=0x7fe70800ded0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 shutdown_connections 2024-03-18T01:35:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe71010bb90 0x7fe7101a78e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6f80a2880 0x7fe6f80a4d40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe71010b1d0 0x7fe7101a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 --2- 172.21.15.7:0/1392665774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe710109fe0 0x7fe7101a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 >> 172.21.15.7:0/1392665774 conn(0x7fe710070740 msgr2=0x7fe71010e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:02.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 shutdown_connections 2024-03-18T01:35:02.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.134+0000 7fe718600640 1 -- 172.21.15.7:0/1392665774 wait complete. 2024-03-18T01:35:02.156 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:35:02.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 -- 172.21.15.7:0/1404783397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 msgr2=0x7f8088074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 --2- 172.21.15.7:0/1404783397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 0x7f8088074640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8074009900 tx=0x7f807402bcc0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 -- 172.21.15.7:0/1404783397 shutdown_connections 2024-03-18T01:35:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 --2- 172.21.15.7:0/1404783397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8088109470 0x7f808810b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 --2- 172.21.15.7:0/1404783397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 0x7f8088074640 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 --2- 172.21.15.7:0/1404783397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80880757c0 0x7f8088073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 -- 172.21.15.7:0/1404783397 >> 172.21.15.7:0/1404783397 conn(0x7f80880fcb70 msgr2=0x7f80880fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 -- 172.21.15.7:0/1404783397 shutdown_connections 2024-03-18T01:35:02.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.378+0000 7f808e800640 1 -- 172.21.15.7:0/1404783397 wait complete. 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808e800640 1 Processor -- start 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808e800640 1 -- start start 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 0x7f80881999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80880757c0 0x7f8088199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808e800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8088109470 0x7f80881a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808e800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f808810f490 con 0x7f8088109470 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808e800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f808810f310 con 0x7f80880741e0 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808e800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f808810f610 con 0x7f80880757c0 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 0x7f80881999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:02.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 0x7f80881999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50094/0 (socket says 172.21.15.7:50094) 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.380+0000 7f808d400640 1 -- 172.21.15.7:0/32600994 learned_addr learned my addr 172.21.15.7:0/32600994 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808de00640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8088109470 0x7f80881a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808ca00640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80880757c0 0x7f8088199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808d400640 1 -- 172.21.15.7:0/32600994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80880757c0 msgr2=0x7f8088199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808d400640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80880757c0 0x7f8088199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808d400640 1 -- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8088109470 msgr2=0x7f80881a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808d400640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8088109470 0x7f80881a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808d400640 1 -- 172.21.15.7:0/32600994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80881a1600 con 0x7f80880741e0 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808de00640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8088109470 0x7f80881a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:02.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808d400640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 0x7f80881999e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f807c00d950 tx=0x7f807c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:02.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f8086000640 1 -- 172.21.15.7:0/32600994 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c017070 con 0x7f80880741e0 2024-03-18T01:35:02.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f8086000640 1 -- 172.21.15.7:0/32600994 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f807c009c30 con 0x7f80880741e0 2024-03-18T01:35:02.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f8086000640 1 -- 172.21.15.7:0/32600994 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807c00f6b0 con 0x7f80880741e0 2024-03-18T01:35:02.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.381+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80881a18f0 con 0x7f80880741e0 2024-03-18T01:35:02.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.382+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80881a1dc0 con 0x7f80880741e0 2024-03-18T01:35:02.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.383+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8088108230 con 0x7f80880741e0 2024-03-18T01:35:02.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.384+0000 7f8086000640 1 -- 172.21.15.7:0/32600994 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f807c0109d0 con 0x7f80880741e0 2024-03-18T01:35:02.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.386+0000 7f8086000640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f80700a2920 0x7f80700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:02.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.386+0000 7f8086000640 1 -- 172.21.15.7:0/32600994 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f807c0ce9a0 con 0x7f80880741e0 2024-03-18T01:35:02.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.386+0000 7f808ca00640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f80700a2920 0x7f80700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:02.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.387+0000 7f808ca00640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f80700a2920 0x7f80700a4de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f807402c2d0 tx=0x7f8074005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:02.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.390+0000 7f8086000640 1 -- 172.21.15.7:0/32600994 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f807c009de0 con 0x7f80880741e0 2024-03-18T01:35:02.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:02 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/957623567' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T01:35:02.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:02 smithi203 ceph-mon[36345]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T01:35:02.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:02 smithi203 ceph-mon[36345]: pgmap v320: 129 pgs: 89 active+clean, 24 creating+peering, 16 unknown; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-18T01:35:02.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:02 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1392665774' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T01:35:02.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:02 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T01:35:02.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:02 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/957623567' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T01:35:02.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:02 smithi046 ceph-mon[36108]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T01:35:02.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:02 smithi046 ceph-mon[36108]: pgmap v320: 129 pgs: 89 active+clean, 24 creating+peering, 16 unknown; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-18T01:35:02.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:02 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1392665774' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T01:35:02.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:02 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T01:35:02.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:02 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/957623567' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T01:35:02.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:02 smithi007 ceph-mon[31996]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T01:35:02.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:02 smithi007 ceph-mon[31996]: pgmap v320: 129 pgs: 89 active+clean, 24 creating+peering, 16 unknown; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-18T01:35:02.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:02 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1392665774' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T01:35:02.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:02 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T01:35:02.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.634+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f808802d1b0 con 0x7f80880741e0 2024-03-18T01:35:02.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.635+0000 7f8086000640 1 -- 172.21.15.7:0/32600994 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11782 (secure 0 0 0) 0x7f807c09b7a0 con 0x7f80880741e0 2024-03-18T01:35:02.636 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:02.636 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:35:01.588308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24596,"qdb_cluster":[24596]},"id":1}]} 2024-03-18T01:35:02.636 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 17 2024-03-18T01:35:02.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f80700a2920 msgr2=0x7f80700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f80700a2920 0x7f80700a4de0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f807402c2d0 tx=0x7f8074005fd0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 msgr2=0x7f80881999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 0x7f80881999e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f807c00d950 tx=0x7f807c00de10 comp rx=0 tx=0).stop 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 shutdown_connections 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f80700a2920 0x7f80700a4de0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8088109470 0x7f80881a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80880757c0 0x7f8088199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 --2- 172.21.15.7:0/32600994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80880741e0 0x7f80881999e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 >> 172.21.15.7:0/32600994 conn(0x7f80880fcb70 msgr2=0x7f8088105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.638+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 shutdown_connections 2024-03-18T01:35:02.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.639+0000 7f808e800640 1 -- 172.21.15.7:0/32600994 wait complete. 2024-03-18T01:35:02.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:35:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.885+0000 7f1d42600640 1 -- 172.21.15.7:0/3704221958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c104a00 msgr2=0x7f1d3c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.885+0000 7f1d42600640 1 --2- 172.21.15.7:0/3704221958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c104a00 0x7f1d3c104e80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009900 tx=0x7f1d2802bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.885+0000 7f1d42600640 1 -- 172.21.15.7:0/3704221958 shutdown_connections 2024-03-18T01:35:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.885+0000 7f1d42600640 1 --2- 172.21.15.7:0/3704221958 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d3c1053c0 0x7f1d3c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.885+0000 7f1d42600640 1 --2- 172.21.15.7:0/3704221958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c104a00 0x7f1d3c104e80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.885+0000 7f1d42600640 1 --2- 172.21.15.7:0/3704221958 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d3c103810 0x7f1d3c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.885+0000 7f1d42600640 1 -- 172.21.15.7:0/3704221958 >> 172.21.15.7:0/3704221958 conn(0x7f1d3c0fed80 msgr2=0x7f1d3c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:02.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.886+0000 7f1d42600640 1 -- 172.21.15.7:0/3704221958 shutdown_connections 2024-03-18T01:35:02.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.886+0000 7f1d42600640 1 -- 172.21.15.7:0/3704221958 wait complete. 2024-03-18T01:35:02.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.886+0000 7f1d42600640 1 Processor -- start 2024-03-18T01:35:02.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d42600640 1 -- start start 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d42600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c103810 0x7f1d3c199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d42600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d3c104a00 0x7f1d3c199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d42600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d3c1053c0 0x7f1d3c1a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d42600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c10f4e0 con 0x7f1d3c1053c0 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d42600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c10f360 con 0x7f1d3c103810 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d42600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c10f660 con 0x7f1d3c104a00 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d41200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c103810 0x7f1d3c199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d41200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c103810 0x7f1d3c199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50114/0 (socket says 172.21.15.7:50114) 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d41c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d3c1053c0 0x7f1d3c1a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d41200640 1 -- 172.21.15.7:0/3698883839 learned_addr learned my addr 172.21.15.7:0/3698883839 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.887+0000 7f1d3be00640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d3c104a00 0x7f1d3c199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:02.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d41200640 1 -- 172.21.15.7:0/3698883839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d3c104a00 msgr2=0x7f1d3c199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d41200640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d3c104a00 0x7f1d3c199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d41200640 1 -- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d3c1053c0 msgr2=0x7f1d3c1a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d41200640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d3c1053c0 0x7f1d3c1a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d41200640 1 -- 172.21.15.7:0/3698883839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c1a1650 con 0x7f1d3c103810 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d41c00640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d3c1053c0 0x7f1d3c1a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d41200640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c103810 0x7f1d3c199a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000e9c0 tx=0x7f1d3000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d39600640 1 -- 172.21.15.7:0/3698883839 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3000cd80 con 0x7f1d3c103810 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d39600640 1 -- 172.21.15.7:0/3698883839 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d300035f0 con 0x7f1d3c103810 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d39600640 1 -- 172.21.15.7:0/3698883839 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d30010690 con 0x7f1d3c103810 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d3c1a1940 con 0x7f1d3c103810 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.888+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d3c1a1e60 con 0x7f1d3c103810 2024-03-18T01:35:02.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.889+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d3c10aad0 con 0x7f1d3c103810 2024-03-18T01:35:02.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.891+0000 7f1d39600640 1 -- 172.21.15.7:0/3698883839 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f1d300037a0 con 0x7f1d3c103810 2024-03-18T01:35:02.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.892+0000 7f1d39600640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d240a2920 0x7f1d240a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:02.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.893+0000 7f1d39600640 1 -- 172.21.15.7:0/3698883839 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1d300cefd0 con 0x7f1d3c103810 2024-03-18T01:35:02.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.893+0000 7f1d3be00640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d240a2920 0x7f1d240a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:02.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.894+0000 7f1d3be00640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d240a2920 0x7f1d240a4de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1d2802c240 tx=0x7f1d2802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:02.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:02.896+0000 7f1d39600640 1 -- 172.21.15.7:0/3698883839 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1d3009bdd0 con 0x7f1d3c103810 2024-03-18T01:35:03.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.102+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d3c062480 con 0x7f1d3c103810 2024-03-18T01:35:03.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.104+0000 7f1d39600640 1 -- 172.21.15.7:0/3698883839 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7f1d300860e0 con 0x7f1d3c103810 2024-03-18T01:35:03.105 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:03.105 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:03.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d240a2920 msgr2=0x7f1d240a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d240a2920 0x7f1d240a4de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1d2802c240 tx=0x7f1d2802df90 comp rx=0 tx=0).stop 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c103810 msgr2=0x7f1d3c199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c103810 0x7f1d3c199a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000e9c0 tx=0x7f1d3000ee80 comp rx=0 tx=0).stop 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 shutdown_connections 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d240a2920 0x7f1d240a4de0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1d3c1053c0 0x7f1d3c1a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d3c104a00 0x7f1d3c199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 --2- 172.21.15.7:0/3698883839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d3c103810 0x7f1d3c199a30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.107+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 >> 172.21.15.7:0/3698883839 conn(0x7f1d3c0fed80 msgr2=0x7f1d3c107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.108+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 shutdown_connections 2024-03-18T01:35:03.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.108+0000 7f1d42600640 1 -- 172.21.15.7:0/3698883839 wait complete. 2024-03-18T01:35:03.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:35:03.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 -- 172.21.15.7:0/3378531905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 msgr2=0x7f1574109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 --2- 172.21.15.7:0/3378531905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 0x7f1574109410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f155c009900 tx=0x7f155c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 -- 172.21.15.7:0/3378531905 shutdown_connections 2024-03-18T01:35:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 --2- 172.21.15.7:0/3378531905 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1574109950 0x7f1574110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 --2- 172.21.15.7:0/3378531905 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 0x7f1574109410 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 --2- 172.21.15.7:0/3378531905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1574107e40 0x7f1574108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 -- 172.21.15.7:0/3378531905 >> 172.21.15.7:0/3378531905 conn(0x7f1574076dd0 msgr2=0x7f1574079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 -- 172.21.15.7:0/3378531905 shutdown_connections 2024-03-18T01:35:03.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.342+0000 7f157b000640 1 -- 172.21.15.7:0/3378531905 wait complete. 2024-03-18T01:35:03.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.343+0000 7f157b000640 1 Processor -- start 2024-03-18T01:35:03.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.343+0000 7f157b000640 1 -- start start 2024-03-18T01:35:03.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f157b000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1574107e40 0x7f157419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:03.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f157b000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 0x7f157419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f157b000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1574109950 0x7f15741a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f157b000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1574113b30 con 0x7f1574108f90 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f157b000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f15741139b0 con 0x7f1574109950 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f157b000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1574113cb0 con 0x7f1574107e40 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1579000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1574109950 0x7f15741a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1579000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1574109950 0x7f15741a56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50124/0 (socket says 172.21.15.7:50124) 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1579000640 1 -- 172.21.15.7:0/193472088 learned_addr learned my addr 172.21.15.7:0/193472088 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1573e00640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1574107e40 0x7f157419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f156be00640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 0x7f157419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1579000640 1 -- 172.21.15.7:0/193472088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1574107e40 msgr2=0x7f157419e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1579000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1574107e40 0x7f157419e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1579000640 1 -- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 msgr2=0x7f157419e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1579000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 0x7f157419e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f1579000640 1 -- 172.21.15.7:0/193472088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15741a5d60 con 0x7f1574109950 2024-03-18T01:35:03.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.344+0000 7f156be00640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 0x7f157419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:03.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.345+0000 7f1579000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1574109950 0x7f15741a56f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f156c00efd0 tx=0x7f156c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:03.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.345+0000 7f1571600640 1 -- 172.21.15.7:0/193472088 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c003b70 con 0x7f1574109950 2024-03-18T01:35:03.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.345+0000 7f1571600640 1 -- 172.21.15.7:0/193472088 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f156c003d10 con 0x7f1574109950 2024-03-18T01:35:03.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.345+0000 7f1571600640 1 -- 172.21.15.7:0/193472088 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156c0109f0 con 0x7f1574109950 2024-03-18T01:35:03.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.345+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15741a5ff0 con 0x7f1574109950 2024-03-18T01:35:03.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.345+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15741a64c0 con 0x7f1574109950 2024-03-18T01:35:03.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.346+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f157410f0f0 con 0x7f1574109950 2024-03-18T01:35:03.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.348+0000 7f1571600640 1 -- 172.21.15.7:0/193472088 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f156c004020 con 0x7f1574109950 2024-03-18T01:35:03.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.349+0000 7f1571600640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f15580a2800 0x7f15580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:03.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.349+0000 7f1571600640 1 -- 172.21.15.7:0/193472088 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f156c0cf1b0 con 0x7f1574109950 2024-03-18T01:35:03.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.349+0000 7f1573e00640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f15580a2800 0x7f15580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:03.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.350+0000 7f1573e00640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f15580a2800 0x7f15580a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1564009bd0 tx=0x7f1564005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:03.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.353+0000 7f1571600640 1 -- 172.21.15.7:0/193472088 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f156c09bfb0 con 0x7f1574109950 2024-03-18T01:35:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:03 smithi203 ceph-mon[36345]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T01:35:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:03 smithi203 ceph-mon[36345]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T01:35:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:03 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T01:35:03.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:03 smithi203 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:03.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:03 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/32600994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:03.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:03 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3698883839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:03.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:03 smithi046 ceph-mon[36108]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T01:35:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:03 smithi046 ceph-mon[36108]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T01:35:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:03 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T01:35:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:03 smithi046 ceph-mon[36108]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:03 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/32600994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:03 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3698883839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:03.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:03 smithi007 ceph-mon[31996]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T01:35:03.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:03 smithi007 ceph-mon[31996]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T01:35:03.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:03 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T01:35:03.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:03 smithi007 ceph-mon[31996]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:03.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:03 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/32600994' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:03.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:03 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3698883839' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:03.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.557+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15740623e0 con 0x7f1574109950 2024-03-18T01:35:03.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.559+0000 7f1571600640 1 -- 172.21.15.7:0/193472088 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7f156c09b8b0 con 0x7f1574109950 2024-03-18T01:35:03.560 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:03.560 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:03.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f15580a2800 msgr2=0x7f15580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f15580a2800 0x7f15580a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1564009bd0 tx=0x7f1564005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:03.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1574109950 msgr2=0x7f15741a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1574109950 0x7f15741a56f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f156c00efd0 tx=0x7f156c00c6a0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 shutdown_connections 2024-03-18T01:35:03.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1574109950 0x7f15741a56f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f15580a2800 0x7f15580a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1574108f90 0x7f157419e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 --2- 172.21.15.7:0/193472088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1574107e40 0x7f157419e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 >> 172.21.15.7:0/193472088 conn(0x7f1574076dd0 msgr2=0x7f157410c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:03.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 shutdown_connections 2024-03-18T01:35:03.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.563+0000 7f157b000640 1 -- 172.21.15.7:0/193472088 wait complete. 2024-03-18T01:35:03.598 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:35:03.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 -- 172.21.15.7:0/1455799359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 msgr2=0x7fab34102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 --2- 172.21.15.7:0/1455799359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 0x7fab34102c90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fab20009900 tx=0x7fab2002bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 -- 172.21.15.7:0/1455799359 shutdown_connections 2024-03-18T01:35:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 --2- 172.21.15.7:0/1455799359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab341031d0 0x7fab34109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 --2- 172.21.15.7:0/1455799359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 0x7fab34102c90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 --2- 172.21.15.7:0/1455799359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab34101620 0x7fab34101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 -- 172.21.15.7:0/1455799359 >> 172.21.15.7:0/1455799359 conn(0x7fab340fcb90 msgr2=0x7fab340fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 -- 172.21.15.7:0/1455799359 shutdown_connections 2024-03-18T01:35:03.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.818+0000 7fab39400640 1 -- 172.21.15.7:0/1455799359 wait complete. 2024-03-18T01:35:03.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.819+0000 7fab39400640 1 Processor -- start 2024-03-18T01:35:03.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.819+0000 7fab39400640 1 -- start start 2024-03-18T01:35:03.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab39400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab34101620 0x7fab34197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:03.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab39400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 0x7fab34197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab39400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab341031d0 0x7fab3419edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab39400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fab3410d2b0 con 0x7fab34101620 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab39400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fab3410d130 con 0x7fab34102810 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab39400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab3410d430 con 0x7fab341031d0 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab34101620 0x7fab34197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 0x7fab34197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 0x7fab34197d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50144/0 (socket says 172.21.15.7:50144) 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab34101620 0x7fab34197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44372/0 (socket says 172.21.15.7:44372) 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33400640 1 -- 172.21.15.7:0/2523048285 learned_addr learned my addr 172.21.15.7:0/2523048285 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab38a00640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab341031d0 0x7fab3419edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33400640 1 -- 172.21.15.7:0/2523048285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab341031d0 msgr2=0x7fab3419edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab341031d0 0x7fab3419edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33400640 1 -- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab34101620 msgr2=0x7fab34197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab34101620 0x7fab34197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33400640 1 -- 172.21.15.7:0/2523048285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab3419f420 con 0x7fab34102810 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.820+0000 7fab33e00640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab34101620 0x7fab34197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.821+0000 7fab33400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 0x7fab34197d40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fab20009230 tx=0x7fab2002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.821+0000 7fab30c00640 1 -- 172.21.15.7:0/2523048285 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20041070 con 0x7fab34102810 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.821+0000 7fab30c00640 1 -- 172.21.15.7:0/2523048285 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab20004450 con 0x7fab34102810 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.821+0000 7fab30c00640 1 -- 172.21.15.7:0/2523048285 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab20003e20 con 0x7fab34102810 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.821+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab3419f6b0 con 0x7fab34102810 2024-03-18T01:35:03.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.821+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab3419fbf0 con 0x7fab34102810 2024-03-18T01:35:03.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.822+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab34108870 con 0x7fab34102810 2024-03-18T01:35:03.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.824+0000 7fab30c00640 1 -- 172.21.15.7:0/2523048285 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fab20044070 con 0x7fab34102810 2024-03-18T01:35:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.825+0000 7fab30c00640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fab180a2920 0x7fab180a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.825+0000 7fab30c00640 1 -- 172.21.15.7:0/2523048285 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fab200f0800 con 0x7fab34102810 2024-03-18T01:35:03.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.825+0000 7fab33e00640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fab180a2920 0x7fab180a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:03.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.826+0000 7fab33e00640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fab180a2920 0x7fab180a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab34102670 tx=0x7fab24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:03.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:03.829+0000 7fab30c00640 1 -- 172.21.15.7:0/2523048285 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab200bcfb0 con 0x7fab34102810 2024-03-18T01:35:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.073+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab3402d1b0 con 0x7fab34102810 2024-03-18T01:35:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.074+0000 7fab30c00640 1 -- 172.21.15.7:0/2523048285 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11782 (secure 0 0 0) 0x7fab200c1050 con 0x7fab34102810 2024-03-18T01:35:04.074 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:04.074 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24448,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24475,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24512,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24542,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24544,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24562,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:35:01.588308+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24596},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24596,"qdb_cluster":[24596]},"id":1}]} 2024-03-18T01:35:04.075 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 17 2024-03-18T01:35:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fab180a2920 msgr2=0x7fab180a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fab180a2920 0x7fab180a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab34102670 tx=0x7fab24005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 msgr2=0x7fab34197d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 0x7fab34197d40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fab20009230 tx=0x7fab2002c650 comp rx=0 tx=0).stop 2024-03-18T01:35:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 shutdown_connections 2024-03-18T01:35:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab341031d0 0x7fab3419edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab34102810 0x7fab34197d40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fab180a2920 0x7fab180a4de0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 --2- 172.21.15.7:0/2523048285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fab34101620 0x7fab34197800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 >> 172.21.15.7:0/2523048285 conn(0x7fab340fcb90 msgr2=0x7fab34105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 shutdown_connections 2024-03-18T01:35:04.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.077+0000 7fab39400640 1 -- 172.21.15.7:0/2523048285 wait complete. 2024-03-18T01:35:04.096 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 -- 172.21.15.7:0/1385927096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1018e0 msgr2=0x7fce5c101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 --2- 172.21.15.7:0/1385927096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1018e0 0x7fce5c101d60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fce4c009900 tx=0x7fce4c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 -- 172.21.15.7:0/1385927096 shutdown_connections 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 --2- 172.21.15.7:0/1385927096 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce5c1022a0 0x7fce5c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 --2- 172.21.15.7:0/1385927096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1018e0 0x7fce5c101d60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 --2- 172.21.15.7:0/1385927096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1006f0 0x7fce5c100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 -- 172.21.15.7:0/1385927096 >> 172.21.15.7:0/1385927096 conn(0x7fce5c0fbc60 msgr2=0x7fce5c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 -- 172.21.15.7:0/1385927096 shutdown_connections 2024-03-18T01:35:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.314+0000 7fce64800640 1 -- 172.21.15.7:0/1385927096 wait complete. 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.315+0000 7fce64800640 1 Processor -- start 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.316+0000 7fce64800640 1 -- start start 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.316+0000 7fce64800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1006f0 0x7fce5c199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.316+0000 7fce64800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce5c1018e0 0x7fce5c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.316+0000 7fce64800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1022a0 0x7fce5c1a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.316+0000 7fce64800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c10c350 con 0x7fce5c1018e0 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.316+0000 7fce64800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c10c1d0 con 0x7fce5c1022a0 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.316+0000 7fce64800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c10c4d0 con 0x7fce5c1006f0 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce61400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce5c1018e0 0x7fce5c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce61e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1006f0 0x7fce5c199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce62800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1022a0 0x7fce5c1a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce61400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce5c1018e0 0x7fce5c19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44396/0 (socket says 172.21.15.7:44396) 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce61400640 1 -- 172.21.15.7:0/108038990 learned_addr learned my addr 172.21.15.7:0/108038990 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce62800640 1 -- 172.21.15.7:0/108038990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1006f0 msgr2=0x7fce5c199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce62800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1006f0 0x7fce5c199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce62800640 1 -- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce5c1018e0 msgr2=0x7fce5c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce62800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce5c1018e0 0x7fce5c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce62800640 1 -- 172.21.15.7:0/108038990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c1a1910 con 0x7fce5c1022a0 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce62800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1022a0 0x7fce5c1a1270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fce5800d680 tx=0x7fce5800db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce56a00640 1 -- 172.21.15.7:0/108038990 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58017070 con 0x7fce5c1022a0 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce56a00640 1 -- 172.21.15.7:0/108038990 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce58009820 con 0x7fce5c1022a0 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.317+0000 7fce56a00640 1 -- 172.21.15.7:0/108038990 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce58004e00 con 0x7fce5c1022a0 2024-03-18T01:35:04.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.318+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5c1a1c00 con 0x7fce5c1022a0 2024-03-18T01:35:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.318+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5c1a2100 con 0x7fce5c1022a0 2024-03-18T01:35:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.319+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce5c1a22d0 con 0x7fce5c1022a0 2024-03-18T01:35:04.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.320+0000 7fce56a00640 1 -- 172.21.15.7:0/108038990 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fce580099d0 con 0x7fce5c1022a0 2024-03-18T01:35:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.322+0000 7fce56a00640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fce440a2800 0x7fce440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.322+0000 7fce56a00640 1 -- 172.21.15.7:0/108038990 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fce580ce5b0 con 0x7fce5c1022a0 2024-03-18T01:35:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.322+0000 7fce61e00640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fce440a2800 0x7fce440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.323+0000 7fce61e00640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fce440a2800 0x7fce440a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fce5c19b1d0 tx=0x7fce50009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:04.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.331+0000 7fce56a00640 1 -- 172.21.15.7:0/108038990 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce5809ad60 con 0x7fce5c1022a0 2024-03-18T01:35:04.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:04 smithi203 ceph-mon[36345]: pgmap v322: 129 pgs: 103 active+clean, 16 creating+peering, 10 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-03-18T01:35:04.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:04 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/193472088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:04.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:04 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2523048285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:04.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:04 smithi046 ceph-mon[36108]: pgmap v322: 129 pgs: 103 active+clean, 16 creating+peering, 10 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-03-18T01:35:04.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:04 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/193472088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:04.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:04 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2523048285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.535+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce5c1a24a0 con 0x7fce5c1022a0 2024-03-18T01:35:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.537+0000 7fce56a00640 1 -- 172.21.15.7:0/108038990 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7fce5809ee00 con 0x7fce5c1022a0 2024-03-18T01:35:04.537 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:04.538 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.539+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fce440a2800 msgr2=0x7fce440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.539+0000 7fce64800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fce440a2800 0x7fce440a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fce5c19b1d0 tx=0x7fce50009670 comp rx=0 tx=0).stop 2024-03-18T01:35:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1022a0 msgr2=0x7fce5c1a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1022a0 0x7fce5c1a1270 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fce5800d680 tx=0x7fce5800db40 comp rx=0 tx=0).stop 2024-03-18T01:35:04.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 shutdown_connections 2024-03-18T01:35:04.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fce5c1022a0 0x7fce5c1a1270 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fce440a2800 0x7fce440a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fce5c1018e0 0x7fce5c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 --2- 172.21.15.7:0/108038990 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce5c1006f0 0x7fce5c199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 >> 172.21.15.7:0/108038990 conn(0x7fce5c0fbc60 msgr2=0x7fce5c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 shutdown_connections 2024-03-18T01:35:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.540+0000 7fce64800640 1 -- 172.21.15.7:0/108038990 wait complete. 2024-03-18T01:35:04.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:04 smithi007 ceph-mon[31996]: pgmap v322: 129 pgs: 103 active+clean, 16 creating+peering, 10 unknown; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-03-18T01:35:04.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:04 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/193472088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:04.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:04 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2523048285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:04.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T01:34:56.125856+0000', 'modified': '2024-03-18T01:35:01.588308+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24596}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24596': {'gid': 24596, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6839/1986652812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6838', 'nonce': 1986652812}, {'type': 'v1', 'addr': '172.21.15.46:6839', 'nonce': 1986652812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24596, 'qdb_cluster': [24596]} 2024-03-18T01:35:04.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T01:35:04.561 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T01:35:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 -- 172.21.15.7:0/1018374483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550101650 msgr2=0x7fa550101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 --2- 172.21.15.7:0/1018374483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550101650 0x7fa550101a70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa540009ca0 tx=0x7fa54002c100 comp rx=0 tx=0).stop 2024-03-18T01:35:04.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 -- 172.21.15.7:0/1018374483 shutdown_connections 2024-03-18T01:35:04.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 --2- 172.21.15.7:0/1018374483 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa550103200 0x7fa550109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 --2- 172.21.15.7:0/1018374483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa550102840 0x7fa550102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 --2- 172.21.15.7:0/1018374483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550101650 0x7fa550101a70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 -- 172.21.15.7:0/1018374483 >> 172.21.15.7:0/1018374483 conn(0x7fa5500fcbc0 msgr2=0x7fa5500ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 -- 172.21.15.7:0/1018374483 shutdown_connections 2024-03-18T01:35:04.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.789+0000 7fa54f200640 1 -- 172.21.15.7:0/1018374483 wait complete. 2024-03-18T01:35:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.790+0000 7fa54f200640 1 Processor -- start 2024-03-18T01:35:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54f200640 1 -- start start 2024-03-18T01:35:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54f200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa550101650 0x7fa550199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54f200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa550102840 0x7fa55019a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550103200 0x7fa5501a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54f200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010d2d0 con 0x7fa550101650 2024-03-18T01:35:04.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010d150 con 0x7fa550103200 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54f200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa55010d450 con 0x7fa550102840 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550103200 0x7fa5501a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550103200 0x7fa5501a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50176/0 (socket says 172.21.15.7:50176) 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54e800640 1 -- 172.21.15.7:0/2165384125 learned_addr learned my addr 172.21.15.7:0/2165384125 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa54de00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa550101650 0x7fa550199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.791+0000 7fa545400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa550102840 0x7fa55019a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa54e800640 1 -- 172.21.15.7:0/2165384125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa550102840 msgr2=0x7fa55019a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa54e800640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa550102840 0x7fa55019a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa54e800640 1 -- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa550101650 msgr2=0x7fa550199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa54e800640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa550101650 0x7fa550199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa54e800640 1 -- 172.21.15.7:0/2165384125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5501a17a0 con 0x7fa550103200 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa54e800640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550103200 0x7fa5501a1130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa54800d750 tx=0x7fa54800dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:04.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa547400640 1 -- 172.21.15.7:0/2165384125 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa548017070 con 0x7fa550103200 2024-03-18T01:35:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa547400640 1 -- 172.21.15.7:0/2165384125 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa548005500 con 0x7fa550103200 2024-03-18T01:35:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa54f200640 1 -- 172.21.15.7:0/2165384125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5501a1a90 con 0x7fa550103200 2024-03-18T01:35:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa547400640 1 -- 172.21.15.7:0/2165384125 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa548004d90 con 0x7fa550103200 2024-03-18T01:35:04.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.792+0000 7fa54f200640 1 -- 172.21.15.7:0/2165384125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5501a1fe0 con 0x7fa550103200 2024-03-18T01:35:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.797+0000 7fa557e33740 1 Processor -- start 2024-03-18T01:35:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.797+0000 7fa547400640 1 -- 172.21.15.7:0/2165384125 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fa548004f30 con 0x7fa550103200 2024-03-18T01:35:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.797+0000 7fa557e33740 1 -- start start 2024-03-18T01:35:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.797+0000 7fa557e33740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfcb340 0x56344bfcb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa557e33740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa557e33740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfd0ea0 0x56344bfd3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa557e33740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56344bf2fc50 con 0x56344bfcbd30 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa557e33740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56344bed89d0 con 0x56344bfd0ea0 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa557e33740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56344bf2bcf0 con 0x56344bfcb340 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa54de00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa545400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfd0ea0 0x56344bfd3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa545400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfd0ea0 0x56344bfd3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50184/0 (socket says 172.21.15.7:50184) 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa545400640 1 -- 172.21.15.7:0/640813983 learned_addr learned my addr 172.21.15.7:0/640813983 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa545400640 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfcb340 0x56344bfcb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa54de00640 1 -- 172.21.15.7:0/640813983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfcb340 msgr2=0x56344bfcb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa54de00640 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfcb340 0x56344bfcb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa54de00640 1 -- 172.21.15.7:0/640813983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfd0ea0 msgr2=0x56344bfd3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa54de00640 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfd0ea0 0x56344bfd3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa54de00640 1 -- 172.21.15.7:0/640813983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56344bef2f10 con 0x56344bfcbd30 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.798+0000 7fa545400640 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfd0ea0 0x56344bfd3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:04.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.799+0000 7fa54de00640 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd0830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa540037040 tx=0x7fa540005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ec67e8d1834df3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.799+0000 7fa53f400640 1 -- 172.21.15.7:0/640813983 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540043070 con 0x56344bfcbd30 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.799+0000 7fa53f400640 1 -- 172.21.15.7:0/640813983 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa540003970 con 0x56344bfcbd30 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.799+0000 7fa547400640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5340a2880 0x7fa5340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.799+0000 7fa53f400640 1 -- 172.21.15.7:0/640813983 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540039040 con 0x56344bfcbd30 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.799+0000 7fa547400640 1 -- 172.21.15.7:0/2165384125 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa5480ce540 con 0x7fa550103200 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.799+0000 7fa545400640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5340a2880 0x7fa5340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 -- 172.21.15.7:0/640813983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 msgr2=0x56344bfd0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd0830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa540037040 tx=0x7fa540005ef0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 -- 172.21.15.7:0/640813983 shutdown_connections 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfd0ea0 0x56344bfd3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd0830 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 --2- 172.21.15.7:0/640813983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfcb340 0x56344bfcb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 -- 172.21.15.7:0/640813983 >> 172.21.15.7:0/640813983 conn(0x56344bfc68b0 msgr2=0x56344bfc6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 -- 172.21.15.7:0/640813983 shutdown_connections 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa557e33740 1 -- 172.21.15.7:0/640813983 wait complete. 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.800+0000 7fa545400640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5340a2880 0x7fa5340a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x56344bfcc590 tx=0x7fa5380097d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 1 -- start start 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 1 Processor -- start 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcb340 0x56344bfd5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcbd30 0x56344bfd5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56344bfd3df0 con 0x56344bfcb340 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56344beebe20 con 0x56344bfcbd30 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56344bfd3f70 con 0x56344bfd0ea0 2024-03-18T01:35:04.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa557e33740 10 client.0 _ll_register_callbacks cb 0x56344beeb7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa54de00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcb340 0x56344bfd5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.801+0000 7fa545400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcbd30 0x56344bfd5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa54de00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcb340 0x56344bfd5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:44436/0 (socket says 172.21.15.7:44436) 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa54de00640 1 -- 172.21.15.7:0/193974474 learned_addr learned my addr 172.21.15.7:0/193974474 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa54e800640 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa545400640 1 -- 172.21.15.7:0/193974474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 msgr2=0x56344bfdc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa545400640 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa545400640 1 -- 172.21.15.7:0/193974474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcb340 msgr2=0x56344bfd5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa545400640 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcb340 0x56344bfd5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa545400640 1 -- 172.21.15.7:0/193974474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56344bc442d0 con 0x56344bfcbd30 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa54de00640 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcb340 0x56344bfd5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa545400640 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcbd30 0x56344bfd5950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa53800e780 tx=0x7fa5380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb68ee005e21db12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.802+0000 7fa533e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:35:04.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa533e00640 1 -- 172.21.15.7:0/193974474 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53800fb50 con 0x56344bfcbd30 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa533e00640 1 -- 172.21.15.7:0/193974474 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa53801a070 con 0x56344bfcbd30 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 1 -- 172.21.15.7:0/193974474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa540003970 con 0x56344bfcbd30 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa533e00640 1 -- 172.21.15.7:0/193974474 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fa53800fcf0 con 0x56344bfcbd30 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 10 client.24704 fetch_fsmap learned FSMap version 17 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 1 -- 172.21.15.7:0/193974474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56344bd0e6b0 con 0x56344bfcbd30 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa533e00640 1 -- 172.21.15.7:0/193974474 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa540003970 con 0x56344bfcbd30 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa533e00640 1 -- 172.21.15.7:0/193974474 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fa5380204f0 con 0x56344bfcbd30 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 10 client.24704 fetch_fsmap finished waiting for FSMap version 17 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 10 client.24704 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 20 client.24704 populate_metadata read hostname 'smithi007' 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x56344bfe21d0 0x56344bfe3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 4 client.24704 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.803+0000 7fa557e33740 1 -- 172.21.15.7:0/193974474 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56344bf8d2b0 con 0x56344bfe21d0 2024-03-18T01:35:04.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.804+0000 7fa54de00640 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x56344bfe21d0 0x56344bfe3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.804+0000 7fa54de00640 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x56344bfe21d0 0x56344bfe3640 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d552c65d057c9b2 server_cookie=d116b54a0f58ac5a in_seq=0 out_seq=0 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.804+0000 7fa533e00640 10 client.24704 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa533e00640 1 -- 172.21.15.7:0/193974474 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56344bf8d2b0 con 0x56344bfe21d0 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa533e00640 10 client.24704 handle_command_reply: tid=0 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 client.24704 shutdown 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 -- 172.21.15.7:0/193974474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcbd30 msgr2=0x56344bfd5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcbd30 0x56344bfd5950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa53800e780 tx=0x7fa5380046c0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 -- 172.21.15.7:0/193974474 shutdown_connections 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x56344bfe21d0 0x56344bfe3640 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcbd30 0x56344bfd5950 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa533e00640 0 client.24704 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:35:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 --2- 172.21.15.7:0/193974474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcb340 0x56344bfd5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 -- 172.21.15.7:0/193974474 >> 172.21.15.7:0/193974474 conn(0x56344bfc68b0 msgr2=0x56344bfc5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 -- 172.21.15.7:0/193974474 shutdown_connections 2024-03-18T01:35:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.805+0000 7fa557e33740 1 -- 172.21.15.7:0/193974474 wait complete. 2024-03-18T01:35:04.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.806+0000 7fa557e33740 20 client.24704 trim_cache size 0 max 16384 2024-03-18T01:35:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa557e33740 1 Processor -- start 2024-03-18T01:35:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa557e33740 1 -- start start 2024-03-18T01:35:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa557e33740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa557e33740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfdc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa557e33740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfd5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa557e33740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56344bfe1980 con 0x56344bfcbd30 2024-03-18T01:35:04.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa557e33740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56344bfe2050 con 0x56344bfcb340 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa557e33740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56344bfe2720 con 0x56344bfd0ea0 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa54de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.826+0000 7fa54de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50200/0 (socket says 172.21.15.7:50200) 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54de00640 1 -- 172.21.15.7:0/2890314535 learned_addr learned my addr 172.21.15.7:0/2890314535 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54e800640 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfd5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa545400640 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfdc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54de00640 1 -- 172.21.15.7:0/2890314535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 msgr2=0x56344bfd5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54de00640 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfd5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54de00640 1 -- 172.21.15.7:0/2890314535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 msgr2=0x56344bfdc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54de00640 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfdc950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54de00640 1 -- 172.21.15.7:0/2890314535 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56344bfcc110 con 0x56344bfcb340 2024-03-18T01:35:04.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54e800640 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfd5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa545400640 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfdc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:04.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa54de00640 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd5020 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa5400325f0 tx=0x7fa540009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e325100d9ff3d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.827+0000 7fa53ea00640 1 -- 172.21.15.7:0/2890314535 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540039040 con 0x56344bfcb340 2024-03-18T01:35:04.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa53ea00640 1 -- 172.21.15.7:0/2890314535 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa540009800 con 0x56344bfcb340 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa53ea00640 1 -- 172.21.15.7:0/2890314535 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540043040 con 0x56344bfcb340 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 -- 172.21.15.7:0/2890314535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 msgr2=0x56344bfd5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd5020 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa5400325f0 tx=0x7fa540009360 comp rx=0 tx=0).stop 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 -- 172.21.15.7:0/2890314535 shutdown_connections 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfd5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfdc950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 --2- 172.21.15.7:0/2890314535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd5020 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 -- 172.21.15.7:0/2890314535 >> 172.21.15.7:0/2890314535 conn(0x56344bfc68b0 msgr2=0x56344bfc1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 -- 172.21.15.7:0/2890314535 shutdown_connections 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.828+0000 7fa557e33740 1 -- 172.21.15.7:0/2890314535 wait complete. 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 1 -- start start 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 1 Processor -- start 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56344bfe34c0 con 0x56344bfcbd30 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x56344bfe2df0 con 0x56344bfcb340 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56344bfe3b90 con 0x56344bfd0ea0 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.829+0000 7fa557e33740 10 client.0 _ll_register_callbacks cb 0x7fa538011680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:35:04.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa545400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50204/0 (socket says 172.21.15.7:50204) 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 -- 172.21.15.7:0/1997225744 learned_addr learned my addr 172.21.15.7:0/1997225744 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54e800640 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 -- 172.21.15.7:0/1997225744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 msgr2=0x56344bfdb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 -- 172.21.15.7:0/1997225744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 msgr2=0x56344bfd8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 -- 172.21.15.7:0/1997225744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56344bfdba10 con 0x56344bfcb340 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54e800640 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa54de00640 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd7ec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa540005de0 tx=0x7fa54002c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=836f547f766dfdb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.830+0000 7fa533e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa533e00640 1 -- 172.21.15.7:0/1997225744 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540043030 con 0x56344bfcb340 2024-03-18T01:35:04.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa533e00640 1 -- 172.21.15.7:0/1997225744 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa540051dd0 con 0x56344bfcb340 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 1 -- 172.21.15.7:0/1997225744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa540009800 con 0x56344bfcb340 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa533e00640 1 -- 172.21.15.7:0/1997225744 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fa540039040 con 0x56344bfcb340 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 10 client.24710 fetch_fsmap learned FSMap version 17 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 1 -- 172.21.15.7:0/1997225744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56344c047f40 con 0x56344bfcb340 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa533e00640 1 -- 172.21.15.7:0/1997225744 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa540009800 con 0x56344bfcb340 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa533e00640 1 -- 172.21.15.7:0/1997225744 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fa540034440 con 0x56344bfcb340 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 10 client.24710 fetch_fsmap finished waiting for FSMap version 17 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 10 client.24710 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 20 client.24710 populate_metadata read hostname 'smithi007' 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x56344c04c320 0x56344c04b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 4 client.24710 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.831+0000 7fa557e33740 1 -- 172.21.15.7:0/1997225744 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "status"}) -- 0x56344bf3d190 con 0x56344c04c320 2024-03-18T01:35:04.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.832+0000 7fa545400640 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x56344c04c320 0x56344c04b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.832+0000 7fa545400640 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x56344c04c320 0x56344c04b030 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fcdc5d03db6ab1c server_cookie=5786d4c0cbd9a32b in_seq=0 out_seq=0 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.832+0000 7fa533e00640 10 client.24710 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.832+0000 7fa533e00640 1 -- 172.21.15.7:0/1997225744 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x56344bf3d190 con 0x56344c04c320 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.832+0000 7fa533e00640 10 client.24710 handle_command_reply: tid=0 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.832+0000 7fa557e33740 1 client.24710 shutdown 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.832+0000 7fa557e33740 1 -- 172.21.15.7:0/1997225744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 msgr2=0x56344bfd7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.832+0000 7fa557e33740 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd7ec0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa540005de0 tx=0x7fa54002c100 comp rx=0 tx=0).stop 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 1 -- 172.21.15.7:0/1997225744 shutdown_connections 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x56344c04c320 0x56344c04b030 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56344bfd0ea0 0x56344bfdb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x56344bfcbd30 0x56344bfd8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa533e00640 0 client.24710 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 1 --2- 172.21.15.7:0/1997225744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56344bfcb340 0x56344bfd7ec0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 1 -- 172.21.15.7:0/1997225744 >> 172.21.15.7:0/1997225744 conn(0x56344bfc68b0 msgr2=0x56344bfd5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 1 -- 172.21.15.7:0/1997225744 shutdown_connections 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 1 -- 172.21.15.7:0/1997225744 wait complete. 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa557e33740 20 client.24710 trim_cache size 0 max 16384 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24596, 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:35:04.839 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:35:04.840 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 7.7806564910000002, 2024-03-18T01:35:04.840 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 17, 2024-03-18T01:35:04.840 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 89, 2024-03-18T01:35:04.840 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:35:04.840 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 14.596879268 2024-03-18T01:35:04.840 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa54f200640 1 -- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5340a2880 msgr2=0x7fa5340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa54f200640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5340a2880 0x7fa5340a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x56344bfcc590 tx=0x7fa5380097d0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa54f200640 1 -- 172.21.15.7:0/2165384125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550103200 msgr2=0x7fa5501a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.833+0000 7fa54f200640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550103200 0x7fa5501a1130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa54800d750 tx=0x7fa54800dc10 comp rx=0 tx=0).stop 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.834+0000 7fa54f200640 1 -- 172.21.15.7:0/2165384125 shutdown_connections 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.834+0000 7fa54f200640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa550103200 0x7fa5501a1130 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.834+0000 7fa54f200640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa550102840 0x7fa55019a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.834+0000 7fa54f200640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5340a2880 0x7fa5340a4d40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.834+0000 7fa54f200640 1 --2- 172.21.15.7:0/2165384125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa550101650 0x7fa550199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:04.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.834+0000 7fa54f200640 1 -- 172.21.15.7:0/2165384125 >> 172.21.15.7:0/2165384125 conn(0x7fa5500fcbc0 msgr2=0x7fa550073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:04.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.834+0000 7fa54f200640 1 -- 172.21.15.7:0/2165384125 shutdown_connections 2024-03-18T01:35:04.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:04.834+0000 7fa54f200640 1 -- 172.21.15.7:0/2165384125 wait complete. 2024-03-18T01:35:04.851 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T01:35:04.851 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T01:35:04.852 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T01:35:05.295 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:05 smithi203 ceph-mon[36345]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T01:35:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:05 smithi203 ceph-mon[36345]: Cluster is now healthy 2024-03-18T01:35:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:05 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/108038990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:05.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:05 smithi007 ceph-mon[31996]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T01:35:05.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:05 smithi007 ceph-mon[31996]: Cluster is now healthy 2024-03-18T01:35:05.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:05 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/108038990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:05.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:05 smithi046 ceph-mon[36108]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T01:35:05.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:05 smithi046 ceph-mon[36108]: Cluster is now healthy 2024-03-18T01:35:05.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:05 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/108038990' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:06.493 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:06 smithi007 ceph-mon[31996]: pgmap v323: 129 pgs: 120 active+clean, 9 creating+peering; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-18T01:35:06.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:06 smithi046 ceph-mon[36108]: pgmap v323: 129 pgs: 120 active+clean, 9 creating+peering; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-18T01:35:06.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:06 smithi203 ceph-mon[36345]: pgmap v323: 129 pgs: 120 active+clean, 9 creating+peering; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-18T01:35:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.713+0000 7f983ee00700 1 -- 172.21.15.7:0/87099924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078430 msgr2=0x7f9838078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.713+0000 7f983ee00700 1 --2- 172.21.15.7:0/87099924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078430 0x7f9838078810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9828009990 tx=0x7f9828009ca0 comp rx=0 tx=0).stop 2024-03-18T01:35:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.713+0000 7f983ee00700 1 -- 172.21.15.7:0/87099924 shutdown_connections 2024-03-18T01:35:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.713+0000 7f983ee00700 1 --2- 172.21.15.7:0/87099924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9838113140 0x7f9838115530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.713+0000 7f983ee00700 1 --2- 172.21.15.7:0/87099924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078430 0x7f9838078810 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.713+0000 7f983ee00700 1 --2- 172.21.15.7:0/87099924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9838078de0 0x7f9838112c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.713+0000 7f983ee00700 1 -- 172.21.15.7:0/87099924 >> 172.21.15.7:0/87099924 conn(0x7f98380fff70 msgr2=0x7f9838102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:07.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.713+0000 7f983ee00700 1 -- 172.21.15.7:0/87099924 shutdown_connections 2024-03-18T01:35:07.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.714+0000 7f983ee00700 1 -- 172.21.15.7:0/87099924 wait complete. 2024-03-18T01:35:07.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.714+0000 7f983ee00700 1 Processor -- start 2024-03-18T01:35:07.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f983ee00700 1 -- start start 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f983ee00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9838078430 0x7f983819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f983ee00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078de0 0x7f983819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f983ee00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9838113140 0x7f98381a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f983ee00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9838118cf0 con 0x7f9838113140 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f983ee00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9838118b70 con 0x7f9838078de0 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f983ee00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9838118e70 con 0x7f9838078430 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f982fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078de0 0x7f983819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f982fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078de0 0x7f983819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:39132/0 (socket says 172.21.15.7:39132) 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f9837e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9838078430 0x7f983819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.715+0000 7f982fe00700 1 -- 172.21.15.7:0/1766853823 learned_addr learned my addr 172.21.15.7:0/1766853823 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f983ce00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9838113140 0x7f98381a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f982fe00700 1 -- 172.21.15.7:0/1766853823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9838078430 msgr2=0x7f983819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:07.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f982fe00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9838078430 0x7f983819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f982fe00700 1 -- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9838113140 msgr2=0x7f98381a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f982fe00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9838113140 0x7f98381a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f982fe00700 1 -- 172.21.15.7:0/1766853823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98381a41b0 con 0x7f9838078de0 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f9837e00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9838078430 0x7f983819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f982fe00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078de0 0x7f983819f6e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f982000e9a0 tx=0x7f982000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f9835600700 1 -- 172.21.15.7:0/1766853823 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982000ca10 con 0x7f9838078de0 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.716+0000 7f9835600700 1 -- 172.21.15.7:0/1766853823 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f982000cbb0 con 0x7f9838078de0 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.717+0000 7f9835600700 1 -- 172.21.15.7:0/1766853823 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820018680 con 0x7f9838078de0 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.717+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98381a44a0 con 0x7f9838078de0 2024-03-18T01:35:07.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.717+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98381abd20 con 0x7f9838078de0 2024-03-18T01:35:07.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.718+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9838068f90 con 0x7f9838078de0 2024-03-18T01:35:07.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.719+0000 7f9835600700 1 -- 172.21.15.7:0/1766853823 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f9820010430 con 0x7f9838078de0 2024-03-18T01:35:07.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.721+0000 7f9835600700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f981c0a2910 0x7f981c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:07.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.721+0000 7f9835600700 1 -- 172.21.15.7:0/1766853823 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98200ceeb0 con 0x7f9838078de0 2024-03-18T01:35:07.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.721+0000 7f9837e00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f981c0a2910 0x7f981c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:07.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.722+0000 7f9837e00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f981c0a2910 0x7f981c0a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9828006010 tx=0x7f982800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:07.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:07.727+0000 7f9835600700 1 -- 172.21.15.7:0/1766853823 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f982009bcb0 con 0x7f9838078de0 2024-03-18T01:35:08.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.008+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f983802db70 con 0x7f9838078de0 2024-03-18T01:35:08.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.009+0000 7f9835600700 1 -- 172.21.15.7:0/1766853823 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f982009b5b0 con 0x7f9838078de0 2024-03-18T01:35:08.009 INFO:teuthology.orchestra.run.smithi007.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T01:35:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f981c0a2910 msgr2=0x7f981c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f981c0a2910 0x7f981c0a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9828006010 tx=0x7f982800b260 comp rx=0 tx=0).stop 2024-03-18T01:35:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078de0 msgr2=0x7f983819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078de0 0x7f983819f6e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f982000e9a0 tx=0x7f982000ecb0 comp rx=0 tx=0).stop 2024-03-18T01:35:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 shutdown_connections 2024-03-18T01:35:08.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f981c0a2910 0x7f981c0a4dd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9838113140 0x7f98381a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9838078430 0x7f983819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 --2- 172.21.15.7:0/1766853823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9838078de0 0x7f983819f6e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 >> 172.21.15.7:0/1766853823 conn(0x7f98380fff70 msgr2=0x7f9838102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 shutdown_connections 2024-03-18T01:35:08.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08.012+0000 7f983ee00700 1 -- 172.21.15.7:0/1766853823 wait complete. 2024-03-18T01:35:08.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:08 smithi007 ceph-mon[31996]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-18T01:35:08.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:08 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1766853823' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:35:08.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:08 smithi046 ceph-mon[36108]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-18T01:35:08.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:08 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1766853823' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:35:08.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:08 smithi203 ceph-mon[36345]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-18T01:35:08.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:08 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1766853823' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:35:08.791 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-18T01:35:08.791 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T01:35:08.791 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2024-03-18T01:35:08.791 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-18T01:35:08.791 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T01:35:08.792 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-18T01:35:08.792 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T01:35:08.792 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T01:35:08.792 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T01:35:08.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:08.792 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2024-03-18T01:35:08.847 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T01:35:08.847 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T01:35:08.847 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T01:35:08.847 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:35:08.847 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2024-03-18T01:35:08.847 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:35:08.847 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 41966sec preferred_lft 41966sec 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-03-18T01:35:08.848 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:35:08.849 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:35:08.849 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2024-03-18T01:35:08.849 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2024-03-18T01:35:08.849 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T01:35:08.850 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:35:08.850 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:35:08.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add name ceph-brx type bridge 2024-03-18T01:35:08.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr flush dev ceph-brx 2024-03-18T01:35:08.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set ceph-brx up 2024-03-18T01:35:08.850 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T01:35:08.850 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:35:08.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:35:09.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:35:09.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:09.391 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T01:35:09.473 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-18T01:35:09.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:09.476 DEBUG:teuthology.orchestra.run.smithi007:> route 2024-03-18T01:35:09.531 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2024-03-18T01:35:09.531 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T01:35:09.531 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T01:35:09.531 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T01:35:09.531 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T01:35:09.532 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:35:09.532 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:35:09.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T01:35:09.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T01:35:09.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T01:35:09.532 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:35:09.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:35:10.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:35:10.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:10.321 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-03-18T01:35:10.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:10.380 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list-id 2024-03-18T01:35:10.442 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:35:10.442 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:35:10.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T01:35:10.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-18T01:35:10.443 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:35:10.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:10 smithi046 ceph-mon[36108]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:35:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:10 smithi007 ceph-mon[31996]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:35:10.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:35:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:10 smithi203 ceph-mon[36345]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:35:10.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:35:10.667 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-18T01:35:10.668 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:35:10.668 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:35:10.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-18T01:35:10.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T01:35:10.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-18T01:35:10.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-18T01:35:10.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-18T01:35:10.668 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:35:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:10 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:35:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:35:11.015 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:35:11.015 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:35:11.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2024-03-18T01:35:11.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T01:35:11.015 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:35:11.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:35:11.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:35:11.259 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-18T01:35:11.259 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T01:35:11.259 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-18T01:35:11.305 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T01:35:11.306 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T01:35:11.306 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-18T01:35:11.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-03-18T01:35:11.509 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T01:35:11.563 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-03-18T01:35:11.563 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-03-18T01:35:11.563 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/selinux 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-03-18T01:35:11.564 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/ae13e8987546ded52f0bf522631cae3e3b17e9b5c3e517933edd4a59d00689ee/merged 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/28207208c434e4ac655476836de43d7838a129ff7a0df8020bd8d5d40c1fac5c/merged 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/0 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T01:35:11.565 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/6e5583c902cefc6355816e3bdc481ac40496b3a5f7d8f7b3ebae8da410723fb3/merged 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/edf761548a8356e033aaafeafaf3a8c4d69642e01ed3e794df48eb267af10349/merged 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/5e6a93307d90104e018d55c3a6c5487e1b1abdadc7cd4c4e4f509da4a882a0db/merged 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/21fbd314c0a6e73e2b11c5809570de2367c94bce82d27b3e592954a33f8f9ae6/merged 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/159387445a34403381094012efe1e5517013d5a2e5857a238e9104b4c91c788c/merged 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/a55f3951dcada03671ca94c123a559ca066ffbbd6431913e6e30a8ab62a0feca/merged 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/d80b4683e3538a7f5a90b60549c386081d39a4d91aad6400df8690084b26c8e9/merged 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/a1c453230caed3b965be1842cff7eea2301c431c250bc64bdfa4ae86ab8f8704/merged 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T01:35:11.566 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T01:35:11.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:11.568 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-03-18T01:35:11.617 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T01:35:11.617 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-18T01:35:11.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo modprobe fuse 2024-03-18T01:35:11.698 DEBUG:teuthology.orchestra.run.smithi007:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T01:35:11.745 INFO:teuthology.orchestra.run.smithi007.stdout:/proc 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/sys 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/dev 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/security 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/shm 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/pts 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/run 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/cgroup 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/pstore 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/bpf 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/ 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/selinux 2024-03-18T01:35:11.746 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/hugepages 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/dev/mqueue 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/debug 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/tracing 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/fs/fuse/connections 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/sys/kernel/config 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/1000 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/ceph 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay 2024-03-18T01:35:11.747 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/ae13e8987546ded52f0bf522631cae3e3b17e9b5c3e517933edd4a59d00689ee/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/28207208c434e4ac655476836de43d7838a129ff7a0df8020bd8d5d40c1fac5c/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/run/user/0 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/6e5583c902cefc6355816e3bdc481ac40496b3a5f7d8f7b3ebae8da410723fb3/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/edf761548a8356e033aaafeafaf3a8c4d69642e01ed3e794df48eb267af10349/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/5e6a93307d90104e018d55c3a6c5487e1b1abdadc7cd4c4e4f509da4a882a0db/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/21fbd314c0a6e73e2b11c5809570de2367c94bce82d27b3e592954a33f8f9ae6/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/159387445a34403381094012efe1e5517013d5a2e5857a238e9104b4c91c788c/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/a55f3951dcada03671ca94c123a559ca066ffbbd6431913e6e30a8ab62a0feca/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/d80b4683e3538a7f5a90b60549c386081d39a4d91aad6400df8690084b26c8e9/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/var/lib/containers/storage/overlay/a1c453230caed3b965be1842cff7eea2301c431c250bc64bdfa4ae86ab8f8704/merged 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T01:35:11.748 INFO:teuthology.orchestra.run.smithi007.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T01:35:11.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:11.749 DEBUG:teuthology.orchestra.run.smithi007:> ls /sys/fs/fuse/connections 2024-03-18T01:35:11.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:2024-03-18T01:35:11.821+0000 7f8e1717c480 -1 init, newargv = 0x55fd73a99520 newargc=15 2024-03-18T01:35:11.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[76004]: starting ceph client 2024-03-18T01:35:11.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[76004]: starting fuse 2024-03-18T01:35:11.840 INFO:teuthology.orchestra.run.smithi007.stdout:118 2024-03-18T01:35:11.840 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-18T01:35:11.840 DEBUG:teuthology.orchestra.run.smithi007:> sudo stdin-killer -- python3 -c ' 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> import glob 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> import re 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> import os 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> import subprocess 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> def _find_admin_socket(client_name): 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> files = glob.glob(asok_path) 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> # Given a non-glob path, it better be there 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> if "*" not in asok_path: 2024-03-18T01:35:11.841 DEBUG:teuthology.orchestra.run.smithi007:> assert(len(files) == 1) 2024-03-18T01:35:11.842 DEBUG:teuthology.orchestra.run.smithi007:> return files[0] 2024-03-18T01:35:11.842 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-18T01:35:11.842 DEBUG:teuthology.orchestra.run.smithi007:> for f in files: 2024-03-18T01:35:11.842 DEBUG:teuthology.orchestra.run.smithi007:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T01:35:11.842 DEBUG:teuthology.orchestra.run.smithi007:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T01:35:11.842 DEBUG:teuthology.orchestra.run.smithi007:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T01:35:11.842 DEBUG:teuthology.orchestra.run.smithi007:> contents = proc_f.read() 2024-03-18T01:35:11.842 DEBUG:teuthology.orchestra.run.smithi007:> if mountpoint in contents: 2024-03-18T01:35:11.843 DEBUG:teuthology.orchestra.run.smithi007:> return f 2024-03-18T01:35:11.843 DEBUG:teuthology.orchestra.run.smithi007:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T01:35:11.843 DEBUG:teuthology.orchestra.run.smithi007:> 2024-03-18T01:35:11.843 DEBUG:teuthology.orchestra.run.smithi007:> print(_find_admin_socket("client.admin")) 2024-03-18T01:35:11.843 DEBUG:teuthology.orchestra.run.smithi007:> ' 2024-03-18T01:35:12.045 INFO:teuthology.orchestra.run.smithi007.stdout:/var/run/ceph/ceph-client.admin.76004.asok 2024-03-18T01:35:12.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:35:12.063 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76004.asok 2024-03-18T01:35:12.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:12.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76004.asok status 2024-03-18T01:35:12.304 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:35:12.304 INFO:teuthology.orchestra.run.smithi007.stdout: "metadata": { 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "entity_id": "admin", 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": "smithi007", 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "pid": "76004", 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "root": "/" 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_count": 0, 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "dentry_pinned_count": 0, 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 14601, 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "inst": { 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "name": { 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "client", 2024-03-18T01:35:12.305 INFO:teuthology.orchestra.run.smithi007.stdout: "num": 14601 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2517328280 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": { 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "192.168.0.1:0", 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2517328280 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "inst_str": "client.14601 192.168.0.1:0/2517328280", 2024-03-18T01:35:12.306 INFO:teuthology.orchestra.run.smithi007.stdout: "addr_str": "192.168.0.1:0/2517328280", 2024-03-18T01:35:12.307 INFO:teuthology.orchestra.run.smithi007.stdout: "inode_count": 1, 2024-03-18T01:35:12.307 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_epoch": 17, 2024-03-18T01:35:12.307 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch": 89, 2024-03-18T01:35:12.307 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_epoch_barrier": 0, 2024-03-18T01:35:12.307 INFO:teuthology.orchestra.run.smithi007.stdout: "blocklisted": false, 2024-03-18T01:35:12.307 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs" 2024-03-18T01:35:12.307 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:35:12.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:12.321 DEBUG:teuthology.orchestra.run.smithi007:> stat --file-system '--printf=%T 2024-03-18T01:35:12.321 DEBUG:teuthology.orchestra.run.smithi007:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-18T01:35:12.370 INFO:teuthology.orchestra.run.smithi007.stdout:fuseblk 2024-03-18T01:35:12.371 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-18T01:35:12.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:12.371 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-18T01:35:12.494 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:12 smithi007 ceph-mon[31996]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:35:12.511 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-18T01:35:12.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:12 smithi046 ceph-mon[36108]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:35:12.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:35:12.638 INFO:teuthology.orchestra.run.smithi007.stdout:# file: . 2024-03-18T01:35:12.639 INFO:teuthology.orchestra.run.smithi007.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-18T01:35:12.639 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:35:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:12 smithi203 ceph-mon[36345]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:35:12.649 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:35:12.649 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:35:12.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:35:12.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.926+0000 7f9dc5000640 1 -- 172.21.15.7:0/2689490169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 msgr2=0x7f9dc0078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:12.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.926+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2689490169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 0x7f9dc0078620 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009ca0 tx=0x7f9db802c100 comp rx=0 tx=0).stop 2024-03-18T01:35:12.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.926+0000 7f9dc5000640 1 -- 172.21.15.7:0/2689490169 shutdown_connections 2024-03-18T01:35:12.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.927+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2689490169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 0x7f9dc0078620 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:12.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.927+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2689490169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc00773e0 0x7f9dc0077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:12.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.927+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2689490169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00761a0 0x7f9dc00765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:12.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.927+0000 7f9dc5000640 1 -- 172.21.15.7:0/2689490169 >> 172.21.15.7:0/2689490169 conn(0x7f9dc00fef90 msgr2=0x7f9dc01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:12.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.927+0000 7f9dc5000640 1 -- 172.21.15.7:0/2689490169 shutdown_connections 2024-03-18T01:35:12.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.927+0000 7f9dc5000640 1 -- 172.21.15.7:0/2689490169 wait complete. 2024-03-18T01:35:12.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.928+0000 7f9dc5000640 1 Processor -- start 2024-03-18T01:35:12.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.928+0000 7f9dc5000640 1 -- start start 2024-03-18T01:35:12.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.928+0000 7f9dc5000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00761a0 0x7f9dc019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:12.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.928+0000 7f9dc5000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc00773e0 0x7f9dc019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:12.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.928+0000 7f9dc5000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 0x7f9dc01a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:12.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.928+0000 7f9dc5000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc0113a90 con 0x7f9dc00773e0 2024-03-18T01:35:12.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.928+0000 7f9dc5000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc0113910 con 0x7f9dc0077d80 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.928+0000 7f9dc5000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc0113c10 con 0x7f9dc00761a0 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 0x7f9dc01a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00761a0 0x7f9dc019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbd400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc00773e0 0x7f9dc019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 0x7f9dc01a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:39152/0 (socket says 172.21.15.7:39152) 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 -- 172.21.15.7:0/2906704026 learned_addr learned my addr 172.21.15.7:0/2906704026 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 -- 172.21.15.7:0/2906704026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00761a0 msgr2=0x7f9dc019e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00761a0 0x7f9dc019e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 -- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc00773e0 msgr2=0x7f9dc019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc00773e0 0x7f9dc019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 -- 172.21.15.7:0/2906704026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc01a5c70 con 0x7f9dc0077d80 2024-03-18T01:35:12.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.929+0000 7f9dbe800640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 0x7f9dc01a5600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9db802c640 tx=0x7f9db8035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:12.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.930+0000 7f9db6a00640 1 -- 172.21.15.7:0/2906704026 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db8039040 con 0x7f9dc0077d80 2024-03-18T01:35:12.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.930+0000 7f9db6a00640 1 -- 172.21.15.7:0/2906704026 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9db8044070 con 0x7f9dc0077d80 2024-03-18T01:35:12.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.930+0000 7f9db6a00640 1 -- 172.21.15.7:0/2906704026 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db8004460 con 0x7f9dc0077d80 2024-03-18T01:35:12.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.930+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc01a5f00 con 0x7f9dc0077d80 2024-03-18T01:35:12.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.930+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc01a63c0 con 0x7f9dc0077d80 2024-03-18T01:35:12.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.931+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc0110370 con 0x7f9dc0077d80 2024-03-18T01:35:12.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.933+0000 7f9db6a00640 1 -- 172.21.15.7:0/2906704026 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f9db8004600 con 0x7f9dc0077d80 2024-03-18T01:35:12.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.934+0000 7f9db6a00640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9da40a2880 0x7f9da40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:12.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.934+0000 7f9db6a00640 1 -- 172.21.15.7:0/2906704026 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9db80f1280 con 0x7f9dc0077d80 2024-03-18T01:35:12.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.934+0000 7f9dbde00640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9da40a2880 0x7f9da40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:12.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.935+0000 7f9dbde00640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9da40a2880 0x7f9da40a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009bd0 tx=0x7f9db0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:12.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:12.938+0000 7f9db6a00640 1 -- 172.21.15.7:0/2906704026 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9db80be0e0 con 0x7f9dc0077d80 2024-03-18T01:35:13.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.145+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9dc00623e0 con 0x7f9dc0077d80 2024-03-18T01:35:13.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.147+0000 7f9db6a00640 1 -- 172.21.15.7:0/2906704026 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7f9db80bd9e0 con 0x7f9dc0077d80 2024-03-18T01:35:13.148 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:13.148 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:13.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9da40a2880 msgr2=0x7f9da40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9da40a2880 0x7f9da40a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009bd0 tx=0x7f9db0005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 msgr2=0x7f9dc01a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 0x7f9dc01a5600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9db802c640 tx=0x7f9db8035d10 comp rx=0 tx=0).stop 2024-03-18T01:35:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 shutdown_connections 2024-03-18T01:35:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9dc0077d80 0x7f9dc01a5600 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9da40a2880 0x7f9da40a4d40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9dc00773e0 0x7f9dc019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 --2- 172.21.15.7:0/2906704026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00761a0 0x7f9dc019e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 >> 172.21.15.7:0/2906704026 conn(0x7f9dc00fef90 msgr2=0x7f9dc0100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:13.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 shutdown_connections 2024-03-18T01:35:13.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.150+0000 7f9dc5000640 1 -- 172.21.15.7:0/2906704026 wait complete. 2024-03-18T01:35:13.173 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:35:13.174 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:35:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 -- 172.21.15.7:0/1218075815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2124107c40 msgr2=0x7f2124108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 --2- 172.21.15.7:0/1218075815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2124107c40 0x7f2124108060 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2118009900 tx=0x7f211802bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 -- 172.21.15.7:0/1218075815 shutdown_connections 2024-03-18T01:35:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 --2- 172.21.15.7:0/1218075815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21241097f0 0x7f2124110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 --2- 172.21.15.7:0/1218075815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2124108e30 0x7f21241092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 --2- 172.21.15.7:0/1218075815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2124107c40 0x7f2124108060 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 -- 172.21.15.7:0/1218075815 >> 172.21.15.7:0/1218075815 conn(0x7f2124076bd0 msgr2=0x7f2124079010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:13.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 -- 172.21.15.7:0/1218075815 shutdown_connections 2024-03-18T01:35:13.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.392+0000 7f212ac00640 1 -- 172.21.15.7:0/1218075815 wait complete. 2024-03-18T01:35:13.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212ac00640 1 Processor -- start 2024-03-18T01:35:13.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212ac00640 1 -- start start 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212ac00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2124107c40 0x7f212419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212ac00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2124108e30 0x7f212419e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212ac00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21241097f0 0x7f21241a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212ac00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f21241138e0 con 0x7f2124108e30 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212ac00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2124113760 con 0x7f21241097f0 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212ac00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2124113a60 con 0x7f2124107c40 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212a200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21241097f0 0x7f21241a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f2128e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2124108e30 0x7f212419e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212a200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21241097f0 0x7f21241a5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:39172/0 (socket says 172.21.15.7:39172) 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f212a200640 1 -- 172.21.15.7:0/3692647724 learned_addr learned my addr 172.21.15.7:0/3692647724 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.393+0000 7f2129800640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2124107c40 0x7f212419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f212a200640 1 -- 172.21.15.7:0/3692647724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2124107c40 msgr2=0x7f212419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f212a200640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2124107c40 0x7f212419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f212a200640 1 -- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2124108e30 msgr2=0x7f212419e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f212a200640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2124108e30 0x7f212419e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f212a200640 1 -- 172.21.15.7:0/3692647724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21241a5ba0 con 0x7f21241097f0 2024-03-18T01:35:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f212a200640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21241097f0 0x7f21241a5530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f211c00eb10 tx=0x7f211c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f2122000640 1 -- 172.21.15.7:0/3692647724 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c004b20 con 0x7f21241097f0 2024-03-18T01:35:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f2122000640 1 -- 172.21.15.7:0/3692647724 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f211c004cc0 con 0x7f21241097f0 2024-03-18T01:35:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f2122000640 1 -- 172.21.15.7:0/3692647724 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c010760 con 0x7f21241097f0 2024-03-18T01:35:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.394+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21241a5e90 con 0x7f21241097f0 2024-03-18T01:35:13.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.395+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21241a6350 con 0x7f21241097f0 2024-03-18T01:35:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.396+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f212410efb0 con 0x7f21241097f0 2024-03-18T01:35:13.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.397+0000 7f2122000640 1 -- 172.21.15.7:0/3692647724 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f211c005020 con 0x7f21241097f0 2024-03-18T01:35:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.399+0000 7f2122000640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f210c0a2880 0x7f210c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.399+0000 7f2122000640 1 -- 172.21.15.7:0/3692647724 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f211c0cf1a0 con 0x7f21241097f0 2024-03-18T01:35:13.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.399+0000 7f2129800640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f210c0a2880 0x7f210c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:13.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.400+0000 7f2129800640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f210c0a2880 0x7f210c0a4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f211802c170 tx=0x7f211802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:13.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.403+0000 7f2122000640 1 -- 172.21.15.7:0/3692647724 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f211c09bfa0 con 0x7f21241097f0 2024-03-18T01:35:13.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:13 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2906704026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:13.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:13 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2906704026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:13.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.599+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2124062480 con 0x7f21241097f0 2024-03-18T01:35:13.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.600+0000 7f2122000640 1 -- 172.21.15.7:0/3692647724 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f211c09b8a0 con 0x7f21241097f0 2024-03-18T01:35:13.600 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.602+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f210c0a2880 msgr2=0x7f210c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.602+0000 7f212ac00640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f210c0a2880 0x7f210c0a4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f211802c170 tx=0x7f211802df90 comp rx=0 tx=0).stop 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.602+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21241097f0 msgr2=0x7f21241a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.602+0000 7f212ac00640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21241097f0 0x7f21241a5530 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f211c00eb10 tx=0x7f211c00c2d0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.603+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 shutdown_connections 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.603+0000 7f212ac00640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21241097f0 0x7f21241a5530 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.603+0000 7f212ac00640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f210c0a2880 0x7f210c0a4d40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.603+0000 7f212ac00640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2124108e30 0x7f212419e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.603+0000 7f212ac00640 1 --2- 172.21.15.7:0/3692647724 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2124107c40 0x7f212419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.603+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 >> 172.21.15.7:0/3692647724 conn(0x7f2124076bd0 msgr2=0x7f212410c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:13.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.603+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 shutdown_connections 2024-03-18T01:35:13.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.603+0000 7f212ac00640 1 -- 172.21.15.7:0/3692647724 wait complete. 2024-03-18T01:35:13.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:35:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:13 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2906704026' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.844+0000 7ff686800640 1 -- 172.21.15.7:0/4095100441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 msgr2=0x7ff680110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.844+0000 7ff686800640 1 --2- 172.21.15.7:0/4095100441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 0x7ff680110250 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff678009be0 tx=0x7ff67802bef0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.844+0000 7ff686800640 1 -- 172.21.15.7:0/4095100441 shutdown_connections 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.844+0000 7ff686800640 1 --2- 172.21.15.7:0/4095100441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 0x7ff680110250 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.844+0000 7ff686800640 1 --2- 172.21.15.7:0/4095100441 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff680108f70 0x7ff6801093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.844+0000 7ff686800640 1 --2- 172.21.15.7:0/4095100441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff680107e20 0x7ff680108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.844+0000 7ff686800640 1 -- 172.21.15.7:0/4095100441 >> 172.21.15.7:0/4095100441 conn(0x7ff680076dd0 msgr2=0x7ff6800791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.845+0000 7ff686800640 1 -- 172.21.15.7:0/4095100441 shutdown_connections 2024-03-18T01:35:13.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.845+0000 7ff686800640 1 -- 172.21.15.7:0/4095100441 wait complete. 2024-03-18T01:35:13.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.845+0000 7ff686800640 1 Processor -- start 2024-03-18T01:35:13.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff686800640 1 -- start start 2024-03-18T01:35:13.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff686800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff680107e20 0x7ff68019e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff686800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff680108f70 0x7ff68019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff686800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 0x7ff6801a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff686800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff680113aa0 con 0x7ff680107e20 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff686800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff680113920 con 0x7ff680109930 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff686800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff680113c20 con 0x7ff680108f70 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff67f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff680107e20 0x7ff68019e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff67fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 0x7ff6801a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff67f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff680107e20 0x7ff68019e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58794/0 (socket says 172.21.15.7:58794) 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff67f400640 1 -- 172.21.15.7:0/258570711 learned_addr learned my addr 172.21.15.7:0/258570711 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.846+0000 7ff67ea00640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff680108f70 0x7ff68019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff67fe00640 1 -- 172.21.15.7:0/258570711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff680108f70 msgr2=0x7ff68019e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff67fe00640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff680108f70 0x7ff68019e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff67fe00640 1 -- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff680107e20 msgr2=0x7ff68019e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:13.848 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff67fe00640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff680107e20 0x7ff68019e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff67fe00640 1 -- 172.21.15.7:0/258570711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6801a5ca0 con 0x7ff680109930 2024-03-18T01:35:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff67fe00640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 0x7ff6801a5630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff67802c430 tx=0x7ff67802cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff677e00640 1 -- 172.21.15.7:0/258570711 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678043020 con 0x7ff680109930 2024-03-18T01:35:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff677e00640 1 -- 172.21.15.7:0/258570711 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff678003e50 con 0x7ff680109930 2024-03-18T01:35:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff677e00640 1 -- 172.21.15.7:0/258570711 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678034690 con 0x7ff680109930 2024-03-18T01:35:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.847+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6801a5f30 con 0x7ff680109930 2024-03-18T01:35:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.848+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6801a63a0 con 0x7ff680109930 2024-03-18T01:35:13.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.848+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff68010f130 con 0x7ff680109930 2024-03-18T01:35:13.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.850+0000 7ff677e00640 1 -- 172.21.15.7:0/258570711 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7ff678046020 con 0x7ff680109930 2024-03-18T01:35:13.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.852+0000 7ff677e00640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff6640a2880 0x7ff6640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:13.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.852+0000 7ff677e00640 1 -- 172.21.15.7:0/258570711 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff6780f18b0 con 0x7ff680109930 2024-03-18T01:35:13.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.852+0000 7ff67f400640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff6640a2880 0x7ff6640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:13.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.853+0000 7ff67f400640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff6640a2880 0x7ff6640a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff670009c90 tx=0x7ff670005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:13.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:13.854+0000 7ff677e00640 1 -- 172.21.15.7:0/258570711 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff67804d020 con 0x7ff680109930 2024-03-18T01:35:14.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.048+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff68002d400 con 0x7ff680109930 2024-03-18T01:35:14.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.048+0000 7ff677e00640 1 -- 172.21.15.7:0/258570711 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7ff6780be710 con 0x7ff680109930 2024-03-18T01:35:14.048 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:35:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff6640a2880 msgr2=0x7ff6640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff6640a2880 0x7ff6640a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff670009c90 tx=0x7ff670005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:14.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 msgr2=0x7ff6801a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 0x7ff6801a5630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff67802c430 tx=0x7ff67802cb90 comp rx=0 tx=0).stop 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 shutdown_connections 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff680109930 0x7ff6801a5630 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff680108f70 0x7ff68019e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff6640a2880 0x7ff6640a4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 --2- 172.21.15.7:0/258570711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff680107e20 0x7ff68019e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 >> 172.21.15.7:0/258570711 conn(0x7ff680076dd0 msgr2=0x7ff68010c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 shutdown_connections 2024-03-18T01:35:14.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.051+0000 7ff686800640 1 -- 172.21.15.7:0/258570711 wait complete. 2024-03-18T01:35:14.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:35:14.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 -- 172.21.15.7:0/719363426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 msgr2=0x7f3d9c109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 --2- 172.21.15.7:0/719363426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 0x7f3d9c109760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3d94009900 tx=0x7f3d9402bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:14.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 -- 172.21.15.7:0/719363426 shutdown_connections 2024-03-18T01:35:14.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 --2- 172.21.15.7:0/719363426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 0x7f3d9c109760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 --2- 172.21.15.7:0/719363426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c1049e0 0x7f3d9c106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 --2- 172.21.15.7:0/719363426 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d9c1020b0 0x7f3d9c1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 -- 172.21.15.7:0/719363426 >> 172.21.15.7:0/719363426 conn(0x7f3d9c0fbac0 msgr2=0x7f3d9c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:14.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 -- 172.21.15.7:0/719363426 shutdown_connections 2024-03-18T01:35:14.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.295+0000 7f3da1600640 1 -- 172.21.15.7:0/719363426 wait complete. 2024-03-18T01:35:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.296+0000 7f3da1600640 1 Processor -- start 2024-03-18T01:35:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.296+0000 7f3da1600640 1 -- start start 2024-03-18T01:35:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.296+0000 7f3da1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c1020b0 0x7f3d9c101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.296+0000 7f3da1600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d9c1049e0 0x7f3d9c100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.296+0000 7f3da1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 0x7f3d9c1005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.296+0000 7f3da1600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c10d310 con 0x7f3d9c1049e0 2024-03-18T01:35:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.296+0000 7f3da1600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c10d190 con 0x7f3d9c107310 2024-03-18T01:35:14.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.296+0000 7f3da1600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9c10d490 con 0x7f3d9c1020b0 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3d9b400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d9c1049e0 0x7f3d9c100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3d9be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c1020b0 0x7f3d9c101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3da0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 0x7f3d9c1005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3da0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 0x7f3d9c1005a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:39196/0 (socket says 172.21.15.7:39196) 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3d9be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c1020b0 0x7f3d9c101a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:37142/0 (socket says 172.21.15.7:37142) 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3da0c00640 1 -- 172.21.15.7:0/3221282032 learned_addr learned my addr 172.21.15.7:0/3221282032 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3da0c00640 1 -- 172.21.15.7:0/3221282032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c1020b0 msgr2=0x7f3d9c101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3da0c00640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c1020b0 0x7f3d9c101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3da0c00640 1 -- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d9c1049e0 msgr2=0x7f3d9c100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3da0c00640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d9c1049e0 0x7f3d9c100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.297+0000 7f3da0c00640 1 -- 172.21.15.7:0/3221282032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9c100e30 con 0x7f3d9c107310 2024-03-18T01:35:14.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.298+0000 7f3da0c00640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 0x7f3d9c1005a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d9402c8e0 tx=0x7f3d9402c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.298+0000 7f3d98c00640 1 -- 172.21.15.7:0/3221282032 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94043020 con 0x7f3d9c107310 2024-03-18T01:35:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.298+0000 7f3d98c00640 1 -- 172.21.15.7:0/3221282032 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d940054e0 con 0x7f3d9c107310 2024-03-18T01:35:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.298+0000 7f3d98c00640 1 -- 172.21.15.7:0/3221282032 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94003ce0 con 0x7f3d9c107310 2024-03-18T01:35:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.298+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9c1a1a10 con 0x7f3d9c107310 2024-03-18T01:35:14.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.298+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9c1a1fb0 con 0x7f3d9c107310 2024-03-18T01:35:14.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.299+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d9c0fd5b0 con 0x7f3d9c107310 2024-03-18T01:35:14.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.300+0000 7f3d98c00640 1 -- 172.21.15.7:0/3221282032 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f3d94046020 con 0x7f3d9c107310 2024-03-18T01:35:14.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.302+0000 7f3d98c00640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3d800a2880 0x7f3d800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:14.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.302+0000 7f3d98c00640 1 -- 172.21.15.7:0/3221282032 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3d940f0d00 con 0x7f3d9c107310 2024-03-18T01:35:14.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.302+0000 7f3d9be00640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3d800a2880 0x7f3d800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:14.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.303+0000 7f3d9be00640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3d800a2880 0x7f3d800a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c005fd0 tx=0x7f3d8c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:14.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.306+0000 7f3d98c00640 1 -- 172.21.15.7:0/3221282032 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3d9404d020 con 0x7f3d9c107310 2024-03-18T01:35:14.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.500+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3d9c0623e0 con 0x7f3d9c107310 2024-03-18T01:35:14.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.501+0000 7f3d98c00640 1 -- 172.21.15.7:0/3221282032 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f3d940bdb60 con 0x7f3d9c107310 2024-03-18T01:35:14.501 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.503+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3d800a2880 msgr2=0x7f3d800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.503+0000 7f3da1600640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3d800a2880 0x7f3d800a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3d8c005fd0 tx=0x7f3d8c009500 comp rx=0 tx=0).stop 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 msgr2=0x7f3d9c1005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 0x7f3d9c1005a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d9402c8e0 tx=0x7f3d9402c6a0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 shutdown_connections 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3d9c107310 0x7f3d9c1005a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3d800a2880 0x7f3d800a4d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3d9c1049e0 0x7f3d9c100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 --2- 172.21.15.7:0/3221282032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d9c1020b0 0x7f3d9c101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 >> 172.21.15.7:0/3221282032 conn(0x7f3d9c0fbac0 msgr2=0x7f3d9c102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:14.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 shutdown_connections 2024-03-18T01:35:14.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.504+0000 7f3da1600640 1 -- 172.21.15.7:0/3221282032 wait complete. 2024-03-18T01:35:14.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:14 smithi046 ceph-mon[36108]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 370 B/s wr, 0 op/s 2024-03-18T01:35:14.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:14 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3692647724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:35:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:14 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/258570711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:35:14.519 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:14 smithi007 ceph-mon[31996]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 370 B/s wr, 0 op/s 2024-03-18T01:35:14.519 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:14 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3692647724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:35:14.519 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:14 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/258570711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:35:14.521 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:35:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:14 smithi203 ceph-mon[36345]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 370 B/s wr, 0 op/s 2024-03-18T01:35:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:14 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3692647724' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:35:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:14 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/258570711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:35:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 -- 172.21.15.7:0/621678112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec1094f0 msgr2=0x7f17ec10b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 --2- 172.21.15.7:0/621678112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec1094f0 0x7f17ec10b980 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009900 tx=0x7f17e802bc40 comp rx=0 tx=0).stop 2024-03-18T01:35:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 -- 172.21.15.7:0/621678112 shutdown_connections 2024-03-18T01:35:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 --2- 172.21.15.7:0/621678112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec1094f0 0x7f17ec10b980 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 --2- 172.21.15.7:0/621678112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17ec074240 0x7f17ec0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 --2- 172.21.15.7:0/621678112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec075820 0x7f17ec073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 -- 172.21.15.7:0/621678112 >> 172.21.15.7:0/621678112 conn(0x7f17ec0fcba0 msgr2=0x7f17ec0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 -- 172.21.15.7:0/621678112 shutdown_connections 2024-03-18T01:35:14.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.739+0000 7f17f3200640 1 -- 172.21.15.7:0/621678112 wait complete. 2024-03-18T01:35:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.740+0000 7f17f3200640 1 Processor -- start 2024-03-18T01:35:14.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.740+0000 7f17f3200640 1 -- start start 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f3200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17ec074240 0x7f17ec199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f3200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec075820 0x7f17ec19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1094f0 0x7f17ec1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f3200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec10f4e0 con 0x7f17ec074240 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f3200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec10f360 con 0x7f17ec075820 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec10f660 con 0x7f17ec1094f0 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f1e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17ec074240 0x7f17ec199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f1e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17ec074240 0x7f17ec199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:58822/0 (socket says 172.21.15.7:58822) 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f1e00640 1 -- 172.21.15.7:0/3453474347 learned_addr learned my addr 172.21.15.7:0/3453474347 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f1400640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec075820 0x7f17ec19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f2800640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1094f0 0x7f17ec1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:14.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f1e00640 1 -- 172.21.15.7:0/3453474347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1094f0 msgr2=0x7f17ec1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.741+0000 7f17f1e00640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1094f0 0x7f17ec1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17f1e00640 1 -- 172.21.15.7:0/3453474347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec075820 msgr2=0x7f17ec19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17f1e00640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec075820 0x7f17ec19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17f1e00640 1 -- 172.21.15.7:0/3453474347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec1a16e0 con 0x7f17ec074240 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17f1400640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec075820 0x7f17ec19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17f1e00640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17ec074240 0x7f17ec199ac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f17e400eb90 tx=0x7f17e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17e2a00640 1 -- 172.21.15.7:0/3453474347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4004d20 con 0x7f17ec074240 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec1a19d0 con 0x7f17ec074240 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17e2a00640 1 -- 172.21.15.7:0/3453474347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e400ee90 con 0x7f17ec074240 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17e2a00640 1 -- 172.21.15.7:0/3453474347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4010830 con 0x7f17ec074240 2024-03-18T01:35:14.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.742+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec1a1ed0 con 0x7f17ec074240 2024-03-18T01:35:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.743+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17ec108220 con 0x7f17ec074240 2024-03-18T01:35:14.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.746+0000 7f17e2a00640 1 -- 172.21.15.7:0/3453474347 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f17e4005020 con 0x7f17ec074240 2024-03-18T01:35:14.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.747+0000 7f17e2a00640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17d80a2800 0x7f17d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.747+0000 7f17e2a00640 1 -- 172.21.15.7:0/3453474347 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f17e40cf3e0 con 0x7f17ec074240 2024-03-18T01:35:14.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.747+0000 7f17f1400640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17d80a2800 0x7f17d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:14.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.748+0000 7f17f1400640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17d80a2800 0x7f17d80a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f17ec19afd0 tx=0x7f17dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:14.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.752+0000 7f17e2a00640 1 -- 172.21.15.7:0/3453474347 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17e409c1e0 con 0x7f17ec074240 2024-03-18T01:35:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.957+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f17ec0623e0 con 0x7f17ec074240 2024-03-18T01:35:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.958+0000 7f17e2a00640 1 -- 172.21.15.7:0/3453474347 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f17e409bae0 con 0x7f17ec074240 2024-03-18T01:35:14.958 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:35:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17d80a2800 msgr2=0x7f17d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17d80a2800 0x7f17d80a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f17ec19afd0 tx=0x7f17dc005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17ec074240 msgr2=0x7f17ec199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17ec074240 0x7f17ec199ac0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f17e400eb90 tx=0x7f17e400c6a0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 shutdown_connections 2024-03-18T01:35:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17ec1094f0 0x7f17ec1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f17ec075820 0x7f17ec19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f17d80a2800 0x7f17d80a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 --2- 172.21.15.7:0/3453474347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f17ec074240 0x7f17ec199ac0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 >> 172.21.15.7:0/3453474347 conn(0x7f17ec0fcba0 msgr2=0x7f17ec1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 shutdown_connections 2024-03-18T01:35:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:14.961+0000 7f17f3200640 1 -- 172.21.15.7:0/3453474347 wait complete. 2024-03-18T01:35:14.981 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.201+0000 7fe210c00640 1 -- 172.21.15.7:0/311732466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c108f70 msgr2=0x7fe20c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.201+0000 7fe210c00640 1 --2- 172.21.15.7:0/311732466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c108f70 0x7fe20c1093f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4009860 tx=0x7fe1f402bb70 comp rx=0 tx=0).stop 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.201+0000 7fe210c00640 1 -- 172.21.15.7:0/311732466 shutdown_connections 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.201+0000 7fe210c00640 1 --2- 172.21.15.7:0/311732466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20c109930 0x7fe20c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.201+0000 7fe210c00640 1 --2- 172.21.15.7:0/311732466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c108f70 0x7fe20c1093f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.201+0000 7fe210c00640 1 --2- 172.21.15.7:0/311732466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe20c107e20 0x7fe20c108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.201+0000 7fe210c00640 1 -- 172.21.15.7:0/311732466 >> 172.21.15.7:0/311732466 conn(0x7fe20c076dd0 msgr2=0x7fe20c0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.201+0000 7fe210c00640 1 -- 172.21.15.7:0/311732466 shutdown_connections 2024-03-18T01:35:15.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.202+0000 7fe210c00640 1 -- 172.21.15.7:0/311732466 wait complete. 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.203+0000 7fe210c00640 1 Processor -- start 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.203+0000 7fe210c00640 1 -- start start 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.203+0000 7fe210c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c107e20 0x7fe20c19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.203+0000 7fe210c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe20c108f70 0x7fe20c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.203+0000 7fe210c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20c109930 0x7fe20c1a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.203+0000 7fe210c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c113aa0 con 0x7fe20c108f70 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.203+0000 7fe210c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c113920 con 0x7fe20c107e20 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.203+0000 7fe210c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe20c113c20 con 0x7fe20c109930 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c107e20 0x7fe20c19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c107e20 0x7fe20c19e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:39230/0 (socket says 172.21.15.7:39230) 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 -- 172.21.15.7:0/1786937750 learned_addr learned my addr 172.21.15.7:0/1786937750 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe203e00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe20c108f70 0x7fe20c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:15.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe20a800640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20c109930 0x7fe20c1a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 -- 172.21.15.7:0/1786937750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20c109930 msgr2=0x7fe20c1a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20c109930 0x7fe20c1a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 -- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe20c108f70 msgr2=0x7fe20c19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe20c108f70 0x7fe20c19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 -- 172.21.15.7:0/1786937750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20c1a5d60 con 0x7fe20c107e20 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe203e00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe20c108f70 0x7fe20c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.204+0000 7fe209e00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c107e20 0x7fe20c19e140 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00e9f0 tx=0x7fe1fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.205+0000 7fe202a00640 1 -- 172.21.15.7:0/1786937750 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc00cd80 con 0x7fe20c107e20 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.205+0000 7fe202a00640 1 -- 172.21.15.7:0/1786937750 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1fc010910 con 0x7fe20c107e20 2024-03-18T01:35:15.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.205+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20c1a5ff0 con 0x7fe20c107e20 2024-03-18T01:35:15.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.205+0000 7fe202a00640 1 -- 172.21.15.7:0/1786937750 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc018990 con 0x7fe20c107e20 2024-03-18T01:35:15.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.205+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe20c1a64b0 con 0x7fe20c107e20 2024-03-18T01:35:15.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.206+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe20c10f200 con 0x7fe20c107e20 2024-03-18T01:35:15.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.208+0000 7fe202a00640 1 -- 172.21.15.7:0/1786937750 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fe1fc010ac0 con 0x7fe20c107e20 2024-03-18T01:35:15.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.209+0000 7fe202a00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.210+0000 7fe202a00640 1 -- 172.21.15.7:0/1786937750 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe1fc0cf190 con 0x7fe20c107e20 2024-03-18T01:35:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.210+0000 7fe203e00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:15.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.211+0000 7fe203e00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40349e0 tx=0x7fe1f40095b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:15.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.218+0000 7fe202a00640 1 -- 172.21.15.7:0/1786937750 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1fc09bf90 con 0x7fe20c107e20 2024-03-18T01:35:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.442+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe20c02d1b0 con 0x7fe20c107e20 2024-03-18T01:35:15.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.442+0000 7fe202a00640 1 -- 172.21.15.7:0/1786937750 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+582 (secure 0 0 0) 0x7fe1fc084e20 con 0x7fe20c107e20 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/1169030810 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3833186711 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/4234190466 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/1791220024 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/1169030810 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/1791220024 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3317891911 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2698516506 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2862047901 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:15.443 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3780594555 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:15.444 INFO:teuthology.orchestra.run.smithi007.stderr:listed 10 entries 2024-03-18T01:35:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.445+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe1f00a2800 msgr2=0x7fe1f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40349e0 tx=0x7fe1f40095b0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c107e20 msgr2=0x7fe20c19e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c107e20 0x7fe20c19e140 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00e9f0 tx=0x7fe1fc00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 shutdown_connections 2024-03-18T01:35:15.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20c109930 0x7fe20c1a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe1f00a2800 0x7fe1f00a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe20c108f70 0x7fe20c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 --2- 172.21.15.7:0/1786937750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe20c107e20 0x7fe20c19e140 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 >> 172.21.15.7:0/1786937750 conn(0x7fe20c076dd0 msgr2=0x7fe20c10c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 shutdown_connections 2024-03-18T01:35:15.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.446+0000 7fe210c00640 1 -- 172.21.15.7:0/1786937750 wait complete. 2024-03-18T01:35:15.465 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:35:15.465 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:35:15.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:35:15.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:15 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3221282032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:35:15.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:15 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3453474347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:35:15.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:15 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3221282032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:35:15.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:15 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3453474347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:35:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:15 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3221282032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:35:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:15 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3453474347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:35:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 -- 172.21.15.7:0/2831488269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350075820 msgr2=0x7fc350073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 --2- 172.21.15.7:0/2831488269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350075820 0x7fc350073c70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc340009cd0 tx=0x7fc34002c0e0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 -- 172.21.15.7:0/2831488269 shutdown_connections 2024-03-18T01:35:15.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 --2- 172.21.15.7:0/2831488269 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc35010dda0 0x7fc350110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 --2- 172.21.15.7:0/2831488269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc350074240 0x7fc3500746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 --2- 172.21.15.7:0/2831488269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350075820 0x7fc350073c70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 -- 172.21.15.7:0/2831488269 >> 172.21.15.7:0/2831488269 conn(0x7fc3500fcd40 msgr2=0x7fc3500ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 -- 172.21.15.7:0/2831488269 shutdown_connections 2024-03-18T01:35:15.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.734+0000 7fc355e00640 1 -- 172.21.15.7:0/2831488269 wait complete. 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.735+0000 7fc355e00640 1 Processor -- start 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.735+0000 7fc355e00640 1 -- start start 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.735+0000 7fc355e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350074240 0x7fc35019e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.735+0000 7fc355e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc350075820 0x7fc35019e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.735+0000 7fc355e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35010dda0 0x7fc3501a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.735+0000 7fc355e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc350113a40 con 0x7fc350075820 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.735+0000 7fc355e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3501138c0 con 0x7fc350074240 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.735+0000 7fc355e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc350113bc0 con 0x7fc35010dda0 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350074240 0x7fc35019e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:15.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350074240 0x7fc35019e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:39248/0 (socket says 172.21.15.7:39248) 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 -- 172.21.15.7:0/452533657 learned_addr learned my addr 172.21.15.7:0/452533657 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc346000640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc350075820 0x7fc35019e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34f400640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35010dda0 0x7fc3501a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 -- 172.21.15.7:0/452533657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35010dda0 msgr2=0x7fc3501a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35010dda0 0x7fc3501a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 -- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc350075820 msgr2=0x7fc35019e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc350075820 0x7fc35019e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 -- 172.21.15.7:0/452533657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3501a5cf0 con 0x7fc350074240 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.736+0000 7fc34ea00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350074240 0x7fc35019e0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc340000f80 tx=0x7fc340035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.737+0000 7fc354a00640 1 -- 172.21.15.7:0/452533657 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340039040 con 0x7fc350074240 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.737+0000 7fc354a00640 1 -- 172.21.15.7:0/452533657 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc340044070 con 0x7fc350074240 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.737+0000 7fc354a00640 1 -- 172.21.15.7:0/452533657 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3400044a0 con 0x7fc350074240 2024-03-18T01:35:15.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.737+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3501a5f80 con 0x7fc350074240 2024-03-18T01:35:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.737+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3501a6390 con 0x7fc350074240 2024-03-18T01:35:15.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.738+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc350103f70 con 0x7fc350074240 2024-03-18T01:35:15.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.740+0000 7fc354a00640 1 -- 172.21.15.7:0/452533657 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fc340004640 con 0x7fc350074240 2024-03-18T01:35:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.741+0000 7fc354a00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3340a2880 0x7fc3340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.742+0000 7fc354a00640 1 -- 172.21.15.7:0/452533657 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc3400f1e90 con 0x7fc350074240 2024-03-18T01:35:15.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.742+0000 7fc346000640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3340a2880 0x7fc3340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:15.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.742+0000 7fc346000640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3340a2880 0x7fc3340a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc35019f5e0 tx=0x7fc33800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:15.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.743+0000 7fc354a00640 1 -- 172.21.15.7:0/452533657 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc3400bec90 con 0x7fc350074240 2024-03-18T01:35:15.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.951+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc35002d400 con 0x7fc350074240 2024-03-18T01:35:15.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.953+0000 7fc354a00640 1 -- 172.21.15.7:0/452533657 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7fc3400098b0 con 0x7fc350074240 2024-03-18T01:35:15.954 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:15.954 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.956+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3340a2880 msgr2=0x7fc3340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.956+0000 7fc355e00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3340a2880 0x7fc3340a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc35019f5e0 tx=0x7fc33800b390 comp rx=0 tx=0).stop 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.956+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350074240 msgr2=0x7fc35019e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.956+0000 7fc355e00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350074240 0x7fc35019e0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc340000f80 tx=0x7fc340035d90 comp rx=0 tx=0).stop 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.956+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 shutdown_connections 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.957+0000 7fc355e00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc35010dda0 0x7fc3501a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.957+0000 7fc355e00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3340a2880 0x7fc3340a4d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.957+0000 7fc355e00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc350075820 0x7fc35019e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.957+0000 7fc355e00640 1 --2- 172.21.15.7:0/452533657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc350074240 0x7fc35019e0d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.957+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 >> 172.21.15.7:0/452533657 conn(0x7fc3500fcd40 msgr2=0x7fc350101140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:15.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.957+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 shutdown_connections 2024-03-18T01:35:15.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:15.957+0000 7fc355e00640 1 -- 172.21.15.7:0/452533657 wait complete. 2024-03-18T01:35:15.975 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:35:15.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:35:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 -- 172.21.15.7:0/2722194933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec104a10 msgr2=0x7f3fec104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/2722194933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec104a10 0x7f3fec104e90 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009900 tx=0x7f3fd802bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 -- 172.21.15.7:0/2722194933 shutdown_connections 2024-03-18T01:35:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/2722194933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fec1053d0 0x7f3fec10bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/2722194933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec104a10 0x7f3fec104e90 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/2722194933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fec103820 0x7f3fec103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 -- 172.21.15.7:0/2722194933 >> 172.21.15.7:0/2722194933 conn(0x7f3fec0fedb0 msgr2=0x7f3fec1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 -- 172.21.15.7:0/2722194933 shutdown_connections 2024-03-18T01:35:16.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.249+0000 7f3ff2e00640 1 -- 172.21.15.7:0/2722194933 wait complete. 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.250+0000 7f3ff2e00640 1 Processor -- start 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.250+0000 7f3ff2e00640 1 -- start start 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.250+0000 7f3ff2e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fec103820 0x7f3fec199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.250+0000 7f3ff2e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fec104a10 0x7f3fec19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.250+0000 7f3ff2e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec1053d0 0x7f3fec1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.250+0000 7f3ff2e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec10f480 con 0x7f3fec103820 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.250+0000 7f3ff2e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec10f300 con 0x7f3fec1053d0 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.250+0000 7f3ff2e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec10f600 con 0x7f3fec104a10 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fec103820 0x7f3fec199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fec103820 0x7f3fec199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42550/0 (socket says 172.21.15.7:42550) 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 -- 172.21.15.7:0/344318777 learned_addr learned my addr 172.21.15.7:0/344318777 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff2400640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec1053d0 0x7f3fec1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1000640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fec104a10 0x7f3fec19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:16.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 -- 172.21.15.7:0/344318777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fec104a10 msgr2=0x7f3fec19a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fec104a10 0x7f3fec19a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 -- 172.21.15.7:0/344318777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec1053d0 msgr2=0x7f3fec1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec1053d0 0x7f3fec1a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 -- 172.21.15.7:0/344318777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fec1a1770 con 0x7f3fec103820 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff2400640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec1053d0 0x7f3fec1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.251+0000 7f3ff1a00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fec103820 0x7f3fec199b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000e9c0 tx=0x7f3fe000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.252+0000 7f3fea000640 1 -- 172.21.15.7:0/344318777 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe000ccc0 con 0x7f3fec103820 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.252+0000 7f3fea000640 1 -- 172.21.15.7:0/344318777 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fe00035f0 con 0x7f3fec103820 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.252+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fec1a1a60 con 0x7f3fec103820 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.252+0000 7f3fea000640 1 -- 172.21.15.7:0/344318777 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe000ce60 con 0x7f3fec103820 2024-03-18T01:35:16.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.252+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fec1a1f80 con 0x7f3fec103820 2024-03-18T01:35:16.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.253+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fec04f540 con 0x7f3fec103820 2024-03-18T01:35:16.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.255+0000 7f3fea000640 1 -- 172.21.15.7:0/344318777 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f3fe00037a0 con 0x7f3fec103820 2024-03-18T01:35:16.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.257+0000 7f3fea000640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3fd40a2800 0x7f3fd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:16.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.257+0000 7f3fea000640 1 -- 172.21.15.7:0/344318777 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3fe00cf060 con 0x7f3fec103820 2024-03-18T01:35:16.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.257+0000 7f3ff1000640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3fd40a2800 0x7f3fd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:16.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.258+0000 7f3ff1000640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3fd40a2800 0x7f3fd40a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3fd802c7b0 tx=0x7f3fd8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:16.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.264+0000 7f3fea000640 1 -- 172.21.15.7:0/344318777 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3fe009be60 con 0x7f3fec103820 2024-03-18T01:35:16.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.467+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3fec062410 con 0x7f3fec103820 2024-03-18T01:35:16.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.468+0000 7f3fea000640 1 -- 172.21.15.7:0/344318777 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f3fe009b760 con 0x7f3fec103820 2024-03-18T01:35:16.468 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3fd40a2800 msgr2=0x7f3fd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3fd40a2800 0x7f3fd40a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3fd802c7b0 tx=0x7f3fd8005fd0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fec103820 msgr2=0x7f3fec199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fec103820 0x7f3fec199b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000e9c0 tx=0x7f3fe000edc0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 shutdown_connections 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fec1053d0 0x7f3fec1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3fec104a10 0x7f3fec19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3fd40a2800 0x7f3fd40a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 --2- 172.21.15.7:0/344318777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3fec103820 0x7f3fec199b50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 >> 172.21.15.7:0/344318777 conn(0x7f3fec0fedb0 msgr2=0x7f3fec107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 shutdown_connections 2024-03-18T01:35:16.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.471+0000 7f3ff2e00640 1 -- 172.21.15.7:0/344318777 wait complete. 2024-03-18T01:35:16.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:35:16.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:16 smithi046 ceph-mon[36108]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:16 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1786937750' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:35:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:16 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/452533657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:16.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:16 smithi007 ceph-mon[31996]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:16.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:16 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1786937750' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:35:16.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:16 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/452533657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:16 smithi203 ceph-mon[36345]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:16 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1786937750' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:35:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:16 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/452533657' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 -- 172.21.15.7:0/2416997794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c1097f0 msgr2=0x7f032c110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 --2- 172.21.15.7:0/2416997794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c1097f0 0x7f032c110080 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0324009be0 tx=0x7f032402bef0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 -- 172.21.15.7:0/2416997794 shutdown_connections 2024-03-18T01:35:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 --2- 172.21.15.7:0/2416997794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c1097f0 0x7f032c110080 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 --2- 172.21.15.7:0/2416997794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f032c108e30 0x7f032c1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 --2- 172.21.15.7:0/2416997794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f032c107c40 0x7f032c108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 -- 172.21.15.7:0/2416997794 >> 172.21.15.7:0/2416997794 conn(0x7f032c076bd0 msgr2=0x7f032c079010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:16.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 -- 172.21.15.7:0/2416997794 shutdown_connections 2024-03-18T01:35:16.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.720+0000 7f032be00640 1 -- 172.21.15.7:0/2416997794 wait complete. 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.721+0000 7f032be00640 1 Processor -- start 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.721+0000 7f032be00640 1 -- start start 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.721+0000 7f032be00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f032c107c40 0x7f032c19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.721+0000 7f032be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c108e30 0x7f032c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.721+0000 7f032be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f032c1097f0 0x7f032c1a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f032be00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f032c1138a0 con 0x7f032c107c40 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f032be00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f032c113720 con 0x7f032c108e30 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f032be00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f032c113a20 con 0x7f032c1097f0 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f032aa00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f032c107c40 0x7f032c19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c108e30 0x7f032c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c108e30 0x7f032c19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46146/0 (socket says 172.21.15.7:46146) 2024-03-18T01:35:16.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 -- 172.21.15.7:0/3368104780 learned_addr learned my addr 172.21.15.7:0/3368104780 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f032b400640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f032c1097f0 0x7f032c1a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 -- 172.21.15.7:0/3368104780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f032c1097f0 msgr2=0x7f032c1a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f032c1097f0 0x7f032c1a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 -- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f032c107c40 msgr2=0x7f032c19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f032c107c40 0x7f032c19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 -- 172.21.15.7:0/3368104780 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c1a5ad0 con 0x7f032c108e30 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f032aa00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f032c107c40 0x7f032c19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.722+0000 7f0323e00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c108e30 0x7f032c19e3f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f031400e9c0 tx=0x7f031400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.723+0000 7f0323400640 1 -- 172.21.15.7:0/3368104780 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031400cde0 con 0x7f032c108e30 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.723+0000 7f0323400640 1 -- 172.21.15.7:0/3368104780 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03140035f0 con 0x7f032c108e30 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.723+0000 7f0323400640 1 -- 172.21.15.7:0/3368104780 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0314010690 con 0x7f032c108e30 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.723+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032c1a5dc0 con 0x7f032c108e30 2024-03-18T01:35:16.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.723+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f032c1a6290 con 0x7f032c108e30 2024-03-18T01:35:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.724+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f032c10ef60 con 0x7f032c108e30 2024-03-18T01:35:16.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.726+0000 7f0323400640 1 -- 172.21.15.7:0/3368104780 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f03140037a0 con 0x7f032c108e30 2024-03-18T01:35:16.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.727+0000 7f0323400640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f03100a2880 0x7f03100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:16.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.727+0000 7f0323400640 1 -- 172.21.15.7:0/3368104780 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f03140cefb0 con 0x7f032c108e30 2024-03-18T01:35:16.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.728+0000 7f032aa00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f03100a2880 0x7f03100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:16.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.728+0000 7f032aa00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f03100a2880 0x7f03100a4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f032c108c90 tx=0x7f031c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:16.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.731+0000 7f0323400640 1 -- 172.21.15.7:0/3368104780 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f031409bdb0 con 0x7f032c108e30 2024-03-18T01:35:16.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.937+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f032c062480 con 0x7f032c108e30 2024-03-18T01:35:16.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.938+0000 7f0323400640 1 -- 172.21.15.7:0/3368104780 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f03140bb020 con 0x7f032c108e30 2024-03-18T01:35:16.938 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:35:16.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f03100a2880 msgr2=0x7f03100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f03100a2880 0x7f03100a4d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f032c108c90 tx=0x7f031c005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c108e30 msgr2=0x7f032c19e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c108e30 0x7f032c19e3f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f031400e9c0 tx=0x7f031400ee80 comp rx=0 tx=0).stop 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 shutdown_connections 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f032c1097f0 0x7f032c1a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f032c108e30 0x7f032c19e3f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f03100a2880 0x7f03100a4d40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 --2- 172.21.15.7:0/3368104780 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f032c107c40 0x7f032c19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 >> 172.21.15.7:0/3368104780 conn(0x7f032c076bd0 msgr2=0x7f032c10c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 shutdown_connections 2024-03-18T01:35:16.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:16.941+0000 7f032be00640 1 -- 172.21.15.7:0/3368104780 wait complete. 2024-03-18T01:35:16.959 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:35:17.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.184+0000 7f9801000640 1 -- 172.21.15.7:0/1024096554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc104a30 msgr2=0x7f97fc104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.184+0000 7f9801000640 1 --2- 172.21.15.7:0/1024096554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc104a30 0x7f97fc104eb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009900 tx=0x7f97e402bc50 comp rx=0 tx=0).stop 2024-03-18T01:35:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.184+0000 7f9801000640 1 -- 172.21.15.7:0/1024096554 shutdown_connections 2024-03-18T01:35:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.184+0000 7f9801000640 1 --2- 172.21.15.7:0/1024096554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc1053f0 0x7f97fc10bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.184+0000 7f9801000640 1 --2- 172.21.15.7:0/1024096554 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc104a30 0x7f97fc104eb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.184+0000 7f9801000640 1 --2- 172.21.15.7:0/1024096554 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc103840 0x7f97fc103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.184+0000 7f9801000640 1 -- 172.21.15.7:0/1024096554 >> 172.21.15.7:0/1024096554 conn(0x7f97fc0fedb0 msgr2=0x7f97fc1011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.185+0000 7f9801000640 1 -- 172.21.15.7:0/1024096554 shutdown_connections 2024-03-18T01:35:17.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.185+0000 7f9801000640 1 -- 172.21.15.7:0/1024096554 wait complete. 2024-03-18T01:35:17.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.185+0000 7f9801000640 1 Processor -- start 2024-03-18T01:35:17.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f9801000640 1 -- start start 2024-03-18T01:35:17.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f9801000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc103840 0x7f97fc199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f9801000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc104a30 0x7f97fc19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f9801000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc1053f0 0x7f97fc1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f9801000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc10f4c0 con 0x7f97fc104a30 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f9801000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc10f340 con 0x7f97fc103840 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f9801000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc10f640 con 0x7f97fc1053f0 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f97fb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc103840 0x7f97fc199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f97fb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc103840 0x7f97fc199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46164/0 (socket says 172.21.15.7:46164) 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f97fb400640 1 -- 172.21.15.7:0/1267509279 learned_addr learned my addr 172.21.15.7:0/1267509279 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f97f3e00640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc104a30 0x7f97fc19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.186+0000 7f97fbe00640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc1053f0 0x7f97fc1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97fb400640 1 -- 172.21.15.7:0/1267509279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc1053f0 msgr2=0x7f97fc1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97fb400640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc1053f0 0x7f97fc1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97fb400640 1 -- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc104a30 msgr2=0x7f97fc19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97fb400640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc104a30 0x7f97fc19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97fb400640 1 -- 172.21.15.7:0/1267509279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc1a1830 con 0x7f97fc103840 2024-03-18T01:35:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97fb400640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc103840 0x7f97fc199c20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00e9c0 tx=0x7f97ec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97f8c00640 1 -- 172.21.15.7:0/1267509279 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec00cd80 con 0x7f97fc103840 2024-03-18T01:35:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97f8c00640 1 -- 172.21.15.7:0/1267509279 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97ec0035f0 con 0x7f97fc103840 2024-03-18T01:35:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f97f8c00640 1 -- 172.21.15.7:0/1267509279 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec010690 con 0x7f97fc103840 2024-03-18T01:35:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97fc1a1b20 con 0x7f97fc103840 2024-03-18T01:35:17.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.187+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97fc1a2020 con 0x7f97fc103840 2024-03-18T01:35:17.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.188+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97fc10ac30 con 0x7f97fc103840 2024-03-18T01:35:17.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.190+0000 7f97f8c00640 1 -- 172.21.15.7:0/1267509279 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f97ec0037a0 con 0x7f97fc103840 2024-03-18T01:35:17.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.191+0000 7f97f8c00640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f97e00a2880 0x7f97e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:17.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.192+0000 7f97f8c00640 1 -- 172.21.15.7:0/1267509279 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f97ec0cf230 con 0x7f97fc103840 2024-03-18T01:35:17.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.192+0000 7f97f3e00640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f97e00a2880 0x7f97e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:17.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.192+0000 7f97f3e00640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f97e00a2880 0x7f97e00a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f97e4006010 tx=0x7f97e4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:17.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.195+0000 7f97f8c00640 1 -- 172.21.15.7:0/1267509279 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97ec09c030 con 0x7f97fc103840 2024-03-18T01:35:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.402+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f97fc062520 con 0x7f97fc103840 2024-03-18T01:35:17.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.403+0000 7f97f8c00640 1 -- 172.21.15.7:0/1267509279 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f97ec09b930 con 0x7f97fc103840 2024-03-18T01:35:17.403 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:35:17.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f97e00a2880 msgr2=0x7f97e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f97e00a2880 0x7f97e00a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f97e4006010 tx=0x7f97e4000bc0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc103840 msgr2=0x7f97fc199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc103840 0x7f97fc199c20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f97ec00e9c0 tx=0x7f97ec00ee80 comp rx=0 tx=0).stop 2024-03-18T01:35:17.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 shutdown_connections 2024-03-18T01:35:17.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97fc1053f0 0x7f97fc1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f97e00a2880 0x7f97e00a4d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f97fc104a30 0x7f97fc19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 --2- 172.21.15.7:0/1267509279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97fc103840 0x7f97fc199c20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.405+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 >> 172.21.15.7:0/1267509279 conn(0x7f97fc0fedb0 msgr2=0x7f97fc107e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:17.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.406+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 shutdown_connections 2024-03-18T01:35:17.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.406+0000 7f9801000640 1 -- 172.21.15.7:0/1267509279 wait complete. 2024-03-18T01:35:17.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:35:17.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:17 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/344318777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:35:17.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:17 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3368104780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:35:17.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:17 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/344318777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:35:17.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:17 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3368104780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:35:17.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:17 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/344318777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:35:17.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:17 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3368104780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 -- 172.21.15.7:0/2561383285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe654104c00 msgr2=0x7fe654105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 --2- 172.21.15.7:0/2561383285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe654104c00 0x7fe654105080 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe640009ca0 tx=0x7fe64002c100 comp rx=0 tx=0).stop 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 -- 172.21.15.7:0/2561383285 shutdown_connections 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 --2- 172.21.15.7:0/2561383285 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6541055c0 0x7fe65410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 --2- 172.21.15.7:0/2561383285 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe654104c00 0x7fe654105080 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 --2- 172.21.15.7:0/2561383285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe654103a10 0x7fe654103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 -- 172.21.15.7:0/2561383285 >> 172.21.15.7:0/2561383285 conn(0x7fe6540fefa0 msgr2=0x7fe6541013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 -- 172.21.15.7:0/2561383285 shutdown_connections 2024-03-18T01:35:17.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.642+0000 7fe65ae00640 1 -- 172.21.15.7:0/2561383285 wait complete. 2024-03-18T01:35:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.643+0000 7fe65ae00640 1 Processor -- start 2024-03-18T01:35:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.643+0000 7fe65ae00640 1 -- start start 2024-03-18T01:35:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe65ae00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe654103a10 0x7fe654199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe65ae00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe654104c00 0x7fe65419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe65ae00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6541055c0 0x7fe6541a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe65ae00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410f670 con 0x7fe654103a10 2024-03-18T01:35:17.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe65ae00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410f4f0 con 0x7fe6541055c0 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe65ae00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe65410f7f0 con 0x7fe654104c00 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe653e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe654103a10 0x7fe654199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe658e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6541055c0 0x7fe6541a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe653400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe654104c00 0x7fe65419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe658e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6541055c0 0x7fe6541a12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46178/0 (socket says 172.21.15.7:46178) 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe658e00640 1 -- 172.21.15.7:0/33799406 learned_addr learned my addr 172.21.15.7:0/33799406 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe658e00640 1 -- 172.21.15.7:0/33799406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe654104c00 msgr2=0x7fe65419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe658e00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe654104c00 0x7fe65419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe658e00640 1 -- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe654103a10 msgr2=0x7fe654199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe658e00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe654103a10 0x7fe654199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.644+0000 7fe658e00640 1 -- 172.21.15.7:0/33799406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6541a1940 con 0x7fe6541055c0 2024-03-18T01:35:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.645+0000 7fe658e00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6541055c0 0x7fe6541a12a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00e970 tx=0x7fe64c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.645+0000 7fe650c00640 1 -- 172.21.15.7:0/33799406 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c00ccc0 con 0x7fe6541055c0 2024-03-18T01:35:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.645+0000 7fe650c00640 1 -- 172.21.15.7:0/33799406 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe64c00ce60 con 0x7fe6541055c0 2024-03-18T01:35:17.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.645+0000 7fe650c00640 1 -- 172.21.15.7:0/33799406 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c010490 con 0x7fe6541055c0 2024-03-18T01:35:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.645+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6541a1c30 con 0x7fe6541055c0 2024-03-18T01:35:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.645+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6541a2130 con 0x7fe6541055c0 2024-03-18T01:35:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.647+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe65410ad30 con 0x7fe6541055c0 2024-03-18T01:35:17.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.648+0000 7fe650c00640 1 -- 172.21.15.7:0/33799406 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fe64c005020 con 0x7fe6541055c0 2024-03-18T01:35:17.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.649+0000 7fe650c00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6380a2880 0x7fe6380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:17.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.649+0000 7fe650c00640 1 -- 172.21.15.7:0/33799406 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe64c0cf170 con 0x7fe6541055c0 2024-03-18T01:35:17.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.650+0000 7fe653e00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6380a2880 0x7fe6380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:17.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.650+0000 7fe653e00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6380a2880 0x7fe6380a4d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe65419b200 tx=0x7fe644005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:17.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.653+0000 7fe650c00640 1 -- 172.21.15.7:0/33799406 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe64c09bf70 con 0x7fe6541055c0 2024-03-18T01:35:17.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.858+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe65402d1b0 con 0x7fe6541055c0 2024-03-18T01:35:17.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.859+0000 7fe650c00640 1 -- 172.21.15.7:0/33799406 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fe64c09b870 con 0x7fe6541055c0 2024-03-18T01:35:17.860 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:35:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6380a2880 msgr2=0x7fe6380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6380a2880 0x7fe6380a4d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe65419b200 tx=0x7fe644005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6541055c0 msgr2=0x7fe6541a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:17.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6541055c0 0x7fe6541a12a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00e970 tx=0x7fe64c00ed70 comp rx=0 tx=0).stop 2024-03-18T01:35:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 shutdown_connections 2024-03-18T01:35:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6541055c0 0x7fe6541a12a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe654104c00 0x7fe65419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6380a2880 0x7fe6380a4d40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 --2- 172.21.15.7:0/33799406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe654103a10 0x7fe654199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 >> 172.21.15.7:0/33799406 conn(0x7fe6540fefa0 msgr2=0x7fe654107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 shutdown_connections 2024-03-18T01:35:17.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:17.862+0000 7fe65ae00640 1 -- 172.21.15.7:0/33799406 wait complete. 2024-03-18T01:35:17.880 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T01:35:18.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 -- 172.21.15.7:0/3692647278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 msgr2=0x7faacc105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:18.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 --2- 172.21.15.7:0/3692647278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 0x7faacc105080 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7faab8009900 tx=0x7faab802bc40 comp rx=0 tx=0).stop 2024-03-18T01:35:18.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 -- 172.21.15.7:0/3692647278 shutdown_connections 2024-03-18T01:35:18.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 --2- 172.21.15.7:0/3692647278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faacc1055c0 0x7faacc10be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 --2- 172.21.15.7:0/3692647278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 0x7faacc105080 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 --2- 172.21.15.7:0/3692647278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faacc103a10 0x7faacc103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 -- 172.21.15.7:0/3692647278 >> 172.21.15.7:0/3692647278 conn(0x7faacc0fefa0 msgr2=0x7faacc1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:18.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 -- 172.21.15.7:0/3692647278 shutdown_connections 2024-03-18T01:35:18.106 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.105+0000 7faad4200640 1 -- 172.21.15.7:0/3692647278 wait complete. 2024-03-18T01:35:18.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.106+0000 7faad4200640 1 Processor -- start 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.106+0000 7faad4200640 1 -- start start 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad4200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faacc103a10 0x7faacc199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad4200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 0x7faacc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad4200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faacc1055c0 0x7faacc1a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad4200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7faacc10f6e0 con 0x7faacc103a10 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad4200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faacc10f560 con 0x7faacc104c00 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad4200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faacc10f860 con 0x7faacc1055c0 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad1800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faacc103a10 0x7faacc199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad1800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faacc103a10 0x7faacc199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42608/0 (socket says 172.21.15.7:42608) 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad1800640 1 -- 172.21.15.7:0/4249841086 learned_addr learned my addr 172.21.15.7:0/4249841086 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad0e00640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 0x7faacc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:18.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.107+0000 7faad2200640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faacc1055c0 0x7faacc1a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad1800640 1 -- 172.21.15.7:0/4249841086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faacc1055c0 msgr2=0x7faacc1a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad1800640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faacc1055c0 0x7faacc1a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad1800640 1 -- 172.21.15.7:0/4249841086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 msgr2=0x7faacc19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad1800640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 0x7faacc19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad1800640 1 -- 172.21.15.7:0/4249841086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faacc1a1950 con 0x7faacc103a10 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad0e00640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 0x7faacc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad1800640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faacc103a10 0x7faacc199d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faac000d950 tx=0x7faac000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faaca000640 1 -- 172.21.15.7:0/4249841086 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0017070 con 0x7faacc103a10 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faacc1a1c40 con 0x7faacc103a10 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faacc1a2190 con 0x7faacc103a10 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faaca000640 1 -- 172.21.15.7:0/4249841086 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faac0009c30 con 0x7faacc103a10 2024-03-18T01:35:18.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.108+0000 7faaca000640 1 -- 172.21.15.7:0/4249841086 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac000f600 con 0x7faacc103a10 2024-03-18T01:35:18.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.110+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaac0052c0 con 0x7faacc103a10 2024-03-18T01:35:18.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.112+0000 7faaca000640 1 -- 172.21.15.7:0/4249841086 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7faac001a070 con 0x7faacc103a10 2024-03-18T01:35:18.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.113+0000 7faaca000640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faab40a2880 0x7faab40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:18.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.113+0000 7faaca000640 1 -- 172.21.15.7:0/4249841086 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7faac00ce680 con 0x7faacc103a10 2024-03-18T01:35:18.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.113+0000 7faad0e00640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faab40a2880 0x7faab40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:18.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.114+0000 7faad0e00640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faab40a2880 0x7faab40a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faab802c250 tx=0x7faab802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:18.116 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.116+0000 7faaca000640 1 -- 172.21.15.7:0/4249841086 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faac0009de0 con 0x7faacc103a10 2024-03-18T01:35:18.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.333+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7faaac0055b0 con 0x7faacc103a10 2024-03-18T01:35:18.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.334+0000 7faaca000640 1 -- 172.21.15.7:0/4249841086 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+582 (secure 0 0 0) 0x7faac009aeb0 con 0x7faacc103a10 2024-03-18T01:35:18.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/1169030810 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:18.334 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3833186711 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/4234190466 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/1791220024 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/1169030810 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/1791220024 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3317891911 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2698516506 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2862047901 2024-03-19T01:26:21.850377+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3780594555 2024-03-19T01:26:57.023116+0000 2024-03-18T01:35:18.335 INFO:teuthology.orchestra.run.smithi007.stderr:listed 10 entries 2024-03-18T01:35:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faab40a2880 msgr2=0x7faab40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faab40a2880 0x7faab40a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faab802c250 tx=0x7faab802cf90 comp rx=0 tx=0).stop 2024-03-18T01:35:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faacc103a10 msgr2=0x7faacc199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:18.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faacc103a10 0x7faacc199d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faac000d950 tx=0x7faac000de10 comp rx=0 tx=0).stop 2024-03-18T01:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 shutdown_connections 2024-03-18T01:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faacc1055c0 0x7faacc1a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faacc104c00 0x7faacc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faab40a2880 0x7faab40a4d40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 --2- 172.21.15.7:0/4249841086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faacc103a10 0x7faacc199d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 >> 172.21.15.7:0/4249841086 conn(0x7faacc0fefa0 msgr2=0x7faacc107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 shutdown_connections 2024-03-18T01:35:18.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:18.337+0000 7faad4200640 1 -- 172.21.15.7:0/4249841086 wait complete. 2024-03-18T01:35:18.357 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi007.front.sepia.ceph.com... 2024-03-18T01:35:18.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:18.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-18T01:35:18.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi007.stderr:ceph-fuse[76004]: fuse finished with error 0 and tester_r 0 2024-03-18T01:35:18.457 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T01:35:18.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:18 smithi046 ceph-mon[36108]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:18 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1267509279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:35:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:18 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/33799406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:35:18.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:18 smithi007 ceph-mon[31996]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:18.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:18 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1267509279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:35:18.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:18 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/33799406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:35:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:18 smithi203 ceph-mon[36345]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:18 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1267509279' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:35:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:18 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/33799406' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:35:19.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:19 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4249841086' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:35:19.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:19 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4249841086' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:35:19.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:19 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/4249841086' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:35:20.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:20 smithi007 ceph-mon[31996]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:20 smithi203 ceph-mon[36345]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:20.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:20 smithi046 ceph-mon[36108]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:22.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:22 smithi007 ceph-mon[31996]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:22.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:22 smithi203 ceph-mon[36345]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:22.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:22 smithi046 ceph-mon[36108]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T01:35:24.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:24 smithi007 ceph-mon[31996]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T01:35:24.559 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi007.front.sepia.ceph.com 2024-03-18T01:35:24.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:35:24.559 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-18T01:35:24.636 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:35:24.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:24 smithi203 ceph-mon[36345]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T01:35:24.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:24 smithi046 ceph-mon[36108]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T01:35:24.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.864+0000 7f329ae00640 1 -- 172.21.15.7:0/485434261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 msgr2=0x7f329410be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:24.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.864+0000 7f329ae00640 1 --2- 172.21.15.7:0/485434261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 0x7f329410be70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f328c009ca0 tx=0x7f328c02c100 comp rx=0 tx=0).stop 2024-03-18T01:35:24.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.865+0000 7f329ae00640 1 -- 172.21.15.7:0/485434261 shutdown_connections 2024-03-18T01:35:24.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.865+0000 7f329ae00640 1 --2- 172.21.15.7:0/485434261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 0x7f329410be70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:24.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.865+0000 7f329ae00640 1 --2- 172.21.15.7:0/485434261 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3294104c20 0x7f32941050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:24.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.865+0000 7f329ae00640 1 --2- 172.21.15.7:0/485434261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3294103a30 0x7f3294103e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:24.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.865+0000 7f329ae00640 1 -- 172.21.15.7:0/485434261 >> 172.21.15.7:0/485434261 conn(0x7f32940fefa0 msgr2=0x7f32941013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:24.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.865+0000 7f329ae00640 1 -- 172.21.15.7:0/485434261 shutdown_connections 2024-03-18T01:35:24.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.865+0000 7f329ae00640 1 -- 172.21.15.7:0/485434261 wait complete. 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.866+0000 7f329ae00640 1 Processor -- start 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.866+0000 7f329ae00640 1 -- start start 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.866+0000 7f329ae00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3294103a30 0x7f3294199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.866+0000 7f329ae00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3294104c20 0x7f329419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.866+0000 7f329ae00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 0x7f32941a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.866+0000 7f329ae00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f329410f690 con 0x7f3294104c20 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.866+0000 7f329ae00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f329410f510 con 0x7f32941055e0 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.866+0000 7f329ae00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f329410f810 con 0x7f3294103a30 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3293400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3294104c20 0x7f329419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3298e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 0x7f32941a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:24.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3293400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3294104c20 0x7f329419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:42632/0 (socket says 172.21.15.7:42632) 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3293400640 1 -- 172.21.15.7:0/3503318133 learned_addr learned my addr 172.21.15.7:0/3503318133 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3293e00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3294103a30 0x7f3294199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3298e00640 1 -- 172.21.15.7:0/3503318133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3294103a30 msgr2=0x7f3294199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3298e00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3294103a30 0x7f3294199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3298e00640 1 -- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3294104c20 msgr2=0x7f329419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3298e00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3294104c20 0x7f329419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3298e00640 1 -- 172.21.15.7:0/3503318133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32941a1880 con 0x7f32941055e0 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3293400640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3294104c20 0x7f329419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.867+0000 7f3298e00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 0x7f32941a11e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f328c037040 tx=0x7f328c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.868+0000 7f3290c00640 1 -- 172.21.15.7:0/3503318133 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c039040 con 0x7f32941055e0 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.868+0000 7f3290c00640 1 -- 172.21.15.7:0/3503318133 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f328c044070 con 0x7f32941055e0 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.868+0000 7f3290c00640 1 -- 172.21.15.7:0/3503318133 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c004450 con 0x7f32941055e0 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.868+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32941a1b10 con 0x7f32941055e0 2024-03-18T01:35:24.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.868+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32941a2020 con 0x7f32941055e0 2024-03-18T01:35:24.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.869+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329410acf0 con 0x7f32941055e0 2024-03-18T01:35:24.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.871+0000 7f3290c00640 1 -- 172.21.15.7:0/3503318133 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f328c0045f0 con 0x7f32941055e0 2024-03-18T01:35:24.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.872+0000 7f3290c00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32780a2880 0x7f32780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:24.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.872+0000 7f3290c00640 1 -- 172.21.15.7:0/3503318133 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f328c0f1280 con 0x7f32941055e0 2024-03-18T01:35:24.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.872+0000 7f3293e00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32780a2880 0x7f32780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:24.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.873+0000 7f3293e00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32780a2880 0x7f32780a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f329419b140 tx=0x7f3284005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:24.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:24.881+0000 7f3290c00640 1 -- 172.21.15.7:0/3503318133 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f328c0be080 con 0x7f32941055e0 2024-03-18T01:35:25.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.089+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32940623e0 con 0x7f32941055e0 2024-03-18T01:35:25.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.091+0000 7f3290c00640 1 -- 172.21.15.7:0/3503318133 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7f328c0098b0 con 0x7f32941055e0 2024-03-18T01:35:25.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:25.092 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:25.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32780a2880 msgr2=0x7f32780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32780a2880 0x7f32780a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f329419b140 tx=0x7f3284005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:25.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 msgr2=0x7f32941a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 0x7f32941a11e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f328c037040 tx=0x7f328c035d10 comp rx=0 tx=0).stop 2024-03-18T01:35:25.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 shutdown_connections 2024-03-18T01:35:25.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32941055e0 0x7f32941a11e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32780a2880 0x7f32780a4d40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3294104c20 0x7f329419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 --2- 172.21.15.7:0/3503318133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3294103a30 0x7f3294199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 >> 172.21.15.7:0/3503318133 conn(0x7f32940fefa0 msgr2=0x7f3294107ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:25.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 shutdown_connections 2024-03-18T01:35:25.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.094+0000 7f329ae00640 1 -- 172.21.15.7:0/3503318133 wait complete. 2024-03-18T01:35:25.116 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:35:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 -- 172.21.15.7:0/3162735405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 msgr2=0x7fe3b4108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 --2- 172.21.15.7:0/3162735405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 0x7fe3b4108260 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009900 tx=0x7fe3a802bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 -- 172.21.15.7:0/3162735405 shutdown_connections 2024-03-18T01:35:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 --2- 172.21.15.7:0/3162735405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b4109950 0x7fe3b4110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 --2- 172.21.15.7:0/3162735405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3b4108f90 0x7fe3b4109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 --2- 172.21.15.7:0/3162735405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 0x7fe3b4108260 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 -- 172.21.15.7:0/3162735405 >> 172.21.15.7:0/3162735405 conn(0x7fe3b4076dd0 msgr2=0x7fe3b4079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 -- 172.21.15.7:0/3162735405 shutdown_connections 2024-03-18T01:35:25.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.336+0000 7fe3bc400640 1 -- 172.21.15.7:0/3162735405 wait complete. 2024-03-18T01:35:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.337+0000 7fe3bc400640 1 Processor -- start 2024-03-18T01:35:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3bc400640 1 -- start start 2024-03-18T01:35:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3bc400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 0x7fe3b419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:25.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3bc400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3b4108f90 0x7fe3b419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3bc400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b4109950 0x7fe3b41a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3bc400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b4113ae0 con 0x7fe3b4109950 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3bc400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b4113960 con 0x7fe3b4107e40 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3bc400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b4113c60 con 0x7fe3b4108f90 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3b9a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 0x7fe3b419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3b9000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3b4108f90 0x7fe3b419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3ba400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b4109950 0x7fe3b41a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3b9a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 0x7fe3b419e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46238/0 (socket says 172.21.15.7:46238) 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3b9000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3b4108f90 0x7fe3b419e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36338/0 (socket says 172.21.15.7:36338) 2024-03-18T01:35:25.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.338+0000 7fe3b9a00640 1 -- 172.21.15.7:0/850402694 learned_addr learned my addr 172.21.15.7:0/850402694 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b9000640 1 -- 172.21.15.7:0/850402694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 msgr2=0x7fe3b419e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b9000640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 0x7fe3b419e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b9000640 1 -- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b4109950 msgr2=0x7fe3b41a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b9000640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b4109950 0x7fe3b41a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b9000640 1 -- 172.21.15.7:0/850402694 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b41a5d70 con 0x7fe3b4108f90 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b9a00640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 0x7fe3b419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b9000640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3b4108f90 0x7fe3b419e690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe3a000e9c0 tx=0x7fe3a000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b2000640 1 -- 172.21.15.7:0/850402694 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a000ccc0 con 0x7fe3b4108f90 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b2000640 1 -- 172.21.15.7:0/850402694 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3a00035f0 con 0x7fe3b4108f90 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3b41a6060 con 0x7fe3b4108f90 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.339+0000 7fe3b2000640 1 -- 172.21.15.7:0/850402694 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a000ce60 con 0x7fe3b4108f90 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.340+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3b41a6560 con 0x7fe3b4108f90 2024-03-18T01:35:25.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.341+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3b410f150 con 0x7fe3b4108f90 2024-03-18T01:35:25.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.343+0000 7fe3b2000640 1 -- 172.21.15.7:0/850402694 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fe3a00037a0 con 0x7fe3b4108f90 2024-03-18T01:35:25.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.344+0000 7fe3b2000640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe39c0a2800 0x7fe39c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:25.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.345+0000 7fe3b2000640 1 -- 172.21.15.7:0/850402694 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe3a00cefe0 con 0x7fe3b4108f90 2024-03-18T01:35:25.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.345+0000 7fe3b9a00640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe39c0a2800 0x7fe39c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:25.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.345+0000 7fe3b9a00640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe39c0a2800 0x7fe39c0a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8000c00 tx=0x7fe3a802c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:25.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.349+0000 7fe3b2000640 1 -- 172.21.15.7:0/850402694 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3a009be00 con 0x7fe3b4108f90 2024-03-18T01:35:25.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:25 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3503318133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:25.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.553+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3b40623e0 con 0x7fe3b4108f90 2024-03-18T01:35:25.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.555+0000 7fe3b2000640 1 -- 172.21.15.7:0/850402694 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7fe3a009b700 con 0x7fe3b4108f90 2024-03-18T01:35:25.556 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:25.556 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:25.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe39c0a2800 msgr2=0x7fe39c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe39c0a2800 0x7fe39c0a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8000c00 tx=0x7fe3a802c880 comp rx=0 tx=0).stop 2024-03-18T01:35:25.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3b4108f90 msgr2=0x7fe3b419e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3b4108f90 0x7fe3b419e690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe3a000e9c0 tx=0x7fe3a000edc0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 shutdown_connections 2024-03-18T01:35:25.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe39c0a2800 0x7fe39c0a4cc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe3b4109950 0x7fe3b41a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3b4108f90 0x7fe3b419e690 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.558+0000 7fe3bc400640 1 --2- 172.21.15.7:0/850402694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3b4107e40 0x7fe3b419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.559+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 >> 172.21.15.7:0/850402694 conn(0x7fe3b4076dd0 msgr2=0x7fe3b410c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:25.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.559+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 shutdown_connections 2024-03-18T01:35:25.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.559+0000 7fe3bc400640 1 -- 172.21.15.7:0/850402694 wait complete. 2024-03-18T01:35:25.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T01:35:25.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:25 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3503318133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:25.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:25 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3503318133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 -- 172.21.15.7:0/3750183652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52bc109930 msgr2=0x7f52bc110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 --2- 172.21.15.7:0/3750183652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52bc109930 0x7f52bc110250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f52b4009900 tx=0x7f52b402bc50 comp rx=0 tx=0).stop 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 -- 172.21.15.7:0/3750183652 shutdown_connections 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 --2- 172.21.15.7:0/3750183652 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52bc109930 0x7f52bc110250 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 --2- 172.21.15.7:0/3750183652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52bc108f70 0x7f52bc1093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 --2- 172.21.15.7:0/3750183652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52bc107e20 0x7f52bc108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 -- 172.21.15.7:0/3750183652 >> 172.21.15.7:0/3750183652 conn(0x7f52bc076dd0 msgr2=0x7f52bc0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 -- 172.21.15.7:0/3750183652 shutdown_connections 2024-03-18T01:35:25.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.822+0000 7f52c4000640 1 -- 172.21.15.7:0/3750183652 wait complete. 2024-03-18T01:35:25.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.823+0000 7f52c4000640 1 Processor -- start 2024-03-18T01:35:25.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.823+0000 7f52c4000640 1 -- start start 2024-03-18T01:35:25.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.823+0000 7f52c4000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52bc107e20 0x7f52bc19e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:25.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.823+0000 7f52c4000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52bc108f70 0x7f52bc19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:25.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.823+0000 7f52c4000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52bc109930 0x7f52bc1a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.823+0000 7f52c4000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc113b10 con 0x7f52bc108f70 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.823+0000 7f52c4000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc113990 con 0x7f52bc109930 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.823+0000 7f52c4000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc113c90 con 0x7f52bc107e20 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52bc109930 0x7f52bc1a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52bc109930 0x7f52bc1a56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46256/0 (socket says 172.21.15.7:46256) 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 -- 172.21.15.7:0/156360947 learned_addr learned my addr 172.21.15.7:0/156360947 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c1600640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52bc107e20 0x7f52bc19e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c0c00640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52bc108f70 0x7f52bc19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 -- 172.21.15.7:0/156360947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52bc107e20 msgr2=0x7f52bc19e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52bc107e20 0x7f52bc19e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 -- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52bc108f70 msgr2=0x7f52bc19e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:25.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52bc108f70 0x7f52bc19e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 -- 172.21.15.7:0/156360947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52bc1a5d10 con 0x7f52bc109930 2024-03-18T01:35:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52c2000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52bc109930 0x7f52bc1a56a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f52b4036040 tx=0x7f52b402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.824+0000 7f52ba000640 1 -- 172.21.15.7:0/156360947 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4041070 con 0x7f52bc109930 2024-03-18T01:35:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.825+0000 7f52ba000640 1 -- 172.21.15.7:0/156360947 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52b4004450 con 0x7f52bc109930 2024-03-18T01:35:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.825+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52bc1a5fa0 con 0x7f52bc109930 2024-03-18T01:35:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.825+0000 7f52ba000640 1 -- 172.21.15.7:0/156360947 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4034440 con 0x7f52bc109930 2024-03-18T01:35:25.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.825+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52bc1a6460 con 0x7f52bc109930 2024-03-18T01:35:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.826+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52bc10f0d0 con 0x7f52bc109930 2024-03-18T01:35:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.828+0000 7f52ba000640 1 -- 172.21.15.7:0/156360947 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f52b4044070 con 0x7f52bc109930 2024-03-18T01:35:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.829+0000 7f52ba000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52a00a2800 0x7f52a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.829+0000 7f52ba000640 1 -- 172.21.15.7:0/156360947 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f52b40f0d10 con 0x7f52bc109930 2024-03-18T01:35:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.829+0000 7f52c1600640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52a00a2800 0x7f52a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.830+0000 7f52c1600640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52a00a2800 0x7f52a00a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009b60 tx=0x7f52ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:25.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:25.832+0000 7f52ba000640 1 -- 172.21.15.7:0/156360947 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f52b40bdb10 con 0x7f52bc109930 2024-03-18T01:35:26.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.075+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f52bc02d1b0 con 0x7f52bc109930 2024-03-18T01:35:26.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.312+0000 7f52ba000640 1 -- 172.21.15.7:0/156360947 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f52b40bd410 con 0x7f52bc109930 2024-03-18T01:35:26.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52a00a2800 msgr2=0x7f52a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:26.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52a00a2800 0x7f52a00a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009b60 tx=0x7f52ac005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:26.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52bc109930 msgr2=0x7f52bc1a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52bc109930 0x7f52bc1a56a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f52b4036040 tx=0x7f52b402c650 comp rx=0 tx=0).stop 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 shutdown_connections 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52bc109930 0x7f52bc1a56a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52a00a2800 0x7f52a00a4cc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52bc108f70 0x7f52bc19e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 --2- 172.21.15.7:0/156360947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52bc107e20 0x7f52bc19e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.317+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 >> 172.21.15.7:0/156360947 conn(0x7f52bc076dd0 msgr2=0x7f52bc10c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.318+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 shutdown_connections 2024-03-18T01:35:26.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.318+0000 7f52c4000640 1 -- 172.21.15.7:0/156360947 wait complete. 2024-03-18T01:35:26.345 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-18T01:35:26.345 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-18T01:35:26.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:26 smithi007 ceph-mon[31996]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T01:35:26.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:26 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/850402694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:26.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:26 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/156360947' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T01:35:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:26 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T01:35:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.570+0000 7f8733e00640 1 -- 172.21.15.7:0/176160356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c1018d0 msgr2=0x7f872c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.570+0000 7f8733e00640 1 --2- 172.21.15.7:0/176160356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c1018d0 0x7f872c101d50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8718009900 tx=0x7f871802bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.571+0000 7f8733e00640 1 -- 172.21.15.7:0/176160356 shutdown_connections 2024-03-18T01:35:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.571+0000 7f8733e00640 1 --2- 172.21.15.7:0/176160356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f872c102290 0x7f872c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.571+0000 7f8733e00640 1 --2- 172.21.15.7:0/176160356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c1018d0 0x7f872c101d50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.571+0000 7f8733e00640 1 --2- 172.21.15.7:0/176160356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f872c1006e0 0x7f872c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.571+0000 7f8733e00640 1 -- 172.21.15.7:0/176160356 >> 172.21.15.7:0/176160356 conn(0x7f872c0fbc70 msgr2=0x7f872c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:26.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.571+0000 7f8733e00640 1 -- 172.21.15.7:0/176160356 shutdown_connections 2024-03-18T01:35:26.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.571+0000 7f8733e00640 1 -- 172.21.15.7:0/176160356 wait complete. 2024-03-18T01:35:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.572+0000 7f8733e00640 1 Processor -- start 2024-03-18T01:35:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.572+0000 7f8733e00640 1 -- start start 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.572+0000 7f8733e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f872c1006e0 0x7f872c199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.572+0000 7f8733e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f872c1018d0 0x7f872c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.572+0000 7f8733e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c102290 0x7f872c1a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.572+0000 7f8733e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f872c10c360 con 0x7f872c1018d0 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.572+0000 7f8733e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f872c10c1e0 con 0x7f872c102290 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.572+0000 7f8733e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f872c10c4e0 con 0x7f872c1006e0 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8730a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f872c1018d0 0x7f872c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f872c1006e0 0x7f872c199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8730a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f872c1018d0 0x7f872c19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35916/0 (socket says 172.21.15.7:35916) 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8730a00640 1 -- 172.21.15.7:0/2238023949 learned_addr learned my addr 172.21.15.7:0/2238023949 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c102290 0x7f872c1a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731e00640 1 -- 172.21.15.7:0/2238023949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f872c1006e0 msgr2=0x7f872c199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:26.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f872c1006e0 0x7f872c199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731e00640 1 -- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f872c1018d0 msgr2=0x7f872c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f872c1018d0 0x7f872c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731e00640 1 -- 172.21.15.7:0/2238023949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872c1a1960 con 0x7f872c102290 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731400640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f872c1006e0 0x7f872c199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.573+0000 7f8731e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c102290 0x7f872c1a12c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f872400e970 tx=0x7f872400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.574+0000 7f872a000640 1 -- 172.21.15.7:0/2238023949 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872400ccc0 con 0x7f872c102290 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.574+0000 7f872a000640 1 -- 172.21.15.7:0/2238023949 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8724005020 con 0x7f872c102290 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.574+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872c1a1c50 con 0x7f872c102290 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.574+0000 7f872a000640 1 -- 172.21.15.7:0/2238023949 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724004a00 con 0x7f872c102290 2024-03-18T01:35:26.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.574+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872c1a21a0 con 0x7f872c102290 2024-03-18T01:35:26.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.575+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f872c1a2370 con 0x7f872c102290 2024-03-18T01:35:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.576+0000 7f872a000640 1 -- 172.21.15.7:0/2238023949 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f87240051d0 con 0x7f872c102290 2024-03-18T01:35:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.578+0000 7f872a000640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87140a2800 0x7f87140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.578+0000 7f872a000640 1 -- 172.21.15.7:0/2238023949 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f87240cecd0 con 0x7f872c102290 2024-03-18T01:35:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.578+0000 7f8731400640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87140a2800 0x7f87140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.579+0000 7f8731400640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87140a2800 0x7f87140a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f872c19b220 tx=0x7f8720009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:26.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.587+0000 7f872a000640 1 -- 172.21.15.7:0/2238023949 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f872409bad0 con 0x7f872c102290 2024-03-18T01:35:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:26 smithi203 ceph-mon[36345]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T01:35:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:26 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/850402694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:26 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/156360947' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T01:35:26.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:26 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T01:35:26.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:26 smithi046 ceph-mon[36108]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T01:35:26.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:26 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/850402694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:26.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:26 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/156360947' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T01:35:26.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:26 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T01:35:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:26.833+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f872c1a2540 con 0x7f872c102290 2024-03-18T01:35:27.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.316+0000 7f872a000640 1 -- 172.21.15.7:0/2238023949 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f872409b3d0 con 0x7f872c102290 2024-03-18T01:35:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.320+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87140a2800 msgr2=0x7f87140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.320+0000 7f8733e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87140a2800 0x7f87140a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f872c19b220 tx=0x7f8720009500 comp rx=0 tx=0).stop 2024-03-18T01:35:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.320+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c102290 msgr2=0x7f872c1a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.320+0000 7f8733e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c102290 0x7f872c1a12c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f872400e970 tx=0x7f872400ee30 comp rx=0 tx=0).stop 2024-03-18T01:35:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.321+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 shutdown_connections 2024-03-18T01:35:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.321+0000 7f8733e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f872c102290 0x7f872c1a12c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.321+0000 7f8733e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f87140a2800 0x7f87140a4cc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.321+0000 7f8733e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f872c1018d0 0x7f872c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.321+0000 7f8733e00640 1 --2- 172.21.15.7:0/2238023949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f872c1006e0 0x7f872c199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.321+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 >> 172.21.15.7:0/2238023949 conn(0x7f872c0fbc70 msgr2=0x7f872c0fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.321+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 shutdown_connections 2024-03-18T01:35:27.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.321+0000 7f8733e00640 1 -- 172.21.15.7:0/2238023949 wait complete. 2024-03-18T01:35:27.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 -- 172.21.15.7:0/628004183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 msgr2=0x7fe9dc105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 --2- 172.21.15.7:0/628004183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 0x7fe9dc105090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009900 tx=0x7fe9c802bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 -- 172.21.15.7:0/628004183 shutdown_connections 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 --2- 172.21.15.7:0/628004183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9dc1055d0 0x7fe9dc10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 --2- 172.21.15.7:0/628004183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 0x7fe9dc105090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 --2- 172.21.15.7:0/628004183 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9dc103a20 0x7fe9dc103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 -- 172.21.15.7:0/628004183 >> 172.21.15.7:0/628004183 conn(0x7fe9dc0fef90 msgr2=0x7fe9dc1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 -- 172.21.15.7:0/628004183 shutdown_connections 2024-03-18T01:35:27.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.549+0000 7fe9e2800640 1 -- 172.21.15.7:0/628004183 wait complete. 2024-03-18T01:35:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.550+0000 7fe9e2800640 1 Processor -- start 2024-03-18T01:35:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9e2800640 1 -- start start 2024-03-18T01:35:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9e2800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9dc103a20 0x7fe9dc079590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9e2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 0x7fe9dc079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9e2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9dc1055d0 0x7fe9dc076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:27.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9e2800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe9dc10f6f0 con 0x7fe9dc103a20 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9e2800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe9dc10f570 con 0x7fe9dc104c10 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9e2800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9dc10f870 con 0x7fe9dc1055d0 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9db400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9dc103a20 0x7fe9dc079590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9daa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 0x7fe9dc079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.551+0000 7fe9daa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 0x7fe9dc079ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54400/0 (socket says 172.21.15.7:54400) 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9daa00640 1 -- 172.21.15.7:0/2748646546 learned_addr learned my addr 172.21.15.7:0/2748646546 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9dbe00640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9dc1055d0 0x7fe9dc076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9daa00640 1 -- 172.21.15.7:0/2748646546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9dc1055d0 msgr2=0x7fe9dc076100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9daa00640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9dc1055d0 0x7fe9dc076100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9daa00640 1 -- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9dc103a20 msgr2=0x7fe9dc079590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9daa00640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9dc103a20 0x7fe9dc079590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9daa00640 1 -- 172.21.15.7:0/2748646546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9dc0769c0 con 0x7fe9dc104c10 2024-03-18T01:35:27.554 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9dbe00640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9dc1055d0 0x7fe9dc076100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:27.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.552+0000 7fe9daa00640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 0x7fe9dc079ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8005ab0 tx=0x7fe9c802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:27.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.553+0000 7fe9d3e00640 1 -- 172.21.15.7:0/2748646546 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8043020 con 0x7fe9dc104c10 2024-03-18T01:35:27.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.553+0000 7fe9d3e00640 1 -- 172.21.15.7:0/2748646546 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c80054e0 con 0x7fe9dc104c10 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.553+0000 7fe9d3e00640 1 -- 172.21.15.7:0/2748646546 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8003ce0 con 0x7fe9dc104c10 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.553+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9dc1a6150 con 0x7fe9dc104c10 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.553+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9dc1a6500 con 0x7fe9dc104c10 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.554+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9dc10ace0 con 0x7fe9dc104c10 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.556+0000 7fe9d3e00640 1 -- 172.21.15.7:0/2748646546 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fe9c8046020 con 0x7fe9dc104c10 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.557+0000 7fe9d3e00640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe9c00a2880 0x7fe9c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.558+0000 7fe9d3e00640 1 -- 172.21.15.7:0/2748646546 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe9c80f0e30 con 0x7fe9dc104c10 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.558+0000 7fe9db400640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe9c00a2880 0x7fe9c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.559+0000 7fe9db400640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe9c00a2880 0x7fe9c00a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc005fd0 tx=0x7fe9cc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:27.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.561+0000 7fe9d3e00640 1 -- 172.21.15.7:0/2748646546 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe9c804d020 con 0x7fe9dc104c10 2024-03-18T01:35:27.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:27 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T01:35:27.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:27 smithi203 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:27.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:27 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2238023949' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T01:35:27.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:27 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T01:35:27.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:27 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T01:35:27.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:27 smithi046 ceph-mon[36108]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:27.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:27 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2238023949' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T01:35:27.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:27 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T01:35:27.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:27 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T01:35:27.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:27 smithi007 ceph-mon[31996]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:27.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:27 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2238023949' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T01:35:27.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:27 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T01:35:27.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:27.806+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fe9dc02d1b0 con 0x7fe9dc104c10 2024-03-18T01:35:28.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.329+0000 7fe9d3e00640 1 -- 172.21.15.7:0/2748646546 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fe9c80bdc30 con 0x7fe9dc104c10 2024-03-18T01:35:28.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.335+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe9c00a2880 msgr2=0x7fe9c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:28.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.335+0000 7fe9e2800640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe9c00a2880 0x7fe9c00a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc005fd0 tx=0x7fe9cc00b410 comp rx=0 tx=0).stop 2024-03-18T01:35:28.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 msgr2=0x7fe9dc079ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:28.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 0x7fe9dc079ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8005ab0 tx=0x7fe9c802c6c0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 shutdown_connections 2024-03-18T01:35:28.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9dc1055d0 0x7fe9dc076100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9dc104c10 0x7fe9dc079ad0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe9c00a2880 0x7fe9c00a4d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 --2- 172.21.15.7:0/2748646546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe9dc103a20 0x7fe9dc079590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 >> 172.21.15.7:0/2748646546 conn(0x7fe9dc0fef90 msgr2=0x7fe9dc107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:28.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 shutdown_connections 2024-03-18T01:35:28.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.336+0000 7fe9e2800640 1 -- 172.21.15.7:0/2748646546 wait complete. 2024-03-18T01:35:28.367 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T01:35:28.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 -- 172.21.15.7:0/629923568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1049e0 msgr2=0x7f890c106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 --2- 172.21.15.7:0/629923568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1049e0 0x7f890c106dd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009900 tx=0x7f88f402bc40 comp rx=0 tx=0).stop 2024-03-18T01:35:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 -- 172.21.15.7:0/629923568 shutdown_connections 2024-03-18T01:35:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 --2- 172.21.15.7:0/629923568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f890c107310 0x7f890c109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 --2- 172.21.15.7:0/629923568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1049e0 0x7f890c106dd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 --2- 172.21.15.7:0/629923568 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c1020b0 0x7f890c1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 -- 172.21.15.7:0/629923568 >> 172.21.15.7:0/629923568 conn(0x7f890c0fbae0 msgr2=0x7f890c0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 -- 172.21.15.7:0/629923568 shutdown_connections 2024-03-18T01:35:28.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.571+0000 7f890b200640 1 -- 172.21.15.7:0/629923568 wait complete. 2024-03-18T01:35:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.572+0000 7f890b200640 1 Processor -- start 2024-03-18T01:35:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.572+0000 7f890b200640 1 -- start start 2024-03-18T01:35:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f890b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1020b0 0x7f890c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f890b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f890c1049e0 0x7f890c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f890b200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c107310 0x7f890c1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f890b200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f890c10d290 con 0x7f890c107310 2024-03-18T01:35:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f890b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f890c10d110 con 0x7f890c1020b0 2024-03-18T01:35:28.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f890b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f890c10d410 con 0x7f890c1049e0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8909e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1020b0 0x7f890c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8909e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1020b0 0x7f890c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54426/0 (socket says 172.21.15.7:54426) 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8909e00640 1 -- 172.21.15.7:0/622918349 learned_addr learned my addr 172.21.15.7:0/622918349 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8901400640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f890c1049e0 0x7f890c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f890a800640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c107310 0x7f890c1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8909e00640 1 -- 172.21.15.7:0/622918349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f890c1049e0 msgr2=0x7f890c19a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8909e00640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f890c1049e0 0x7f890c19a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8909e00640 1 -- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c107310 msgr2=0x7f890c1a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8909e00640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c107310 0x7f890c1a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.573+0000 7f8909e00640 1 -- 172.21.15.7:0/622918349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890c1a1780 con 0x7f890c1020b0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.574+0000 7f8909e00640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1020b0 0x7f890c199b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00ebc0 tx=0x7f88fc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.574+0000 7f8903400640 1 -- 172.21.15.7:0/622918349 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88fc010630 con 0x7f890c1020b0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.574+0000 7f8903400640 1 -- 172.21.15.7:0/622918349 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88fc010cb0 con 0x7f890c1020b0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.574+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f890c1a1a70 con 0x7f890c1020b0 2024-03-18T01:35:28.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.574+0000 7f8903400640 1 -- 172.21.15.7:0/622918349 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88fc018bd0 con 0x7f890c1020b0 2024-03-18T01:35:28.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.574+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f890c1a1f90 con 0x7f890c1020b0 2024-03-18T01:35:28.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.575+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f890c108e60 con 0x7f890c1020b0 2024-03-18T01:35:28.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.577+0000 7f8903400640 1 -- 172.21.15.7:0/622918349 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7f88fc0107d0 con 0x7f890c1020b0 2024-03-18T01:35:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.578+0000 7f8903400640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f88f00a2800 0x7f88f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.578+0000 7f8903400640 1 -- 172.21.15.7:0/622918349 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f88fc0cf440 con 0x7f890c1020b0 2024-03-18T01:35:28.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.579+0000 7f8901400640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f88f00a2800 0x7f88f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:28.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.579+0000 7f8901400640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f88f00a2800 0x7f88f00a4cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88f402c7c0 tx=0x7f88f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:28.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.582+0000 7f8903400640 1 -- 172.21.15.7:0/622918349 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f88fc09c240 con 0x7f890c1020b0 2024-03-18T01:35:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T01:35:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T01:35:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T01:35:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T01:35:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-18T01:35:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-18T01:35:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2748646546' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T01:35:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:28 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T01:35:28.703 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T01:35:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T01:35:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T01:35:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T01:35:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-18T01:35:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-18T01:35:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2748646546' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T01:35:28.704 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:28 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T01:35:28.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T01:35:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T01:35:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T01:35:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T01:35:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T01:35:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-18T01:35:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-18T01:35:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2748646546' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T01:35:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:28 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T01:35:28.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:28.892+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f890c02d1b0 con 0x7f890c1020b0 2024-03-18T01:35:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.343+0000 7f8903400640 1 -- 172.21.15.7:0/622918349 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f88fc09bb40 con 0x7f890c1020b0 2024-03-18T01:35:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f88f00a2800 msgr2=0x7f88f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f88f00a2800 0x7f88f00a4cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f88f402c7c0 tx=0x7f88f4005fd0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1020b0 msgr2=0x7f890c199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1020b0 0x7f890c199b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f88fc00ebc0 tx=0x7f88fc00c6a0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 shutdown_connections 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f88f00a2800 0x7f88f00a4cc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f890c107310 0x7f890c1a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f890c1049e0 0x7f890c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 --2- 172.21.15.7:0/622918349 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f890c1020b0 0x7f890c199b60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 >> 172.21.15.7:0/622918349 conn(0x7f890c0fbae0 msgr2=0x7f890c1018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 shutdown_connections 2024-03-18T01:35:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.348+0000 7f890b200640 1 -- 172.21.15.7:0/622918349 wait complete. 2024-03-18T01:35:29.376 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-18T01:35:29.376 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-18T01:35:29.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 -- 172.21.15.7:0/473780119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 msgr2=0x7fa0c8073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 --2- 172.21.15.7:0/473780119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 0x7fa0c8073c70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009900 tx=0x7fa0b802bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 -- 172.21.15.7:0/473780119 shutdown_connections 2024-03-18T01:35:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 --2- 172.21.15.7:0/473780119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c81094f0 0x7fa0c810b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 --2- 172.21.15.7:0/473780119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0c8074240 0x7fa0c80746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 --2- 172.21.15.7:0/473780119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 0x7fa0c8073c70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 -- 172.21.15.7:0/473780119 >> 172.21.15.7:0/473780119 conn(0x7fa0c80fcba0 msgr2=0x7fa0c80fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 -- 172.21.15.7:0/473780119 shutdown_connections 2024-03-18T01:35:29.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.565+0000 7fa0cca00640 1 -- 172.21.15.7:0/473780119 wait complete. 2024-03-18T01:35:29.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.566+0000 7fa0cca00640 1 Processor -- start 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.566+0000 7fa0cca00640 1 -- start start 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.566+0000 7fa0cca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8074240 0x7fa0c8199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.566+0000 7fa0cca00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 0x7fa0c819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0cca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0c81094f0 0x7fa0c81a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0cca00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810f470 con 0x7fa0c8075820 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0cca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810f2f0 con 0x7fa0c8074240 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0cca00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c810f5f0 con 0x7fa0c81094f0 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c5e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 0x7fa0c819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c7200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0c81094f0 0x7fa0c81a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8074240 0x7fa0c8199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:29.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c7200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0c81094f0 0x7fa0c81a10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52078/0 (socket says 172.21.15.7:52078) 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c7200640 1 -- 172.21.15.7:0/3626175559 learned_addr learned my addr 172.21.15.7:0/3626175559 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c7200640 1 -- 172.21.15.7:0/3626175559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8074240 msgr2=0x7fa0c8199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c7200640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8074240 0x7fa0c8199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c7200640 1 -- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 msgr2=0x7fa0c819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c7200640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 0x7fa0c819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c7200640 1 -- 172.21.15.7:0/3626175559 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c81a1710 con 0x7fa0c81094f0 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c5e00640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 0x7fa0c819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.567+0000 7fa0c6800640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8074240 0x7fa0c8199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.568+0000 7fa0c7200640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0c81094f0 0x7fa0c81a10a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000eaa0 tx=0x7fa0c000ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.568+0000 7fa0bf400640 1 -- 172.21.15.7:0/3626175559 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c000cdb0 con 0x7fa0c81094f0 2024-03-18T01:35:29.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.568+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c81a1a00 con 0x7fa0c81094f0 2024-03-18T01:35:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.568+0000 7fa0bf400640 1 -- 172.21.15.7:0/3626175559 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0c0005020 con 0x7fa0c81094f0 2024-03-18T01:35:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.568+0000 7fa0bf400640 1 -- 172.21.15.7:0/3626175559 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00116b0 con 0x7fa0c81094f0 2024-03-18T01:35:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.568+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c81a1f50 con 0x7fa0c81094f0 2024-03-18T01:35:29.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.569+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c8108220 con 0x7fa0c81094f0 2024-03-18T01:35:29.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.573+0000 7fa0bf400640 1 -- 172.21.15.7:0/3626175559 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fa0c0005560 con 0x7fa0c81094f0 2024-03-18T01:35:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.574+0000 7fa0bf400640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0ac0a2800 0x7fa0ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.574+0000 7fa0bf400640 1 -- 172.21.15.7:0/3626175559 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa0c00ceeb0 con 0x7fa0c81094f0 2024-03-18T01:35:29.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.574+0000 7fa0c6800640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0ac0a2800 0x7fa0ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:29.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.575+0000 7fa0c6800640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0ac0a2800 0x7fa0ac0a4cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa0b802c7b0 tx=0x7fa0b8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:29.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.578+0000 7fa0bf400640 1 -- 172.21.15.7:0/3626175559 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa0c0016030 con 0x7fa0c81094f0 2024-03-18T01:35:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:29 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T01:35:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:29 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:35:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:29 smithi203 ceph-mon[36345]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T01:35:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:29 smithi203 ceph-mon[36345]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-03-18T01:35:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:29 smithi203 ceph-mon[36345]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-03-18T01:35:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:29 smithi203 ceph-mon[36345]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-18T01:35:29.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:29 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/622918349' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T01:35:29.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:29 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T01:35:29.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:29 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T01:35:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:29 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:35:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:29 smithi046 ceph-mon[36108]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T01:35:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:29 smithi046 ceph-mon[36108]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-03-18T01:35:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:29 smithi046 ceph-mon[36108]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-03-18T01:35:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:29 smithi046 ceph-mon[36108]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-18T01:35:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:29 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/622918349' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T01:35:29.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:29 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T01:35:29.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T01:35:29.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:35:29.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-mon[31996]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T01:35:29.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-mon[31996]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-03-18T01:35:29.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-mon[31996]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-03-18T01:35:29.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-mon[31996]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-18T01:35:29.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/622918349' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T01:35:29.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T01:35:29.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.804+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa0c8061c20 con 0x7fa0ac0a2800 2024-03-18T01:35:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.832+0000 7fa0bf400640 1 -- 172.21.15.7:0/3626175559 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa0c8061c20 con 0x7fa0ac0a2800 2024-03-18T01:35:29.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0ac0a2800 msgr2=0x7fa0ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0ac0a2800 0x7fa0ac0a4cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa0b802c7b0 tx=0x7fa0b8005fd0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0c81094f0 msgr2=0x7fa0c81a10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0c81094f0 0x7fa0c81a10a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000eaa0 tx=0x7fa0c000ef60 comp rx=0 tx=0).stop 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 shutdown_connections 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0c81094f0 0x7fa0c81a10a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0ac0a2800 0x7fa0ac0a4cc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0c8075820 0x7fa0c819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 --2- 172.21.15.7:0/3626175559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0c8074240 0x7fa0c8199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 >> 172.21.15.7:0/3626175559 conn(0x7fa0c80fcba0 msgr2=0x7fa0c81053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 shutdown_connections 2024-03-18T01:35:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:29.835+0000 7fa0cca00640 1 -- 172.21.15.7:0/3626175559 wait complete. 2024-03-18T01:35:29.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-18T01:35:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 -- 172.21.15.7:0/1457920376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc075820 msgr2=0x7ff7bc073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/1457920376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc075820 0x7ff7bc073c70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0009900 tx=0x7ff7b002bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 -- 172.21.15.7:0/1457920376 shutdown_connections 2024-03-18T01:35:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/1457920376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7bc1094e0 0x7ff7bc10b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/1457920376 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc074240 0x7ff7bc0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/1457920376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc075820 0x7ff7bc073c70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 -- 172.21.15.7:0/1457920376 >> 172.21.15.7:0/1457920376 conn(0x7ff7bc0fcb90 msgr2=0x7ff7bc0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:30.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 -- 172.21.15.7:0/1457920376 shutdown_connections 2024-03-18T01:35:30.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.065+0000 7ff7c1e00640 1 -- 172.21.15.7:0/1457920376 wait complete. 2024-03-18T01:35:30.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.066+0000 7ff7c1e00640 1 Processor -- start 2024-03-18T01:35:30.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.066+0000 7ff7c1e00640 1 -- start start 2024-03-18T01:35:30.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.066+0000 7ff7c1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7bc074240 0x7ff7bc199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:30.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c1e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc075820 0x7ff7bc19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc1094e0 0x7ff7bc1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c1e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc10f4d0 con 0x7ff7bc075820 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c1e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc10f350 con 0x7ff7bc074240 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c1e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff7bc10f650 con 0x7ff7bc1094e0 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c0a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7bc074240 0x7ff7bc199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c0a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7bc074240 0x7ff7bc199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54476/0 (socket says 172.21.15.7:54476) 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c0a00640 1 -- 172.21.15.7:0/2427784906 learned_addr learned my addr 172.21.15.7:0/2427784906 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7bbe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc075820 0x7ff7bc19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c1400640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc1094e0 0x7ff7bc1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c0a00640 1 -- 172.21.15.7:0/2427784906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc1094e0 msgr2=0x7ff7bc1a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c0a00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc1094e0 0x7ff7bc1a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c0a00640 1 -- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc075820 msgr2=0x7ff7bc19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c0a00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc075820 0x7ff7bc19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c0a00640 1 -- 172.21.15.7:0/2427784906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7bc1a1730 con 0x7ff7bc074240 2024-03-18T01:35:30.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.067+0000 7ff7c1400640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc1094e0 0x7ff7bc1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:30.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.068+0000 7ff7c0a00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7bc074240 0x7ff7bc199b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0005ab0 tx=0x7ff7b002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:30.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.068+0000 7ff7b9600640 1 -- 172.21.15.7:0/2427784906 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0043020 con 0x7ff7bc074240 2024-03-18T01:35:30.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.068+0000 7ff7b9600640 1 -- 172.21.15.7:0/2427784906 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7b00054e0 con 0x7ff7bc074240 2024-03-18T01:35:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.068+0000 7ff7b9600640 1 -- 172.21.15.7:0/2427784906 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0003ce0 con 0x7ff7bc074240 2024-03-18T01:35:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.069+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7bc1a19c0 con 0x7ff7bc074240 2024-03-18T01:35:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.069+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7bc1a1ec0 con 0x7ff7bc074240 2024-03-18T01:35:30.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.070+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7bc108210 con 0x7ff7bc074240 2024-03-18T01:35:30.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.072+0000 7ff7b9600640 1 -- 172.21.15.7:0/2427784906 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7ff7b0046020 con 0x7ff7bc074240 2024-03-18T01:35:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.073+0000 7ff7b9600640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff7a40a2800 0x7ff7a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.073+0000 7ff7b9600640 1 -- 172.21.15.7:0/2427784906 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff7b00f08a0 con 0x7ff7bc074240 2024-03-18T01:35:30.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.074+0000 7ff7bbe00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff7a40a2800 0x7ff7a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:30.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.074+0000 7ff7bbe00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff7a40a2800 0x7ff7a40a4cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc19b020 tx=0x7ff7a800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:30.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.077+0000 7ff7b9600640 1 -- 172.21.15.7:0/2427784906 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7b004d070 con 0x7ff7bc074240 2024-03-18T01:35:30.302 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:35:29.814+0000 7fa9aa400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T01:35:30.302 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:35:29.814+0000 7fa9aa400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T01:35:30.302 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:35:29.814+0000 7fa9aa400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T01:35:30.303 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:35:29.814+0000 7fa9aa400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T01:35:30.303 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:35:29 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:35:29.814+0000 7fa9aa400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T01:35:30.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.312+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff7bc061c20 con 0x7ff7a40a2800 2024-03-18T01:35:30.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.340+0000 7ff7b9600640 1 -- 172.21.15.7:0/2427784906 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff7bc061c20 con 0x7ff7a40a2800 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff7a40a2800 msgr2=0x7ff7a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff7a40a2800 0x7ff7a40a4cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc19b020 tx=0x7ff7a800b390 comp rx=0 tx=0).stop 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7bc074240 msgr2=0x7ff7bc199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7bc074240 0x7ff7bc199b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff7b0005ab0 tx=0x7ff7b002c6c0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 shutdown_connections 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7bc1094e0 0x7ff7bc1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff7a40a2800 0x7ff7a40a4cc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff7bc075820 0x7ff7bc19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 --2- 172.21.15.7:0/2427784906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7bc074240 0x7ff7bc199b10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 >> 172.21.15.7:0/2427784906 conn(0x7ff7bc0fcb90 msgr2=0x7ff7bc1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 shutdown_connections 2024-03-18T01:35:30.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.343+0000 7ff7c1e00640 1 -- 172.21.15.7:0/2427784906 wait complete. 2024-03-18T01:35:30.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-18T01:35:30.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 -- 172.21.15.7:0/1300824473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb01020b0 msgr2=0x7efeb01044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 --2- 172.21.15.7:0/1300824473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb01020b0 0x7efeb01044a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efea0009900 tx=0x7efea002bc40 comp rx=0 tx=0).stop 2024-03-18T01:35:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 -- 172.21.15.7:0/1300824473 shutdown_connections 2024-03-18T01:35:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 --2- 172.21.15.7:0/1300824473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb0107310 0x7efeb0109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 --2- 172.21.15.7:0/1300824473 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb01049e0 0x7efeb0106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 --2- 172.21.15.7:0/1300824473 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb01020b0 0x7efeb01044a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 -- 172.21.15.7:0/1300824473 >> 172.21.15.7:0/1300824473 conn(0x7efeb00fbac0 msgr2=0x7efeb00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 -- 172.21.15.7:0/1300824473 shutdown_connections 2024-03-18T01:35:30.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.576+0000 7efeb4a00640 1 -- 172.21.15.7:0/1300824473 wait complete. 2024-03-18T01:35:30.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.577+0000 7efeb4a00640 1 Processor -- start 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.577+0000 7efeb4a00640 1 -- start start 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeb4a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb01020b0 0x7efeb0101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeb4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb01049e0 0x7efeb0100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeb4a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb0107310 0x7efeb01005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeb4a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efeb010d2a0 con 0x7efeb0107310 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeb4a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efeb010d120 con 0x7efeb01020b0 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeb4a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efeb010d420 con 0x7efeb01049e0 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeae800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb01020b0 0x7efeb0101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeae800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb01020b0 0x7efeb0101a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54494/0 (socket says 172.21.15.7:54494) 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efea5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb01049e0 0x7efeb0100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:30.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeae800640 1 -- 172.21.15.7:0/3187501560 learned_addr learned my addr 172.21.15.7:0/3187501560 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeaf200640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb0107310 0x7efeb01005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeae800640 1 -- 172.21.15.7:0/3187501560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb01049e0 msgr2=0x7efeb0100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeae800640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb01049e0 0x7efeb0100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeae800640 1 -- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb0107310 msgr2=0x7efeb01005a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeae800640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb0107310 0x7efeb01005a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.578+0000 7efeae800640 1 -- 172.21.15.7:0/3187501560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb0100e30 con 0x7efeb01020b0 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.579+0000 7efeae800640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb01020b0 0x7efeb0101a20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efea0009230 tx=0x7efea0005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.579+0000 7efea7e00640 1 -- 172.21.15.7:0/3187501560 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0041070 con 0x7efeb01020b0 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.579+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb01a1a00 con 0x7efeb01020b0 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.579+0000 7efea7e00640 1 -- 172.21.15.7:0/3187501560 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea00044d0 con 0x7efeb01020b0 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.579+0000 7efea7e00640 1 -- 172.21.15.7:0/3187501560 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0034440 con 0x7efeb01020b0 2024-03-18T01:35:30.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.579+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb01a1f50 con 0x7efeb01020b0 2024-03-18T01:35:30.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.580+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb004f540 con 0x7efeb01020b0 2024-03-18T01:35:30.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.582+0000 7efea7e00640 1 -- 172.21.15.7:0/3187501560 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7efea0044070 con 0x7efeb01020b0 2024-03-18T01:35:30.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.583+0000 7efea7e00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efe940a2800 0x7efe940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:30.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.584+0000 7efea7e00640 1 -- 172.21.15.7:0/3187501560 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7efea00f09b0 con 0x7efeb01020b0 2024-03-18T01:35:30.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.584+0000 7efea5e00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efe940a2800 0x7efe940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:30.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.585+0000 7efea5e00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efe940a2800 0x7efe940a4cc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efeb0101790 tx=0x7efe98009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:30.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.587+0000 7efea7e00640 1 -- 172.21.15.7:0/3187501560 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efea00bd7b0 con 0x7efeb01020b0 2024-03-18T01:35:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:30 smithi203 ceph-mon[36345]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-03-18T01:35:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:30 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T01:35:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:30 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:35:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:30 smithi203 ceph-mon[36345]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T01:35:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:30 smithi203 ceph-mon[36345]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-03-18T01:35:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:30 smithi203 ceph-mon[36345]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-03-18T01:35:30.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:30 smithi203 ceph-mon[36345]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T01:35:30.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:30 smithi203 ceph-mon[36345]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:30 smithi046 ceph-mon[36108]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-03-18T01:35:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:30 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T01:35:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:30 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:35:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:30 smithi046 ceph-mon[36108]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T01:35:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:30 smithi046 ceph-mon[36108]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-03-18T01:35:30.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:30 smithi046 ceph-mon[36108]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-03-18T01:35:30.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:30 smithi046 ceph-mon[36108]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T01:35:30.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:30 smithi046 ceph-mon[36108]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:30 smithi007 ceph-mon[31996]: pgmap v335: 129 pgs: 129 active+clean; 582 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-03-18T01:35:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:30 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T01:35:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:30 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:35:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:30 smithi007 ceph-mon[31996]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T01:35:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:30 smithi007 ceph-mon[31996]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-03-18T01:35:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:30 smithi007 ceph-mon[31996]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-03-18T01:35:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:30 smithi007 ceph-mon[31996]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T01:35:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:30 smithi007 ceph-mon[31996]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:30.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.902+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7efeb0061d70 con 0x7efe940a2800 2024-03-18T01:35:30.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.907+0000 7efea7e00640 1 -- 172.21.15.7:0/3187501560 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7efeb0061d70 con 0x7efe940a2800 2024-03-18T01:35:30.908 INFO:teuthology.orchestra.run.smithi007.stdout:/volumes/qa/sv_0/bb91eeac-5527-43e8-9fcd-e8f63041f9f3 2024-03-18T01:35:30.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efe940a2800 msgr2=0x7efe940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efe940a2800 0x7efe940a4cc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efeb0101790 tx=0x7efe98009680 comp rx=0 tx=0).stop 2024-03-18T01:35:30.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb01020b0 msgr2=0x7efeb0101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb01020b0 0x7efeb0101a20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efea0009230 tx=0x7efea0005d50 comp rx=0 tx=0).stop 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 shutdown_connections 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efe940a2800 0x7efe940a4cc0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efeb0107310 0x7efeb01005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efeb01049e0 0x7efeb0100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 --2- 172.21.15.7:0/3187501560 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efeb01020b0 0x7efeb0101a20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 >> 172.21.15.7:0/3187501560 conn(0x7efeb00fbac0 msgr2=0x7efeb0102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 shutdown_connections 2024-03-18T01:35:30.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:30.910+0000 7efeb4a00640 1 -- 172.21.15.7:0/3187501560 wait complete. 2024-03-18T01:35:30.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-18T01:35:31.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.156+0000 7fa637200640 1 -- 172.21.15.7:0/156941951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 msgr2=0x7fa6380746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.156+0000 7fa637200640 1 --2- 172.21.15.7:0/156941951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 0x7fa6380746c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa624009900 tx=0x7fa62402bc30 comp rx=0 tx=0).stop 2024-03-18T01:35:31.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.156+0000 7fa637200640 1 -- 172.21.15.7:0/156941951 shutdown_connections 2024-03-18T01:35:31.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.156+0000 7fa637200640 1 --2- 172.21.15.7:0/156941951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6381094f0 0x7fa63810b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.156+0000 7fa637200640 1 --2- 172.21.15.7:0/156941951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 0x7fa6380746c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.156+0000 7fa637200640 1 --2- 172.21.15.7:0/156941951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa638075820 0x7fa638073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.156+0000 7fa637200640 1 -- 172.21.15.7:0/156941951 >> 172.21.15.7:0/156941951 conn(0x7fa6380fcba0 msgr2=0x7fa6380fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:31.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.157+0000 7fa637200640 1 -- 172.21.15.7:0/156941951 shutdown_connections 2024-03-18T01:35:31.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.157+0000 7fa637200640 1 -- 172.21.15.7:0/156941951 wait complete. 2024-03-18T01:35:31.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa637200640 1 Processor -- start 2024-03-18T01:35:31.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa637200640 1 -- start start 2024-03-18T01:35:31.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa637200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 0x7fa638199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:31.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa637200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa638075820 0x7fa63819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:31.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa637200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6381094f0 0x7fa6381a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:31.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa637200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa63810f470 con 0x7fa638075820 2024-03-18T01:35:31.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa637200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa63810f2f0 con 0x7fa6381094f0 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa637200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa63810f5f0 con 0x7fa638074240 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa635e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 0x7fa638199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa635e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 0x7fa638199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52158/0 (socket says 172.21.15.7:52158) 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa635e00640 1 -- 172.21.15.7:0/1874318602 learned_addr learned my addr 172.21.15.7:0/1874318602 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa635400640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa638075820 0x7fa63819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.158+0000 7fa636800640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6381094f0 0x7fa6381a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa635e00640 1 -- 172.21.15.7:0/1874318602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6381094f0 msgr2=0x7fa6381a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa635e00640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6381094f0 0x7fa6381a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa635e00640 1 -- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa638075820 msgr2=0x7fa63819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa635e00640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa638075820 0x7fa63819a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa635e00640 1 -- 172.21.15.7:0/1874318602 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6381a1720 con 0x7fa638074240 2024-03-18T01:35:31.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa635e00640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 0x7fa638199b00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa62800d950 tx=0x7fa62800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa62ea00640 1 -- 172.21.15.7:0/1874318602 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628017070 con 0x7fa638074240 2024-03-18T01:35:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa62ea00640 1 -- 172.21.15.7:0/1874318602 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa628009c30 con 0x7fa638074240 2024-03-18T01:35:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.159+0000 7fa62ea00640 1 -- 172.21.15.7:0/1874318602 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62800f6b0 con 0x7fa638074240 2024-03-18T01:35:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.160+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6381a1a10 con 0x7fa638074240 2024-03-18T01:35:31.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.160+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6381a1ee0 con 0x7fa638074240 2024-03-18T01:35:31.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.161+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa638108220 con 0x7fa638074240 2024-03-18T01:35:31.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.163+0000 7fa62ea00640 1 -- 172.21.15.7:0/1874318602 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 137289+0+0 (secure 0 0 0) 0x7fa6280109d0 con 0x7fa638074240 2024-03-18T01:35:31.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.164+0000 7fa62ea00640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa61c0a2880 0x7fa61c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:31.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.164+0000 7fa62ea00640 1 -- 172.21.15.7:0/1874318602 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa6280ce840 con 0x7fa638074240 2024-03-18T01:35:31.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.165+0000 7fa635400640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa61c0a2880 0x7fa61c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:31.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.165+0000 7fa635400640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa61c0a2880 0x7fa61c0a4d40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa62402c7b0 tx=0x7fa624005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:31.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.168+0000 7fa62ea00640 1 -- 172.21.15.7:0/1874318602 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa628009de0 con 0x7fa638074240 2024-03-18T01:35:31.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.433+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa638061cb0 con 0x7fa61c0a2880 2024-03-18T01:35:31.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.465+0000 7fa62ea00640 1 -- 172.21.15.7:0/1874318602 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa638061cb0 con 0x7fa61c0a2880 2024-03-18T01:35:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa61c0a2880 msgr2=0x7fa61c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa61c0a2880 0x7fa61c0a4d40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa62402c7b0 tx=0x7fa624005fd0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 msgr2=0x7fa638199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 0x7fa638199b00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa62800d950 tx=0x7fa62800de10 comp rx=0 tx=0).stop 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 shutdown_connections 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6381094f0 0x7fa6381a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa61c0a2880 0x7fa61c0a4d40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa638075820 0x7fa63819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 --2- 172.21.15.7:0/1874318602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa638074240 0x7fa638199b00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.468+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 >> 172.21.15.7:0/1874318602 conn(0x7fa6380fcba0 msgr2=0x7fa6381053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.469+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 shutdown_connections 2024-03-18T01:35:31.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.469+0000 7fa637200640 1 -- 172.21.15.7:0/1874318602 wait complete. 2024-03-18T01:35:31.489 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-18T01:35:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:35:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T01:35:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T01:35:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: Cluster is now healthy 2024-03-18T01:35:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-03-18T01:35:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T01:35:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-18T01:35:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:35:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:35:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:31 smithi203 ceph-mon[36345]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 -- 172.21.15.7:0/1936568190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341049e0 msgr2=0x7fd434106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 --2- 172.21.15.7:0/1936568190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341049e0 0x7fd434106dd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd424009ca0 tx=0x7fd42402c100 comp rx=0 tx=0).stop 2024-03-18T01:35:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 -- 172.21.15.7:0/1936568190 shutdown_connections 2024-03-18T01:35:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 --2- 172.21.15.7:0/1936568190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd434107310 0x7fd434109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 --2- 172.21.15.7:0/1936568190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341049e0 0x7fd434106dd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 --2- 172.21.15.7:0/1936568190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4341020b0 0x7fd4341044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 -- 172.21.15.7:0/1936568190 >> 172.21.15.7:0/1936568190 conn(0x7fd4340fbac0 msgr2=0x7fd4340fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 -- 172.21.15.7:0/1936568190 shutdown_connections 2024-03-18T01:35:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.721+0000 7fd438a00640 1 -- 172.21.15.7:0/1936568190 wait complete. 2024-03-18T01:35:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.722+0000 7fd438a00640 1 Processor -- start 2024-03-18T01:35:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.722+0000 7fd438a00640 1 -- start start 2024-03-18T01:35:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd438a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341020b0 0x7fd434197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd438a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4341049e0 0x7fd434197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd438a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd434107310 0x7fd43419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd438a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd43410d2b0 con 0x7fd434107310 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd438a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd43410d130 con 0x7fd4341049e0 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd438a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd43410d430 con 0x7fd4341020b0 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd433400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341020b0 0x7fd434197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd432a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4341049e0 0x7fd434197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd433400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341020b0 0x7fd434197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52162/0 (socket says 172.21.15.7:52162) 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd433400640 1 -- 172.21.15.7:0/3113811370 learned_addr learned my addr 172.21.15.7:0/3113811370 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd433e00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd434107310 0x7fd43419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd432a00640 1 -- 172.21.15.7:0/3113811370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341020b0 msgr2=0x7fd434197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd432a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341020b0 0x7fd434197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd432a00640 1 -- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd434107310 msgr2=0x7fd43419ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd432a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd434107310 0x7fd43419ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd432a00640 1 -- 172.21.15.7:0/3113811370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43419f570 con 0x7fd4341049e0 2024-03-18T01:35:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd433e00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd434107310 0x7fd43419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.723+0000 7fd433400640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341020b0 0x7fd434197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.724+0000 7fd432a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4341049e0 0x7fd434197e90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd424037040 tx=0x7fd424035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.724+0000 7fd42be00640 1 -- 172.21.15.7:0/3113811370 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424039040 con 0x7fd4341049e0 2024-03-18T01:35:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.724+0000 7fd42be00640 1 -- 172.21.15.7:0/3113811370 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd424003970 con 0x7fd4341049e0 2024-03-18T01:35:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.724+0000 7fd42be00640 1 -- 172.21.15.7:0/3113811370 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4240043c0 con 0x7fd4341049e0 2024-03-18T01:35:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.724+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43419f800 con 0x7fd4341049e0 2024-03-18T01:35:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.724+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd43419fc70 con 0x7fd4341049e0 2024-03-18T01:35:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.725+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4340fd5b0 con 0x7fd4341049e0 2024-03-18T01:35:31.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.729+0000 7fd42be00640 1 -- 172.21.15.7:0/3113811370 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fd424046030 con 0x7fd4341049e0 2024-03-18T01:35:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.730+0000 7fd42be00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4180a2910 0x7fd4180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:31.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.730+0000 7fd42be00640 1 -- 172.21.15.7:0/3113811370 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd4240f0930 con 0x7fd4341049e0 2024-03-18T01:35:31.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.731+0000 7fd433400640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4180a2910 0x7fd4180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:31.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.731+0000 7fd433400640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4180a2910 0x7fd4180a4dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd41c011500 tx=0x7fd41c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:31.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.735+0000 7fd42be00640 1 -- 172.21.15.7:0/3113811370 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd42404d060 con 0x7fd4341049e0 2024-03-18T01:35:31.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:31.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:35:31.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: Cluster is now healthy 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-mon[36108]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: 2024-03-18T01:35:31.438+0000 7f8f22afa700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-18T01:35:31.774 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: 2024-03-18T01:35:31.438+0000 7f8f22afa700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-18T01:35:31.775 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: 2024-03-18T01:35:31.438+0000 7f8f22afa700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-18T01:35:31.775 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:35:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: 2024-03-18T01:35:31.438+0000 7f8f22afa700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-18T01:35:31.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: Cluster is now healthy 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:35:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:31 smithi007 ceph-mon[31996]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:31.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.973+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fd434061c20 con 0x7fd4180a2910 2024-03-18T01:35:31.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.979+0000 7fd42be00640 1 -- 172.21.15.7:0/3113811370 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fd434061c20 con 0x7fd4180a2910 2024-03-18T01:35:31.981 INFO:teuthology.orchestra.run.smithi007.stdout:/volumes/qa/sv_1/e5720a72-fcfe-434a-81cc-b7152c6f2925 2024-03-18T01:35:31.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4180a2910 msgr2=0x7fd4180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4180a2910 0x7fd4180a4dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd41c011500 tx=0x7fd41c005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:31.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4341049e0 msgr2=0x7fd434197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:31.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4341049e0 0x7fd434197e90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd424037040 tx=0x7fd424035d10 comp rx=0 tx=0).stop 2024-03-18T01:35:31.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 shutdown_connections 2024-03-18T01:35:31.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd4180a2910 0x7fd4180a4dd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd434107310 0x7fd43419ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd4341049e0 0x7fd434197e90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 --2- 172.21.15.7:0/3113811370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd4341020b0 0x7fd434197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:31.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.982+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 >> 172.21.15.7:0/3113811370 conn(0x7fd4340fbac0 msgr2=0x7fd4341029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:31.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.983+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 shutdown_connections 2024-03-18T01:35:31.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:31.983+0000 7fd438a00640 1 -- 172.21.15.7:0/3113811370 wait complete. 2024-03-18T01:35:32.006 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:35:32.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 -- 172.21.15.7:0/432757501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d58105180 msgr2=0x7f6d58105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:32.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 --2- 172.21.15.7:0/432757501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d58105180 0x7f6d58105600 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6d50009ca0 tx=0x7f6d5002c0f0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 -- 172.21.15.7:0/432757501 shutdown_connections 2024-03-18T01:35:32.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 --2- 172.21.15.7:0/432757501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d58105180 0x7f6d58105600 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 --2- 172.21.15.7:0/432757501 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d581027d0 0x7f6d58104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 --2- 172.21.15.7:0/432757501 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d580ffe60 0x7f6d58102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 -- 172.21.15.7:0/432757501 >> 172.21.15.7:0/432757501 conn(0x7f6d580fbaa0 msgr2=0x7f6d580fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 -- 172.21.15.7:0/432757501 shutdown_connections 2024-03-18T01:35:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.251+0000 7f6d5e200640 1 -- 172.21.15.7:0/432757501 wait complete. 2024-03-18T01:35:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.252+0000 7f6d5e200640 1 Processor -- start 2024-03-18T01:35:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5e200640 1 -- start start 2024-03-18T01:35:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5e200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d580ffe60 0x7f6d5819ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5e200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d581027d0 0x7f6d5819e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:32.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5e200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d58105180 0x7f6d581a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:32.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5e200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6d58113840 con 0x7f6d581027d0 2024-03-18T01:35:32.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5e200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6d581136c0 con 0x7f6d58105180 2024-03-18T01:35:32.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5e200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d581139c0 con 0x7f6d580ffe60 2024-03-18T01:35:32.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d580ffe60 0x7f6d5819ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:32.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d580ffe60 0x7f6d5819ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52174/0 (socket says 172.21.15.7:52174) 2024-03-18T01:35:32.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5ce00640 1 -- 172.21.15.7:0/2999922781 learned_addr learned my addr 172.21.15.7:0/2999922781 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d5d800640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d58105180 0x7f6d581a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.253+0000 7f6d57e00640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d581027d0 0x7f6d5819e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:32.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d5ce00640 1 -- 172.21.15.7:0/2999922781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d58105180 msgr2=0x7f6d581a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:32.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d5ce00640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d58105180 0x7f6d581a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d5ce00640 1 -- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d581027d0 msgr2=0x7f6d5819e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:32.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d5ce00640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d581027d0 0x7f6d5819e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d5ce00640 1 -- 172.21.15.7:0/2999922781 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d581a59d0 con 0x7f6d580ffe60 2024-03-18T01:35:32.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d57e00640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d581027d0 0x7f6d5819e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:32.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d5ce00640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d580ffe60 0x7f6d5819ddc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c00e9f0 tx=0x7f6d4c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:32.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d55600640 1 -- 172.21.15.7:0/2999922781 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c00cd80 con 0x7f6d580ffe60 2024-03-18T01:35:32.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d55600640 1 -- 172.21.15.7:0/2999922781 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d4c010910 con 0x7f6d580ffe60 2024-03-18T01:35:32.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d55600640 1 -- 172.21.15.7:0/2999922781 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4c018990 con 0x7f6d580ffe60 2024-03-18T01:35:32.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.254+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d581a5cc0 con 0x7f6d580ffe60 2024-03-18T01:35:32.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.255+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d581a61c0 con 0x7f6d580ffe60 2024-03-18T01:35:32.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.256+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d580fd590 con 0x7f6d580ffe60 2024-03-18T01:35:32.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.258+0000 7f6d55600640 1 -- 172.21.15.7:0/2999922781 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f6d4c010ac0 con 0x7f6d580ffe60 2024-03-18T01:35:32.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.260+0000 7f6d55600640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6d400a2940 0x7f6d400a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:32.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.260+0000 7f6d55600640 1 -- 172.21.15.7:0/2999922781 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6d4c0cf2c0 con 0x7f6d580ffe60 2024-03-18T01:35:32.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.260+0000 7f6d57e00640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6d400a2940 0x7f6d400a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:32.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.261+0000 7f6d57e00640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6d400a2940 0x7f6d400a4e00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009c30 tx=0x7f6d44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.268+0000 7f6d55600640 1 -- 172.21.15.7:0/2999922781 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d4c09bfc0 con 0x7f6d580ffe60 2024-03-18T01:35:32.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.514+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d5802d460 con 0x7f6d580ffe60 2024-03-18T01:35:32.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.515+0000 7f6d55600640 1 -- 172.21.15.7:0/2999922781 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11918 (secure 0 0 0) 0x7f6d4c09b8c0 con 0x7f6d580ffe60 2024-03-18T01:35:32.516 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:32.516 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:35:30.634148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24542,24523,24544,24596,24562]},"id":1}]} 2024-03-18T01:35:32.517 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2024-03-18T01:35:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.517+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6d400a2940 msgr2=0x7f6d400a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.517+0000 7f6d5e200640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6d400a2940 0x7f6d400a4e00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6d44009c30 tx=0x7f6d44005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d580ffe60 msgr2=0x7f6d5819ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d580ffe60 0x7f6d5819ddc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6d4c00e9f0 tx=0x7f6d4c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 shutdown_connections 2024-03-18T01:35:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d58105180 0x7f6d581a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6d400a2940 0x7f6d400a4e00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6d581027d0 0x7f6d5819e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 --2- 172.21.15.7:0/2999922781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d580ffe60 0x7f6d5819ddc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 >> 172.21.15.7:0/2999922781 conn(0x7f6d580fbaa0 msgr2=0x7f6d58100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 shutdown_connections 2024-03-18T01:35:32.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.518+0000 7f6d5e200640 1 -- 172.21.15.7:0/2999922781 wait complete. 2024-03-18T01:35:32.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:35:32.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:32 smithi007 ceph-mon[31996]: pgmap v336: 129 pgs: 129 active+clean; 585 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.7 KiB/s wr, 9 op/s 2024-03-18T01:35:32.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:32 smithi007 ceph-mon[31996]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:32.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:32 smithi007 ceph-mon[31996]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T01:35:32.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:32 smithi007 ceph-mon[31996]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:32.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:32 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2999922781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:32.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.821+0000 7fd710e00640 1 -- 172.21.15.7:0/2471221378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c1012e0 msgr2=0x7fd70c101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:32.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.821+0000 7fd710e00640 1 --2- 172.21.15.7:0/2471221378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c1012e0 0x7fd70c101760 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd704009ca0 tx=0x7fd70402c180 comp rx=0 tx=0).stop 2024-03-18T01:35:32.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.822+0000 7fd710e00640 1 -- 172.21.15.7:0/2471221378 shutdown_connections 2024-03-18T01:35:32.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.822+0000 7fd710e00640 1 --2- 172.21.15.7:0/2471221378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c1012e0 0x7fd70c101760 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.822+0000 7fd710e00640 1 --2- 172.21.15.7:0/2471221378 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd70c100940 0x7fd70c100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.822+0000 7fd710e00640 1 --2- 172.21.15.7:0/2471221378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd70c0fffe0 0x7fd70c100400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.822+0000 7fd710e00640 1 -- 172.21.15.7:0/2471221378 >> 172.21.15.7:0/2471221378 conn(0x7fd70c0fbb60 msgr2=0x7fd70c0fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:32.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.822+0000 7fd710e00640 1 -- 172.21.15.7:0/2471221378 shutdown_connections 2024-03-18T01:35:32.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.822+0000 7fd710e00640 1 -- 172.21.15.7:0/2471221378 wait complete. 2024-03-18T01:35:32.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.823+0000 7fd710e00640 1 Processor -- start 2024-03-18T01:35:32.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.823+0000 7fd710e00640 1 -- start start 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd710e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd70c0fffe0 0x7fd70c199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd710e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c100940 0x7fd70c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd710e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd70c1012e0 0x7fd70c1a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd710e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c10f550 con 0x7fd70c1012e0 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd710e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c10f3d0 con 0x7fd70c100940 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd710e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd70c10f6d0 con 0x7fd70c0fffe0 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd70b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd70c0fffe0 0x7fd70c199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd70aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c100940 0x7fd70c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd70aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c100940 0x7fd70c19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54556/0 (socket says 172.21.15.7:54556) 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd70aa00640 1 -- 172.21.15.7:0/1287299226 learned_addr learned my addr 172.21.15.7:0/1287299226 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd70be00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd70c1012e0 0x7fd70c1a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.824+0000 7fd70aa00640 1 -- 172.21.15.7:0/1287299226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd70c0fffe0 msgr2=0x7fd70c199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:32.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd70aa00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd70c0fffe0 0x7fd70c199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd70aa00640 1 -- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd70c1012e0 msgr2=0x7fd70c1a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd70aa00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd70c1012e0 0x7fd70c1a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd70aa00640 1 -- 172.21.15.7:0/1287299226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70c1a17b0 con 0x7fd70c100940 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd70aa00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c100940 0x7fd70c19a0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd6f800e9f0 tx=0x7fd6f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd703e00640 1 -- 172.21.15.7:0/1287299226 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f800cd80 con 0x7fd70c100940 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd703e00640 1 -- 172.21.15.7:0/1287299226 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6f8010910 con 0x7fd70c100940 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70c1a1aa0 con 0x7fd70c100940 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd703e00640 1 -- 172.21.15.7:0/1287299226 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f8018990 con 0x7fd70c100940 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.825+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd70c1a2040 con 0x7fd70c100940 2024-03-18T01:35:32.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.826+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd70c10ad30 con 0x7fd70c100940 2024-03-18T01:35:32.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.829+0000 7fd703e00640 1 -- 172.21.15.7:0/1287299226 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fd6f8010ac0 con 0x7fd70c100940 2024-03-18T01:35:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.830+0000 7fd703e00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd6f00a2900 0x7fd6f00a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:32.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.830+0000 7fd703e00640 1 -- 172.21.15.7:0/1287299226 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd6f80cf1d0 con 0x7fd70c100940 2024-03-18T01:35:32.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.831+0000 7fd70b400640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd6f00a2900 0x7fd6f00a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:32.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.832+0000 7fd70b400640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd6f00a2900 0x7fd6f00a4dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009bd0 tx=0x7fd6fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:32.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:32.839+0000 7fd703e00640 1 -- 172.21.15.7:0/1287299226 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd6f809be70 con 0x7fd70c100940 2024-03-18T01:35:33.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:32 smithi046 ceph-mon[36108]: pgmap v336: 129 pgs: 129 active+clean; 585 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.7 KiB/s wr, 9 op/s 2024-03-18T01:35:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:32 smithi046 ceph-mon[36108]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:32 smithi046 ceph-mon[36108]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T01:35:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:32 smithi046 ceph-mon[36108]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:32 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2999922781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:33.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.051+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd70c02d3f0 con 0x7fd70c100940 2024-03-18T01:35:33.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.053+0000 7fd703e00640 1 -- 172.21.15.7:0/1287299226 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7fd6f809b770 con 0x7fd70c100940 2024-03-18T01:35:33.054 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:33.055 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd6f00a2900 msgr2=0x7fd6f00a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd6f00a2900 0x7fd6f00a4dc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc009bd0 tx=0x7fd6fc005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:33.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c100940 msgr2=0x7fd70c19a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c100940 0x7fd70c19a0d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd6f800e9f0 tx=0x7fd6f800eeb0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 shutdown_connections 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd6f00a2900 0x7fd6f00a4dc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd70c1012e0 0x7fd70c1a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd70c100940 0x7fd70c19a0d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 --2- 172.21.15.7:0/1287299226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd70c0fffe0 0x7fd70c199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.056+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 >> 172.21.15.7:0/1287299226 conn(0x7fd70c0fbb60 msgr2=0x7fd70c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.057+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 shutdown_connections 2024-03-18T01:35:33.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.057+0000 7fd710e00640 1 -- 172.21.15.7:0/1287299226 wait complete. 2024-03-18T01:35:33.084 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T01:35:33.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T01:35:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:32 smithi203 ceph-mon[36345]: pgmap v336: 129 pgs: 129 active+clean; 585 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.7 KiB/s wr, 9 op/s 2024-03-18T01:35:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:32 smithi203 ceph-mon[36345]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:32 smithi203 ceph-mon[36345]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T01:35:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:32 smithi203 ceph-mon[36345]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:35:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:32 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2999922781' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:35:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 -- 172.21.15.7:0/1336045086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0450077d80 msgr2=0x7f0450078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 --2- 172.21.15.7:0/1336045086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0450077d80 0x7f0450078620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0448009ca0 tx=0x7f044802c100 comp rx=0 tx=0).stop 2024-03-18T01:35:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 -- 172.21.15.7:0/1336045086 shutdown_connections 2024-03-18T01:35:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 --2- 172.21.15.7:0/1336045086 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0450077d80 0x7f0450078620 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 --2- 172.21.15.7:0/1336045086 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04500773e0 0x7f0450077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 --2- 172.21.15.7:0/1336045086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500761a0 0x7f04500765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 -- 172.21.15.7:0/1336045086 >> 172.21.15.7:0/1336045086 conn(0x7f04500fef90 msgr2=0x7f04501013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:33.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 -- 172.21.15.7:0/1336045086 shutdown_connections 2024-03-18T01:35:33.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.322+0000 7f0457800640 1 -- 172.21.15.7:0/1336045086 wait complete. 2024-03-18T01:35:33.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.323+0000 7f0457800640 1 Processor -- start 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0457800640 1 -- start start 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0457800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04500761a0 0x7f045019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0457800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500773e0 0x7f045019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0457800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0450077d80 0x7f04501a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0457800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0450113a90 con 0x7f04500761a0 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0457800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0450113910 con 0x7f04500773e0 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0457800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0450113c10 con 0x7f0450077d80 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f044fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500773e0 0x7f045019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0454e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04500761a0 0x7f045019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f044fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500773e0 0x7f045019e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54578/0 (socket says 172.21.15.7:54578) 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0454e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04500761a0 0x7f045019e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36092/0 (socket says 172.21.15.7:36092) 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f044fe00640 1 -- 172.21.15.7:0/778381933 learned_addr learned my addr 172.21.15.7:0/778381933 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:33.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.324+0000 7f0455800640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0450077d80 0x7f04501a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f044fe00640 1 -- 172.21.15.7:0/778381933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0450077d80 msgr2=0x7f04501a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f044fe00640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0450077d80 0x7f04501a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f044fe00640 1 -- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04500761a0 msgr2=0x7f045019e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f044fe00640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04500761a0 0x7f045019e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f044fe00640 1 -- 172.21.15.7:0/778381933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04501a5c50 con 0x7f04500773e0 2024-03-18T01:35:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f044fe00640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500773e0 0x7f045019e570 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f043c00d950 tx=0x7f043c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f044d600640 1 -- 172.21.15.7:0/778381933 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c017070 con 0x7f04500773e0 2024-03-18T01:35:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f044d600640 1 -- 172.21.15.7:0/778381933 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f043c009c30 con 0x7f04500773e0 2024-03-18T01:35:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.325+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04501a5f40 con 0x7f04500773e0 2024-03-18T01:35:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.326+0000 7f044d600640 1 -- 172.21.15.7:0/778381933 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043c00f6b0 con 0x7f04500773e0 2024-03-18T01:35:33.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.326+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04501a6400 con 0x7f04500773e0 2024-03-18T01:35:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.327+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0450110370 con 0x7f04500773e0 2024-03-18T01:35:33.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.329+0000 7f044d600640 1 -- 172.21.15.7:0/778381933 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f043c0109d0 con 0x7f04500773e0 2024-03-18T01:35:33.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.331+0000 7f044d600640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f04380a2890 0x7f04380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.331+0000 7f044d600640 1 -- 172.21.15.7:0/778381933 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f043c0cead0 con 0x7f04500773e0 2024-03-18T01:35:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.331+0000 7f0454e00640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f04380a2890 0x7f04380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:33.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.331+0000 7f0454e00640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f04380a2890 0x7f04380a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0444009b60 tx=0x7f0444005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:33.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.335+0000 7f044d600640 1 -- 172.21.15.7:0/778381933 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f043c009de0 con 0x7f04500773e0 2024-03-18T01:35:33.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.535+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0450061c20 con 0x7f04380a2890 2024-03-18T01:35:33.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.570+0000 7f044d600640 1 -- 172.21.15.7:0/778381933 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340397 (secure 0 0 0) 0x7f0450061c20 con 0x7f04380a2890 2024-03-18T01:35:33.579 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:33.581 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f04380a2890 msgr2=0x7f04380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f04380a2890 0x7f04380a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0444009b60 tx=0x7f0444005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500773e0 msgr2=0x7f045019e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500773e0 0x7f045019e570 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f043c00d950 tx=0x7f043c00de10 comp rx=0 tx=0).stop 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 shutdown_connections 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0450077d80 0x7f04501a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04500773e0 0x7f045019e570 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f04380a2890 0x7f04380a4d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 --2- 172.21.15.7:0/778381933 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f04500761a0 0x7f045019e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 >> 172.21.15.7:0/778381933 conn(0x7f04500fef90 msgr2=0x7f0450100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:33.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 shutdown_connections 2024-03-18T01:35:33.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.579+0000 7f0457800640 1 -- 172.21.15.7:0/778381933 wait complete. 2024-03-18T01:35:33.603 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-03-18T01:35:33.180013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":638476,"num_objects":91,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":150,"num_read_kb":167,"num_write":390,"num_write_kb":3035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":319,"ondisk_log_size":319,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":362848,"kb_used_data":13696,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578976,"statfs":{"total":1151940427776,"available":1151568871424,"internally_reserved":0,"allocated":14024704,"data_stored":7628913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43609,"num_objects":67,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":24,"num_read_kb":58,"num_write":161,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005686"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.539107+0000","last_change":"2024-03-18T01:34:56.661330+0000","last_active":"2024-03-18T01:35:02.539107+0000","last_peered":"2024-03-18T01:35:02.539107+0000","last_clean":"2024-03-18T01:35:02.539107+0000","last_became_active":"2024-03-18T01:34:56.660949+0000","last_became_peered":"2024-03-18T01:34:56.660949+0000","last_unstale":"2024-03-18T01:35:02.539107+0000","last_undegraded":"2024-03-18T01:35:02.539107+0000","last_fullsized":"2024-03-18T01:35:02.539107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:40.962487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123262+0000","last_change":"2024-03-18T01:34:56.652417+0000","last_active":"2024-03-18T01:35:02.123262+0000","last_peered":"2024-03-18T01:35:02.123262+0000","last_clean":"2024-03-18T01:35:02.123262+0000","last_became_active":"2024-03-18T01:34:56.651851+0000","last_became_peered":"2024-03-18T01:34:56.651851+0000","last_unstale":"2024-03-18T01:35:02.123262+0000","last_undegraded":"2024-03-18T01:35:02.123262+0000","last_fullsized":"2024-03-18T01:35:02.123262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:14:24.172177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122940+0000","last_change":"2024-03-18T01:34:56.647657+0000","last_active":"2024-03-18T01:35:02.122940+0000","last_peered":"2024-03-18T01:35:02.122940+0000","last_clean":"2024-03-18T01:35:02.122940+0000","last_became_active":"2024-03-18T01:34:56.647241+0000","last_became_peered":"2024-03-18T01:34:56.647241+0000","last_unstale":"2024-03-18T01:35:02.122940+0000","last_undegraded":"2024-03-18T01:35:02.122940+0000","last_fullsized":"2024-03-18T01:35:02.122940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:01.706351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583945+0000","last_change":"2024-03-18T01:34:56.649915+0000","last_active":"2024-03-18T01:35:03.583945+0000","last_peered":"2024-03-18T01:35:03.583945+0000","last_clean":"2024-03-18T01:35:03.583945+0000","last_became_active":"2024-03-18T01:34:56.648974+0000","last_became_peered":"2024-03-18T01:34:56.648974+0000","last_unstale":"2024-03-18T01:35:03.583945+0000","last_undegraded":"2024-03-18T01:35:03.583945+0000","last_fullsized":"2024-03-18T01:35:03.583945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:14:35.055677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280299+0000","last_change":"2024-03-18T01:34:56.643992+0000","last_active":"2024-03-18T01:35:02.280299+0000","last_peered":"2024-03-18T01:35:02.280299+0000","last_clean":"2024-03-18T01:35:02.280299+0000","last_became_active":"2024-03-18T01:34:56.643652+0000","last_became_peered":"2024-03-18T01:34:56.643652+0000","last_unstale":"2024-03-18T01:35:02.280299+0000","last_undegraded":"2024-03-18T01:35:02.280299+0000","last_fullsized":"2024-03-18T01:35:02.280299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:12.005357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.509019+0000","last_change":"2024-03-18T01:34:56.649955+0000","last_active":"2024-03-18T01:35:02.509019+0000","last_peered":"2024-03-18T01:35:02.509019+0000","last_clean":"2024-03-18T01:35:02.509019+0000","last_became_active":"2024-03-18T01:34:56.649609+0000","last_became_peered":"2024-03-18T01:34:56.649609+0000","last_unstale":"2024-03-18T01:35:02.509019+0000","last_undegraded":"2024-03-18T01:35:02.509019+0000","last_fullsized":"2024-03-18T01:35:02.509019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:34:28.113733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.118910+0000","last_change":"2024-03-18T01:34:56.647561+0000","last_active":"2024-03-18T01:35:02.118910+0000","last_peered":"2024-03-18T01:35:02.118910+0000","last_clean":"2024-03-18T01:35:02.118910+0000","last_became_active":"2024-03-18T01:34:56.647040+0000","last_became_peered":"2024-03-18T01:34:56.647040+0000","last_unstale":"2024-03-18T01:35:02.118910+0000","last_undegraded":"2024-03-18T01:35:02.118910+0000","last_fullsized":"2024-03-18T01:35:02.118910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:42.433751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584851+0000","last_change":"2024-03-18T01:34:56.652073+0000","last_active":"2024-03-18T01:35:03.584851+0000","last_peered":"2024-03-18T01:35:03.584851+0000","last_clean":"2024-03-18T01:35:03.584851+0000","last_became_active":"2024-03-18T01:34:56.651748+0000","last_became_peered":"2024-03-18T01:34:56.651748+0000","last_unstale":"2024-03-18T01:35:03.584851+0000","last_undegraded":"2024-03-18T01:35:03.584851+0000","last_fullsized":"2024-03-18T01:35:03.584851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:33.148823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083665+0000","last_change":"2024-03-18T01:34:56.645782+0000","last_active":"2024-03-18T01:35:03.083665+0000","last_peered":"2024-03-18T01:35:03.083665+0000","last_clean":"2024-03-18T01:35:03.083665+0000","last_became_active":"2024-03-18T01:34:56.645439+0000","last_became_peered":"2024-03-18T01:34:56.645439+0000","last_unstale":"2024-03-18T01:35:03.083665+0000","last_undegraded":"2024-03-18T01:35:03.083665+0000","last_fullsized":"2024-03-18T01:35:03.083665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:40.242348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.509354+0000","last_change":"2024-03-18T01:34:56.639697+0000","last_active":"2024-03-18T01:35:02.509354+0000","last_peered":"2024-03-18T01:35:02.509354+0000","last_clean":"2024-03-18T01:35:02.509354+0000","last_became_active":"2024-03-18T01:34:56.639339+0000","last_became_peered":"2024-03-18T01:34:56.639339+0000","last_unstale":"2024-03-18T01:35:02.509354+0000","last_undegraded":"2024-03-18T01:35:02.509354+0000","last_fullsized":"2024-03-18T01:35:02.509354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:25:22.741437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119467+0000","last_change":"2024-03-18T01:34:56.655341+0000","last_active":"2024-03-18T01:35:02.119467+0000","last_peered":"2024-03-18T01:35:02.119467+0000","last_clean":"2024-03-18T01:35:02.119467+0000","last_became_active":"2024-03-18T01:34:56.655015+0000","last_became_peered":"2024-03-18T01:34:56.655015+0000","last_unstale":"2024-03-18T01:35:02.119467+0000","last_undegraded":"2024-03-18T01:35:02.119467+0000","last_fullsized":"2024-03-18T01:35:02.119467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:50.766000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583886+0000","last_change":"2024-03-18T01:34:56.648399+0000","last_active":"2024-03-18T01:35:03.583886+0000","last_peered":"2024-03-18T01:35:03.583886+0000","last_clean":"2024-03-18T01:35:03.583886+0000","last_became_active":"2024-03-18T01:34:56.648042+0000","last_became_peered":"2024-03-18T01:34:56.648042+0000","last_unstale":"2024-03-18T01:35:03.583886+0000","last_undegraded":"2024-03-18T01:35:03.583886+0000","last_fullsized":"2024-03-18T01:35:03.583886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:04.972230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588010+0000","last_change":"2024-03-18T01:34:56.653465+0000","last_active":"2024-03-18T01:35:03.588010+0000","last_peered":"2024-03-18T01:35:03.588010+0000","last_clean":"2024-03-18T01:35:03.588010+0000","last_became_active":"2024-03-18T01:34:56.651183+0000","last_became_peered":"2024-03-18T01:34:56.651183+0000","last_unstale":"2024-03-18T01:35:03.588010+0000","last_undegraded":"2024-03-18T01:35:03.588010+0000","last_fullsized":"2024-03-18T01:35:03.588010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:57.983330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122010+0000","last_change":"2024-03-18T01:34:56.658845+0000","last_active":"2024-03-18T01:35:02.122010+0000","last_peered":"2024-03-18T01:35:02.122010+0000","last_clean":"2024-03-18T01:35:02.122010+0000","last_became_active":"2024-03-18T01:34:56.658435+0000","last_became_peered":"2024-03-18T01:34:56.658435+0000","last_unstale":"2024-03-18T01:35:02.122010+0000","last_undegraded":"2024-03-18T01:35:02.122010+0000","last_fullsized":"2024-03-18T01:35:02.122010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:02.176325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122050+0000","last_change":"2024-03-18T01:34:56.660752+0000","last_active":"2024-03-18T01:35:02.122050+0000","last_peered":"2024-03-18T01:35:02.122050+0000","last_clean":"2024-03-18T01:35:02.122050+0000","last_became_active":"2024-03-18T01:34:56.660378+0000","last_became_peered":"2024-03-18T01:34:56.660378+0000","last_unstale":"2024-03-18T01:35:02.122050+0000","last_undegraded":"2024-03-18T01:35:02.122050+0000","last_fullsized":"2024-03-18T01:35:02.122050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:13.143481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083064+0000","last_change":"2024-03-18T01:34:56.656589+0000","last_active":"2024-03-18T01:35:03.083064+0000","last_peered":"2024-03-18T01:35:03.083064+0000","last_clean":"2024-03-18T01:35:03.083064+0000","last_became_active":"2024-03-18T01:34:56.656184+0000","last_became_peered":"2024-03-18T01:34:56.656184+0000","last_unstale":"2024-03-18T01:35:03.083064+0000","last_undegraded":"2024-03-18T01:35:03.083064+0000","last_fullsized":"2024-03-18T01:35:03.083064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:56.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538934+0000","last_change":"2024-03-18T01:34:56.661327+0000","last_active":"2024-03-18T01:35:02.538934+0000","last_peered":"2024-03-18T01:35:02.538934+0000","last_clean":"2024-03-18T01:35:02.538934+0000","last_became_active":"2024-03-18T01:34:56.660767+0000","last_became_peered":"2024-03-18T01:34:56.660767+0000","last_unstale":"2024-03-18T01:35:02.538934+0000","last_undegraded":"2024-03-18T01:35:02.538934+0000","last_fullsized":"2024-03-18T01:35:02.538934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:07:59.066028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587752+0000","last_change":"2024-03-18T01:34:56.650149+0000","last_active":"2024-03-18T01:35:03.587752+0000","last_peered":"2024-03-18T01:35:03.587752+0000","last_clean":"2024-03-18T01:35:03.587752+0000","last_became_active":"2024-03-18T01:34:56.649780+0000","last_became_peered":"2024-03-18T01:34:56.649780+0000","last_unstale":"2024-03-18T01:35:03.587752+0000","last_undegraded":"2024-03-18T01:35:03.587752+0000","last_fullsized":"2024-03-18T01:35:03.587752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:41.605563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584451+0000","last_change":"2024-03-18T01:34:56.653919+0000","last_active":"2024-03-18T01:35:03.584451+0000","last_peered":"2024-03-18T01:35:03.584451+0000","last_clean":"2024-03-18T01:35:03.584451+0000","last_became_active":"2024-03-18T01:34:56.653512+0000","last_became_peered":"2024-03-18T01:34:56.653512+0000","last_unstale":"2024-03-18T01:35:03.584451+0000","last_undegraded":"2024-03-18T01:35:03.584451+0000","last_fullsized":"2024-03-18T01:35:03.584451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:27.419518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584519+0000","last_change":"2024-03-18T01:35:01.149999+0000","last_active":"2024-03-18T01:35:03.584519+0000","last_peered":"2024-03-18T01:35:03.584519+0000","last_clean":"2024-03-18T01:35:03.584519+0000","last_became_active":"2024-03-18T01:35:01.149603+0000","last_became_peered":"2024-03-18T01:35:01.149603+0000","last_unstale":"2024-03-18T01:35:03.584519+0000","last_undegraded":"2024-03-18T01:35:03.584519+0000","last_fullsized":"2024-03-18T01:35:03.584519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:37.619517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.643907+0000","last_change":"2024-03-18T01:34:55.650404+0000","last_active":"2024-03-18T01:35:30.643907+0000","last_peered":"2024-03-18T01:35:30.643907+0000","last_clean":"2024-03-18T01:35:30.643907+0000","last_became_active":"2024-03-18T01:34:55.650068+0000","last_became_peered":"2024-03-18T01:34:55.650068+0000","last_unstale":"2024-03-18T01:35:30.643907+0000","last_undegraded":"2024-03-18T01:35:30.643907+0000","last_fullsized":"2024-03-18T01:35:30.643907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:11.396529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585520+0000","last_change":"2024-03-18T01:34:56.654265+0000","last_active":"2024-03-18T01:35:03.585520+0000","last_peered":"2024-03-18T01:35:03.585520+0000","last_clean":"2024-03-18T01:35:03.585520+0000","last_became_active":"2024-03-18T01:34:56.653886+0000","last_became_peered":"2024-03-18T01:34:56.653886+0000","last_unstale":"2024-03-18T01:35:03.585520+0000","last_undegraded":"2024-03-18T01:35:03.585520+0000","last_fullsized":"2024-03-18T01:35:03.585520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:01:44.069966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583483+0000","last_change":"2024-03-18T01:35:01.155838+0000","last_active":"2024-03-18T01:35:03.583483+0000","last_peered":"2024-03-18T01:35:03.583483+0000","last_clean":"2024-03-18T01:35:03.583483+0000","last_became_active":"2024-03-18T01:35:01.155391+0000","last_became_peered":"2024-03-18T01:35:01.155391+0000","last_unstale":"2024-03-18T01:35:03.583483+0000","last_undegraded":"2024-03-18T01:35:03.583483+0000","last_fullsized":"2024-03-18T01:35:03.583483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:31.885767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585557+0000","last_change":"2024-03-18T01:34:55.658830+0000","last_active":"2024-03-18T01:35:03.585557+0000","last_peered":"2024-03-18T01:35:03.585557+0000","last_clean":"2024-03-18T01:35:03.585557+0000","last_became_active":"2024-03-18T01:34:55.658395+0000","last_became_peered":"2024-03-18T01:34:55.658395+0000","last_unstale":"2024-03-18T01:35:03.585557+0000","last_undegraded":"2024-03-18T01:35:03.585557+0000","last_fullsized":"2024-03-18T01:35:03.585557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:07.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123797+0000","last_change":"2024-03-18T01:34:56.656874+0000","last_active":"2024-03-18T01:35:02.123797+0000","last_peered":"2024-03-18T01:35:02.123797+0000","last_clean":"2024-03-18T01:35:02.123797+0000","last_became_active":"2024-03-18T01:34:56.656525+0000","last_became_peered":"2024-03-18T01:34:56.656525+0000","last_unstale":"2024-03-18T01:35:02.123797+0000","last_undegraded":"2024-03-18T01:35:02.123797+0000","last_fullsized":"2024-03-18T01:35:02.123797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:35:19.363586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583620+0000","last_change":"2024-03-18T01:35:01.153190+0000","last_active":"2024-03-18T01:35:03.583620+0000","last_peered":"2024-03-18T01:35:03.583620+0000","last_clean":"2024-03-18T01:35:03.583620+0000","last_became_active":"2024-03-18T01:35:01.152439+0000","last_became_peered":"2024-03-18T01:35:01.152439+0000","last_unstale":"2024-03-18T01:35:03.583620+0000","last_undegraded":"2024-03-18T01:35:03.583620+0000","last_fullsized":"2024-03-18T01:35:03.583620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:08:40.294721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'11","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.370316+0000","last_change":"2024-03-18T01:34:55.648218+0000","last_active":"2024-03-18T01:35:30.370316+0000","last_peered":"2024-03-18T01:35:30.370316+0000","last_clean":"2024-03-18T01:35:30.370316+0000","last_became_active":"2024-03-18T01:34:55.647779+0000","last_became_peered":"2024-03-18T01:34:55.647779+0000","last_unstale":"2024-03-18T01:35:30.370316+0000","last_undegraded":"2024-03-18T01:35:30.370316+0000","last_fullsized":"2024-03-18T01:35:30.370316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:22:08.136973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083384+0000","last_change":"2024-03-18T01:34:56.649900+0000","last_active":"2024-03-18T01:35:03.083384+0000","last_peered":"2024-03-18T01:35:03.083384+0000","last_clean":"2024-03-18T01:35:03.083384+0000","last_became_active":"2024-03-18T01:34:56.649564+0000","last_became_peered":"2024-03-18T01:34:56.649564+0000","last_unstale":"2024-03-18T01:35:03.083384+0000","last_undegraded":"2024-03-18T01:35:03.083384+0000","last_fullsized":"2024-03-18T01:35:03.083384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:29.861861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583496+0000","last_change":"2024-03-18T01:35:01.147062+0000","last_active":"2024-03-18T01:35:03.583496+0000","last_peered":"2024-03-18T01:35:03.583496+0000","last_clean":"2024-03-18T01:35:03.583496+0000","last_became_active":"2024-03-18T01:35:01.146666+0000","last_became_peered":"2024-03-18T01:35:01.146666+0000","last_unstale":"2024-03-18T01:35:03.583496+0000","last_undegraded":"2024-03-18T01:35:03.583496+0000","last_fullsized":"2024-03-18T01:35:03.583496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:15:21.433263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.369728+0000","last_change":"2024-03-18T01:34:55.645512+0000","last_active":"2024-03-18T01:35:30.369728+0000","last_peered":"2024-03-18T01:35:30.369728+0000","last_clean":"2024-03-18T01:35:30.369728+0000","last_became_active":"2024-03-18T01:34:55.645155+0000","last_became_peered":"2024-03-18T01:34:55.645155+0000","last_unstale":"2024-03-18T01:35:30.369728+0000","last_undegraded":"2024-03-18T01:35:30.369728+0000","last_fullsized":"2024-03-18T01:35:30.369728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:32:17.049289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543742+0000","last_change":"2024-03-18T01:34:56.650449+0000","last_active":"2024-03-18T01:35:03.543742+0000","last_peered":"2024-03-18T01:35:03.543742+0000","last_clean":"2024-03-18T01:35:03.543742+0000","last_became_active":"2024-03-18T01:34:56.650103+0000","last_became_peered":"2024-03-18T01:34:56.650103+0000","last_unstale":"2024-03-18T01:35:03.543742+0000","last_undegraded":"2024-03-18T01:35:03.543742+0000","last_fullsized":"2024-03-18T01:35:03.543742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:07:55.003592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584327+0000","last_change":"2024-03-18T01:35:01.132227+0000","last_active":"2024-03-18T01:35:03.584327+0000","last_peered":"2024-03-18T01:35:03.584327+0000","last_clean":"2024-03-18T01:35:03.584327+0000","last_became_active":"2024-03-18T01:35:01.131767+0000","last_became_peered":"2024-03-18T01:35:01.131767+0000","last_unstale":"2024-03-18T01:35:03.584327+0000","last_undegraded":"2024-03-18T01:35:03.584327+0000","last_fullsized":"2024-03-18T01:35:03.584327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:54.335482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.340349+0000","last_change":"2024-03-18T01:34:55.649277+0000","last_active":"2024-03-18T01:35:27.340349+0000","last_peered":"2024-03-18T01:35:27.340349+0000","last_clean":"2024-03-18T01:35:27.340349+0000","last_became_active":"2024-03-18T01:34:55.648917+0000","last_became_peered":"2024-03-18T01:34:55.648917+0000","last_unstale":"2024-03-18T01:35:27.340349+0000","last_undegraded":"2024-03-18T01:35:27.340349+0000","last_fullsized":"2024-03-18T01:35:27.340349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:00.117647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280348+0000","last_change":"2024-03-18T01:34:56.648394+0000","last_active":"2024-03-18T01:35:02.280348+0000","last_peered":"2024-03-18T01:35:02.280348+0000","last_clean":"2024-03-18T01:35:02.280348+0000","last_became_active":"2024-03-18T01:34:56.648043+0000","last_became_peered":"2024-03-18T01:34:56.648043+0000","last_unstale":"2024-03-18T01:35:02.280348+0000","last_undegraded":"2024-03-18T01:35:02.280348+0000","last_fullsized":"2024-03-18T01:35:02.280348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:54:52.531851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584358+0000","last_change":"2024-03-18T01:35:01.151346+0000","last_active":"2024-03-18T01:35:03.584358+0000","last_peered":"2024-03-18T01:35:03.584358+0000","last_clean":"2024-03-18T01:35:03.584358+0000","last_became_active":"2024-03-18T01:35:01.150916+0000","last_became_peered":"2024-03-18T01:35:01.150916+0000","last_unstale":"2024-03-18T01:35:03.584358+0000","last_undegraded":"2024-03-18T01:35:03.584358+0000","last_fullsized":"2024-03-18T01:35:03.584358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:44:36.000983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.364785+0000","last_change":"2024-03-18T01:34:55.648498+0000","last_active":"2024-03-18T01:35:29.364785+0000","last_peered":"2024-03-18T01:35:29.364785+0000","last_clean":"2024-03-18T01:35:29.364785+0000","last_became_active":"2024-03-18T01:34:55.648152+0000","last_became_peered":"2024-03-18T01:34:55.648152+0000","last_unstale":"2024-03-18T01:35:29.364785+0000","last_undegraded":"2024-03-18T01:35:29.364785+0000","last_fullsized":"2024-03-18T01:35:29.364785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:33.138620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543666+0000","last_change":"2024-03-18T01:34:56.651420+0000","last_active":"2024-03-18T01:35:03.543666+0000","last_peered":"2024-03-18T01:35:03.543666+0000","last_clean":"2024-03-18T01:35:03.543666+0000","last_became_active":"2024-03-18T01:34:56.650909+0000","last_became_peered":"2024-03-18T01:34:56.650909+0000","last_unstale":"2024-03-18T01:35:03.543666+0000","last_undegraded":"2024-03-18T01:35:03.543666+0000","last_fullsized":"2024-03-18T01:35:03.543666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:43:40.548970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.539037+0000","last_change":"2024-03-18T01:35:01.157759+0000","last_active":"2024-03-18T01:35:02.539037+0000","last_peered":"2024-03-18T01:35:02.539037+0000","last_clean":"2024-03-18T01:35:02.539037+0000","last_became_active":"2024-03-18T01:35:01.157288+0000","last_became_peered":"2024-03-18T01:35:01.157288+0000","last_unstale":"2024-03-18T01:35:02.539037+0000","last_undegraded":"2024-03-18T01:35:02.539037+0000","last_fullsized":"2024-03-18T01:35:02.539037+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:05:41.272727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:28.353189+0000","last_change":"2024-03-18T01:34:55.650388+0000","last_active":"2024-03-18T01:35:28.353189+0000","last_peered":"2024-03-18T01:35:28.353189+0000","last_clean":"2024-03-18T01:35:28.353189+0000","last_became_active":"2024-03-18T01:34:55.650055+0000","last_became_peered":"2024-03-18T01:34:55.650055+0000","last_unstale":"2024-03-18T01:35:28.353189+0000","last_undegraded":"2024-03-18T01:35:28.353189+0000","last_fullsized":"2024-03-18T01:35:28.353189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:29:40.758561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584675+0000","last_change":"2024-03-18T01:34:56.658634+0000","last_active":"2024-03-18T01:35:03.584675+0000","last_peered":"2024-03-18T01:35:03.584675+0000","last_clean":"2024-03-18T01:35:03.584675+0000","last_became_active":"2024-03-18T01:34:56.657196+0000","last_became_peered":"2024-03-18T01:34:56.657196+0000","last_unstale":"2024-03-18T01:35:03.584675+0000","last_undegraded":"2024-03-18T01:35:03.584675+0000","last_fullsized":"2024-03-18T01:35:03.584675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:12:36.265949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123736+0000","last_change":"2024-03-18T01:35:01.150431+0000","last_active":"2024-03-18T01:35:02.123736+0000","last_peered":"2024-03-18T01:35:02.123736+0000","last_clean":"2024-03-18T01:35:02.123736+0000","last_became_active":"2024-03-18T01:35:01.149981+0000","last_became_peered":"2024-03-18T01:35:01.149981+0000","last_unstale":"2024-03-18T01:35:02.123736+0000","last_undegraded":"2024-03-18T01:35:02.123736+0000","last_fullsized":"2024-03-18T01:35:02.123736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:34:55.371860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543316+0000","last_change":"2024-03-18T01:34:55.649346+0000","last_active":"2024-03-18T01:35:03.543316+0000","last_peered":"2024-03-18T01:35:03.543316+0000","last_clean":"2024-03-18T01:35:03.543316+0000","last_became_active":"2024-03-18T01:34:55.648958+0000","last_became_peered":"2024-03-18T01:34:55.648958+0000","last_unstale":"2024-03-18T01:35:03.543316+0000","last_undegraded":"2024-03-18T01:35:03.543316+0000","last_fullsized":"2024-03-18T01:35:03.543316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:53.788081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.586327+0000","last_change":"2024-03-18T01:34:56.653525+0000","last_active":"2024-03-18T01:35:03.586327+0000","last_peered":"2024-03-18T01:35:03.586327+0000","last_clean":"2024-03-18T01:35:03.586327+0000","last_became_active":"2024-03-18T01:34:56.653218+0000","last_became_peered":"2024-03-18T01:34:56.653218+0000","last_unstale":"2024-03-18T01:35:03.586327+0000","last_undegraded":"2024-03-18T01:35:03.586327+0000","last_fullsized":"2024-03-18T01:35:03.586327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:32:37.479782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584927+0000","last_change":"2024-03-18T01:35:01.135505+0000","last_active":"2024-03-18T01:35:03.584927+0000","last_peered":"2024-03-18T01:35:03.584927+0000","last_clean":"2024-03-18T01:35:03.584927+0000","last_became_active":"2024-03-18T01:35:01.135055+0000","last_became_peered":"2024-03-18T01:35:01.135055+0000","last_unstale":"2024-03-18T01:35:03.584927+0000","last_undegraded":"2024-03-18T01:35:03.584927+0000","last_fullsized":"2024-03-18T01:35:03.584927+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:52:35.069313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:32.646768+0000","last_change":"2024-03-18T01:34:55.659389+0000","last_active":"2024-03-18T01:35:32.646768+0000","last_peered":"2024-03-18T01:35:32.646768+0000","last_clean":"2024-03-18T01:35:32.646768+0000","last_became_active":"2024-03-18T01:34:55.659090+0000","last_became_peered":"2024-03-18T01:34:55.659090+0000","last_unstale":"2024-03-18T01:35:32.646768+0000","last_undegraded":"2024-03-18T01:35:32.646768+0000","last_fullsized":"2024-03-18T01:35:32.646768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:55:51.731788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083673+0000","last_change":"2024-03-18T01:34:56.650411+0000","last_active":"2024-03-18T01:35:03.083673+0000","last_peered":"2024-03-18T01:35:03.083673+0000","last_clean":"2024-03-18T01:35:03.083673+0000","last_became_active":"2024-03-18T01:34:56.650036+0000","last_became_peered":"2024-03-18T01:34:56.650036+0000","last_unstale":"2024-03-18T01:35:03.083673+0000","last_undegraded":"2024-03-18T01:35:03.083673+0000","last_fullsized":"2024-03-18T01:35:03.083673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:59.912207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.586383+0000","last_change":"2024-03-18T01:35:01.149186+0000","last_active":"2024-03-18T01:35:03.586383+0000","last_peered":"2024-03-18T01:35:03.586383+0000","last_clean":"2024-03-18T01:35:03.586383+0000","last_became_active":"2024-03-18T01:35:01.148721+0000","last_became_peered":"2024-03-18T01:35:01.148721+0000","last_unstale":"2024-03-18T01:35:03.586383+0000","last_undegraded":"2024-03-18T01:35:03.586383+0000","last_fullsized":"2024-03-18T01:35:03.586383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:09:35.509348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.369730+0000","last_change":"2024-03-18T01:34:55.647879+0000","last_active":"2024-03-18T01:35:30.369730+0000","last_peered":"2024-03-18T01:35:30.369730+0000","last_clean":"2024-03-18T01:35:30.369730+0000","last_became_active":"2024-03-18T01:34:55.647549+0000","last_became_peered":"2024-03-18T01:34:55.647549+0000","last_unstale":"2024-03-18T01:35:30.369730+0000","last_undegraded":"2024-03-18T01:35:30.369730+0000","last_fullsized":"2024-03-18T01:35:30.369730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:12.261729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123791+0000","last_change":"2024-03-18T01:34:56.664303+0000","last_active":"2024-03-18T01:35:02.123791+0000","last_peered":"2024-03-18T01:35:02.123791+0000","last_clean":"2024-03-18T01:35:02.123791+0000","last_became_active":"2024-03-18T01:34:56.662597+0000","last_became_peered":"2024-03-18T01:34:56.662597+0000","last_unstale":"2024-03-18T01:35:02.123791+0000","last_undegraded":"2024-03-18T01:35:02.123791+0000","last_fullsized":"2024-03-18T01:35:02.123791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:54:19.581671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583776+0000","last_change":"2024-03-18T01:35:01.153448+0000","last_active":"2024-03-18T01:35:03.583776+0000","last_peered":"2024-03-18T01:35:03.583776+0000","last_clean":"2024-03-18T01:35:03.583776+0000","last_became_active":"2024-03-18T01:35:01.152593+0000","last_became_peered":"2024-03-18T01:35:01.152593+0000","last_unstale":"2024-03-18T01:35:03.583776+0000","last_undegraded":"2024-03-18T01:35:03.583776+0000","last_fullsized":"2024-03-18T01:35:03.583776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:46.363662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543662+0000","last_change":"2024-03-18T01:34:55.655079+0000","last_active":"2024-03-18T01:35:03.543662+0000","last_peered":"2024-03-18T01:35:03.543662+0000","last_clean":"2024-03-18T01:35:03.543662+0000","last_became_active":"2024-03-18T01:34:55.654702+0000","last_became_peered":"2024-03-18T01:34:55.654702+0000","last_unstale":"2024-03-18T01:35:03.543662+0000","last_undegraded":"2024-03-18T01:35:03.543662+0000","last_fullsized":"2024-03-18T01:35:03.543662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:50:46.524614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083490+0000","last_change":"2024-03-18T01:34:56.644441+0000","last_active":"2024-03-18T01:35:03.083490+0000","last_peered":"2024-03-18T01:35:03.083490+0000","last_clean":"2024-03-18T01:35:03.083490+0000","last_became_active":"2024-03-18T01:34:56.643924+0000","last_became_peered":"2024-03-18T01:34:56.643924+0000","last_unstale":"2024-03-18T01:35:03.083490+0000","last_undegraded":"2024-03-18T01:35:03.083490+0000","last_fullsized":"2024-03-18T01:35:03.083490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:44.260300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.586491+0000","last_change":"2024-03-18T01:35:01.150022+0000","last_active":"2024-03-18T01:35:03.586491+0000","last_peered":"2024-03-18T01:35:03.586491+0000","last_clean":"2024-03-18T01:35:03.586491+0000","last_became_active":"2024-03-18T01:35:01.149510+0000","last_became_peered":"2024-03-18T01:35:01.149510+0000","last_unstale":"2024-03-18T01:35:03.586491+0000","last_undegraded":"2024-03-18T01:35:03.586491+0000","last_fullsized":"2024-03-18T01:35:03.586491+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:40:36.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'26","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:32.649982+0000","last_change":"2024-03-18T01:34:55.642397+0000","last_active":"2024-03-18T01:35:32.649982+0000","last_peered":"2024-03-18T01:35:32.649982+0000","last_clean":"2024-03-18T01:35:32.649982+0000","last_became_active":"2024-03-18T01:34:55.642046+0000","last_became_peered":"2024-03-18T01:34:55.642046+0000","last_unstale":"2024-03-18T01:35:32.649982+0000","last_undegraded":"2024-03-18T01:35:32.649982+0000","last_fullsized":"2024-03-18T01:35:32.649982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:48:31.631414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41906,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":45,"num_write":28,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280585+0000","last_change":"2024-03-18T01:34:56.644388+0000","last_active":"2024-03-18T01:35:02.280585+0000","last_peered":"2024-03-18T01:35:02.280585+0000","last_clean":"2024-03-18T01:35:02.280585+0000","last_became_active":"2024-03-18T01:34:56.644014+0000","last_became_peered":"2024-03-18T01:34:56.644014+0000","last_unstale":"2024-03-18T01:35:02.280585+0000","last_undegraded":"2024-03-18T01:35:02.280585+0000","last_fullsized":"2024-03-18T01:35:02.280585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:06:18.193600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.544060+0000","last_change":"2024-03-18T01:35:01.148796+0000","last_active":"2024-03-18T01:35:03.544060+0000","last_peered":"2024-03-18T01:35:03.544060+0000","last_clean":"2024-03-18T01:35:03.544060+0000","last_became_active":"2024-03-18T01:35:01.148329+0000","last_became_peered":"2024-03-18T01:35:01.148329+0000","last_unstale":"2024-03-18T01:35:03.544060+0000","last_undegraded":"2024-03-18T01:35:03.544060+0000","last_fullsized":"2024-03-18T01:35:03.544060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:14:36.688878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.642536+0000","last_change":"2024-03-18T01:34:55.649600+0000","last_active":"2024-03-18T01:35:30.642536+0000","last_peered":"2024-03-18T01:35:30.642536+0000","last_clean":"2024-03-18T01:35:30.642536+0000","last_became_active":"2024-03-18T01:34:55.649192+0000","last_became_peered":"2024-03-18T01:34:55.649192+0000","last_unstale":"2024-03-18T01:35:30.642536+0000","last_undegraded":"2024-03-18T01:35:30.642536+0000","last_fullsized":"2024-03-18T01:35:30.642536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:55.294501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588223+0000","last_change":"2024-03-18T01:34:56.657786+0000","last_active":"2024-03-18T01:35:03.588223+0000","last_peered":"2024-03-18T01:35:03.588223+0000","last_clean":"2024-03-18T01:35:03.588223+0000","last_became_active":"2024-03-18T01:34:56.657432+0000","last_became_peered":"2024-03-18T01:34:56.657432+0000","last_unstale":"2024-03-18T01:35:03.588223+0000","last_undegraded":"2024-03-18T01:35:03.588223+0000","last_fullsized":"2024-03-18T01:35:03.588223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:13:39.498562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538899+0000","last_change":"2024-03-18T01:35:01.150495+0000","last_active":"2024-03-18T01:35:02.538899+0000","last_peered":"2024-03-18T01:35:02.538899+0000","last_clean":"2024-03-18T01:35:02.538899+0000","last_became_active":"2024-03-18T01:35:01.150033+0000","last_became_peered":"2024-03-18T01:35:01.150033+0000","last_unstale":"2024-03-18T01:35:02.538899+0000","last_undegraded":"2024-03-18T01:35:02.538899+0000","last_fullsized":"2024-03-18T01:35:02.538899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:38.433827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.340026+0000","last_change":"2024-03-18T01:34:55.658820+0000","last_active":"2024-03-18T01:35:27.340026+0000","last_peered":"2024-03-18T01:35:27.340026+0000","last_clean":"2024-03-18T01:35:27.340026+0000","last_became_active":"2024-03-18T01:34:55.658446+0000","last_became_peered":"2024-03-18T01:34:55.658446+0000","last_unstale":"2024-03-18T01:35:27.340026+0000","last_undegraded":"2024-03-18T01:35:27.340026+0000","last_fullsized":"2024-03-18T01:35:27.340026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:19:21.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119186+0000","last_change":"2024-03-18T01:34:56.652890+0000","last_active":"2024-03-18T01:35:02.119186+0000","last_peered":"2024-03-18T01:35:02.119186+0000","last_clean":"2024-03-18T01:35:02.119186+0000","last_became_active":"2024-03-18T01:34:56.652444+0000","last_became_peered":"2024-03-18T01:34:56.652444+0000","last_unstale":"2024-03-18T01:35:02.119186+0000","last_undegraded":"2024-03-18T01:35:02.119186+0000","last_fullsized":"2024-03-18T01:35:02.119186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:58:47.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587675+0000","last_change":"2024-03-18T01:35:01.154700+0000","last_active":"2024-03-18T01:35:03.587675+0000","last_peered":"2024-03-18T01:35:03.587675+0000","last_clean":"2024-03-18T01:35:03.587675+0000","last_became_active":"2024-03-18T01:35:01.148453+0000","last_became_peered":"2024-03-18T01:35:01.148453+0000","last_unstale":"2024-03-18T01:35:03.587675+0000","last_undegraded":"2024-03-18T01:35:03.587675+0000","last_fullsized":"2024-03-18T01:35:03.587675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:08:40.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.359311+0000","last_change":"2024-03-18T01:34:55.643861+0000","last_active":"2024-03-18T01:35:29.359311+0000","last_peered":"2024-03-18T01:35:29.359311+0000","last_clean":"2024-03-18T01:35:29.359311+0000","last_became_active":"2024-03-18T01:34:55.643352+0000","last_became_peered":"2024-03-18T01:34:55.643352+0000","last_unstale":"2024-03-18T01:35:29.359311+0000","last_undegraded":"2024-03-18T01:35:29.359311+0000","last_fullsized":"2024-03-18T01:35:29.359311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:56:53.685571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.118990+0000","last_change":"2024-03-18T01:34:56.646533+0000","last_active":"2024-03-18T01:35:02.118990+0000","last_peered":"2024-03-18T01:35:02.118990+0000","last_clean":"2024-03-18T01:35:02.118990+0000","last_became_active":"2024-03-18T01:34:56.646180+0000","last_became_peered":"2024-03-18T01:34:56.646180+0000","last_unstale":"2024-03-18T01:35:02.118990+0000","last_undegraded":"2024-03-18T01:35:02.118990+0000","last_fullsized":"2024-03-18T01:35:02.118990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:43:58.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588967+0000","last_change":"2024-03-18T01:35:01.158762+0000","last_active":"2024-03-18T01:35:03.588967+0000","last_peered":"2024-03-18T01:35:03.588967+0000","last_clean":"2024-03-18T01:35:03.588967+0000","last_became_active":"2024-03-18T01:35:01.158281+0000","last_became_peered":"2024-03-18T01:35:01.158281+0000","last_unstale":"2024-03-18T01:35:03.588967+0000","last_undegraded":"2024-03-18T01:35:03.588967+0000","last_fullsized":"2024-03-18T01:35:03.588967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:32:19.681361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.362929+0000","last_change":"2024-03-18T01:34:55.658245+0000","last_active":"2024-03-18T01:35:29.362929+0000","last_peered":"2024-03-18T01:35:29.362929+0000","last_clean":"2024-03-18T01:35:29.362929+0000","last_became_active":"2024-03-18T01:34:55.657916+0000","last_became_peered":"2024-03-18T01:34:55.657916+0000","last_unstale":"2024-03-18T01:35:29.362929+0000","last_undegraded":"2024-03-18T01:35:29.362929+0000","last_fullsized":"2024-03-18T01:35:29.362929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:18:49.207721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543317+0000","last_change":"2024-03-18T01:34:56.648646+0000","last_active":"2024-03-18T01:35:03.543317+0000","last_peered":"2024-03-18T01:35:03.543317+0000","last_clean":"2024-03-18T01:35:03.543317+0000","last_became_active":"2024-03-18T01:34:56.648257+0000","last_became_peered":"2024-03-18T01:34:56.648257+0000","last_unstale":"2024-03-18T01:35:03.543317+0000","last_undegraded":"2024-03-18T01:35:03.543317+0000","last_fullsized":"2024-03-18T01:35:03.543317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:53.788445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543374+0000","last_change":"2024-03-18T01:35:01.136912+0000","last_active":"2024-03-18T01:35:03.543374+0000","last_peered":"2024-03-18T01:35:03.543374+0000","last_clean":"2024-03-18T01:35:03.543374+0000","last_became_active":"2024-03-18T01:35:01.136611+0000","last_became_peered":"2024-03-18T01:35:01.136611+0000","last_unstale":"2024-03-18T01:35:03.543374+0000","last_undegraded":"2024-03-18T01:35:03.543374+0000","last_fullsized":"2024-03-18T01:35:03.543374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:31.089605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280837+0000","last_change":"2024-03-18T01:34:55.648345+0000","last_active":"2024-03-18T01:35:02.280837+0000","last_peered":"2024-03-18T01:35:02.280837+0000","last_clean":"2024-03-18T01:35:02.280837+0000","last_became_active":"2024-03-18T01:34:55.647113+0000","last_became_peered":"2024-03-18T01:34:55.647113+0000","last_unstale":"2024-03-18T01:35:02.280837+0000","last_undegraded":"2024-03-18T01:35:02.280837+0000","last_fullsized":"2024-03-18T01:35:02.280837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:20.904129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538825+0000","last_change":"2024-03-18T01:34:56.649147+0000","last_active":"2024-03-18T01:35:02.538825+0000","last_peered":"2024-03-18T01:35:02.538825+0000","last_clean":"2024-03-18T01:35:02.538825+0000","last_became_active":"2024-03-18T01:34:56.648815+0000","last_became_peered":"2024-03-18T01:34:56.648815+0000","last_unstale":"2024-03-18T01:35:02.538825+0000","last_undegraded":"2024-03-18T01:35:02.538825+0000","last_fullsized":"2024-03-18T01:35:02.538825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:53.560679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280436+0000","last_change":"2024-03-18T01:35:01.156778+0000","last_active":"2024-03-18T01:35:02.280436+0000","last_peered":"2024-03-18T01:35:02.280436+0000","last_clean":"2024-03-18T01:35:02.280436+0000","last_became_active":"2024-03-18T01:35:01.156367+0000","last_became_peered":"2024-03-18T01:35:01.156367+0000","last_unstale":"2024-03-18T01:35:02.280436+0000","last_undegraded":"2024-03-18T01:35:02.280436+0000","last_fullsized":"2024-03-18T01:35:02.280436+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:46.223915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587230+0000","last_change":"2024-03-18T01:34:55.658722+0000","last_active":"2024-03-18T01:35:03.587230+0000","last_peered":"2024-03-18T01:35:03.587230+0000","last_clean":"2024-03-18T01:35:03.587230+0000","last_became_active":"2024-03-18T01:34:55.658321+0000","last_became_peered":"2024-03-18T01:34:55.658321+0000","last_unstale":"2024-03-18T01:35:03.587230+0000","last_undegraded":"2024-03-18T01:35:03.587230+0000","last_fullsized":"2024-03-18T01:35:03.587230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:05.753226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.509200+0000","last_change":"2024-03-18T01:34:56.649415+0000","last_active":"2024-03-18T01:35:02.509200+0000","last_peered":"2024-03-18T01:35:02.509200+0000","last_clean":"2024-03-18T01:35:02.509200+0000","last_became_active":"2024-03-18T01:34:56.649019+0000","last_became_peered":"2024-03-18T01:34:56.649019+0000","last_unstale":"2024-03-18T01:35:02.509200+0000","last_undegraded":"2024-03-18T01:35:02.509200+0000","last_fullsized":"2024-03-18T01:35:02.509200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:28.473683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588723+0000","last_change":"2024-03-18T01:35:01.148937+0000","last_active":"2024-03-18T01:35:03.588723+0000","last_peered":"2024-03-18T01:35:03.588723+0000","last_clean":"2024-03-18T01:35:03.588723+0000","last_became_active":"2024-03-18T01:35:01.148415+0000","last_became_peered":"2024-03-18T01:35:01.148415+0000","last_unstale":"2024-03-18T01:35:03.588723+0000","last_undegraded":"2024-03-18T01:35:03.588723+0000","last_fullsized":"2024-03-18T01:35:03.588723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:45.926970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.645164+0000","last_change":"2024-03-18T01:34:55.653443+0000","last_active":"2024-03-18T01:35:30.645164+0000","last_peered":"2024-03-18T01:35:30.645164+0000","last_clean":"2024-03-18T01:35:30.645164+0000","last_became_active":"2024-03-18T01:34:55.653072+0000","last_became_peered":"2024-03-18T01:34:55.653072+0000","last_unstale":"2024-03-18T01:35:30.645164+0000","last_undegraded":"2024-03-18T01:35:30.645164+0000","last_fullsized":"2024-03-18T01:35:30.645164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:26.376636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.509045+0000","last_change":"2024-03-18T01:34:56.647373+0000","last_active":"2024-03-18T01:35:02.509045+0000","last_peered":"2024-03-18T01:35:02.509045+0000","last_clean":"2024-03-18T01:35:02.509045+0000","last_became_active":"2024-03-18T01:34:56.646986+0000","last_became_peered":"2024-03-18T01:34:56.646986+0000","last_unstale":"2024-03-18T01:35:02.509045+0000","last_undegraded":"2024-03-18T01:35:02.509045+0000","last_fullsized":"2024-03-18T01:35:02.509045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:09:00.829251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588905+0000","last_change":"2024-03-18T01:35:01.152767+0000","last_active":"2024-03-18T01:35:03.588905+0000","last_peered":"2024-03-18T01:35:03.588905+0000","last_clean":"2024-03-18T01:35:03.588905+0000","last_became_active":"2024-03-18T01:35:01.150619+0000","last_became_peered":"2024-03-18T01:35:01.150619+0000","last_unstale":"2024-03-18T01:35:03.588905+0000","last_undegraded":"2024-03-18T01:35:03.588905+0000","last_fullsized":"2024-03-18T01:35:03.588905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:57:28.258251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543419+0000","last_change":"2024-03-18T01:34:56.650541+0000","last_active":"2024-03-18T01:35:03.543419+0000","last_peered":"2024-03-18T01:35:03.543419+0000","last_clean":"2024-03-18T01:35:03.543419+0000","last_became_active":"2024-03-18T01:34:56.650072+0000","last_became_peered":"2024-03-18T01:34:56.650072+0000","last_unstale":"2024-03-18T01:35:03.543419+0000","last_undegraded":"2024-03-18T01:35:03.543419+0000","last_fullsized":"2024-03-18T01:35:03.543419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:51.224883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:28.355820+0000","last_change":"2024-03-18T01:34:55.664297+0000","last_active":"2024-03-18T01:35:28.355820+0000","last_peered":"2024-03-18T01:35:28.355820+0000","last_clean":"2024-03-18T01:35:28.355820+0000","last_became_active":"2024-03-18T01:34:55.663938+0000","last_became_peered":"2024-03-18T01:34:55.663938+0000","last_unstale":"2024-03-18T01:35:28.355820+0000","last_undegraded":"2024-03-18T01:35:28.355820+0000","last_fullsized":"2024-03-18T01:35:28.355820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:51.771987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119459+0000","last_change":"2024-03-18T01:35:01.141736+0000","last_active":"2024-03-18T01:35:02.119459+0000","last_peered":"2024-03-18T01:35:02.119459+0000","last_clean":"2024-03-18T01:35:02.119459+0000","last_became_active":"2024-03-18T01:35:01.141381+0000","last_became_peered":"2024-03-18T01:35:01.141381+0000","last_unstale":"2024-03-18T01:35:02.119459+0000","last_undegraded":"2024-03-18T01:35:02.119459+0000","last_fullsized":"2024-03-18T01:35:02.119459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:25:04.190250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584951+0000","last_change":"2024-03-18T01:34:56.666439+0000","last_active":"2024-03-18T01:35:03.584951+0000","last_peered":"2024-03-18T01:35:03.584951+0000","last_clean":"2024-03-18T01:35:03.584951+0000","last_became_active":"2024-03-18T01:34:56.664556+0000","last_became_peered":"2024-03-18T01:34:56.664556+0000","last_unstale":"2024-03-18T01:35:03.584951+0000","last_undegraded":"2024-03-18T01:35:03.584951+0000","last_fullsized":"2024-03-18T01:35:03.584951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:08:48.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.358418+0000","last_change":"2024-03-18T01:34:55.656484+0000","last_active":"2024-03-18T01:35:29.358418+0000","last_peered":"2024-03-18T01:35:29.358418+0000","last_clean":"2024-03-18T01:35:29.358418+0000","last_became_active":"2024-03-18T01:34:55.656125+0000","last_became_peered":"2024-03-18T01:34:55.656125+0000","last_unstale":"2024-03-18T01:35:29.358418+0000","last_undegraded":"2024-03-18T01:35:29.358418+0000","last_fullsized":"2024-03-18T01:35:29.358418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:58:25.944403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.118836+0000","last_change":"2024-03-18T01:33:35.824651+0000","last_active":"2024-03-18T01:35:02.118836+0000","last_peered":"2024-03-18T01:35:02.118836+0000","last_clean":"2024-03-18T01:35:02.118836+0000","last_became_active":"2024-03-18T01:33:35.824106+0000","last_became_peered":"2024-03-18T01:33:35.824106+0000","last_unstale":"2024-03-18T01:35:02.118836+0000","last_undegraded":"2024-03-18T01:35:02.118836+0000","last_fullsized":"2024-03-18T01:35:02.118836+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_clean_scrub_stamp":"2024-03-18T01:29:41.724134+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:06.422316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543884+0000","last_change":"2024-03-18T01:35:01.157735+0000","last_active":"2024-03-18T01:35:03.543884+0000","last_peered":"2024-03-18T01:35:03.543884+0000","last_clean":"2024-03-18T01:35:03.543884+0000","last_became_active":"2024-03-18T01:35:01.156954+0000","last_became_peered":"2024-03-18T01:35:01.156954+0000","last_unstale":"2024-03-18T01:35:03.543884+0000","last_undegraded":"2024-03-18T01:35:03.543884+0000","last_fullsized":"2024-03-18T01:35:03.543884+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:00.106809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.586697+0000","last_change":"2024-03-18T01:34:56.646808+0000","last_active":"2024-03-18T01:35:03.586697+0000","last_peered":"2024-03-18T01:35:03.586697+0000","last_clean":"2024-03-18T01:35:03.586697+0000","last_became_active":"2024-03-18T01:34:56.646437+0000","last_became_peered":"2024-03-18T01:34:56.646437+0000","last_unstale":"2024-03-18T01:35:03.586697+0000","last_undegraded":"2024-03-18T01:35:03.586697+0000","last_fullsized":"2024-03-18T01:35:03.586697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:46:09.703889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:28.349311+0000","last_change":"2024-03-18T01:34:55.645365+0000","last_active":"2024-03-18T01:35:28.349311+0000","last_peered":"2024-03-18T01:35:28.349311+0000","last_clean":"2024-03-18T01:35:28.349311+0000","last_became_active":"2024-03-18T01:34:55.645018+0000","last_became_peered":"2024-03-18T01:34:55.645018+0000","last_unstale":"2024-03-18T01:35:28.349311+0000","last_undegraded":"2024-03-18T01:35:28.349311+0000","last_fullsized":"2024-03-18T01:35:28.349311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:08:31.869328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.539005+0000","last_change":"2024-03-18T01:35:01.147294+0000","last_active":"2024-03-18T01:35:02.539005+0000","last_peered":"2024-03-18T01:35:02.539005+0000","last_clean":"2024-03-18T01:35:02.539005+0000","last_became_active":"2024-03-18T01:35:01.146858+0000","last_became_peered":"2024-03-18T01:35:01.146858+0000","last_unstale":"2024-03-18T01:35:02.539005+0000","last_undegraded":"2024-03-18T01:35:02.539005+0000","last_fullsized":"2024-03-18T01:35:02.539005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:45:24.010746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.364151+0000","last_change":"2024-03-18T01:34:55.658110+0000","last_active":"2024-03-18T01:35:29.364151+0000","last_peered":"2024-03-18T01:35:29.364151+0000","last_clean":"2024-03-18T01:35:29.364151+0000","last_became_active":"2024-03-18T01:34:55.657802+0000","last_became_peered":"2024-03-18T01:34:55.657802+0000","last_unstale":"2024-03-18T01:35:29.364151+0000","last_undegraded":"2024-03-18T01:35:29.364151+0000","last_fullsized":"2024-03-18T01:35:29.364151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:34.264115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588758+0000","last_change":"2024-03-18T01:34:56.656689+0000","last_active":"2024-03-18T01:35:03.588758+0000","last_peered":"2024-03-18T01:35:03.588758+0000","last_clean":"2024-03-18T01:35:03.588758+0000","last_became_active":"2024-03-18T01:34:56.655653+0000","last_became_peered":"2024-03-18T01:34:56.655653+0000","last_unstale":"2024-03-18T01:35:03.588758+0000","last_undegraded":"2024-03-18T01:35:03.588758+0000","last_fullsized":"2024-03-18T01:35:03.588758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:58.370904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584710+0000","last_change":"2024-03-18T01:35:01.137419+0000","last_active":"2024-03-18T01:35:03.584710+0000","last_peered":"2024-03-18T01:35:03.584710+0000","last_clean":"2024-03-18T01:35:03.584710+0000","last_became_active":"2024-03-18T01:35:01.137033+0000","last_became_peered":"2024-03-18T01:35:01.137033+0000","last_unstale":"2024-03-18T01:35:03.584710+0000","last_undegraded":"2024-03-18T01:35:03.584710+0000","last_fullsized":"2024-03-18T01:35:03.584710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:13:48.382329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.365780+0000","last_change":"2024-03-18T01:34:55.663134+0000","last_active":"2024-03-18T01:35:29.365780+0000","last_peered":"2024-03-18T01:35:29.365780+0000","last_clean":"2024-03-18T01:35:29.365780+0000","last_became_active":"2024-03-18T01:34:55.662083+0000","last_became_peered":"2024-03-18T01:34:55.662083+0000","last_unstale":"2024-03-18T01:35:29.365780+0000","last_undegraded":"2024-03-18T01:35:29.365780+0000","last_fullsized":"2024-03-18T01:35:29.365780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:01:09.186599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585802+0000","last_change":"2024-03-18T01:34:56.664761+0000","last_active":"2024-03-18T01:35:03.585802+0000","last_peered":"2024-03-18T01:35:03.585802+0000","last_clean":"2024-03-18T01:35:03.585802+0000","last_became_active":"2024-03-18T01:34:56.663657+0000","last_became_peered":"2024-03-18T01:34:56.663657+0000","last_unstale":"2024-03-18T01:35:03.585802+0000","last_undegraded":"2024-03-18T01:35:03.585802+0000","last_fullsized":"2024-03-18T01:35:03.585802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:26:53.075000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584043+0000","last_change":"2024-03-18T01:35:01.133002+0000","last_active":"2024-03-18T01:35:03.584043+0000","last_peered":"2024-03-18T01:35:03.584043+0000","last_clean":"2024-03-18T01:35:03.584043+0000","last_became_active":"2024-03-18T01:35:01.132542+0000","last_became_peered":"2024-03-18T01:35:01.132542+0000","last_unstale":"2024-03-18T01:35:03.584043+0000","last_undegraded":"2024-03-18T01:35:03.584043+0000","last_fullsized":"2024-03-18T01:35:03.584043+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:23.823409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.342714+0000","last_change":"2024-03-18T01:34:55.648924+0000","last_active":"2024-03-18T01:35:27.342714+0000","last_peered":"2024-03-18T01:35:27.342714+0000","last_clean":"2024-03-18T01:35:27.342714+0000","last_became_active":"2024-03-18T01:34:55.648548+0000","last_became_peered":"2024-03-18T01:34:55.648548+0000","last_unstale":"2024-03-18T01:35:27.342714+0000","last_undegraded":"2024-03-18T01:35:27.342714+0000","last_fullsized":"2024-03-18T01:35:27.342714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:26:38.916856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123556+0000","last_change":"2024-03-18T01:34:56.663142+0000","last_active":"2024-03-18T01:35:02.123556+0000","last_peered":"2024-03-18T01:35:02.123556+0000","last_clean":"2024-03-18T01:35:02.123556+0000","last_became_active":"2024-03-18T01:34:56.660256+0000","last_became_peered":"2024-03-18T01:34:56.660256+0000","last_unstale":"2024-03-18T01:35:02.123556+0000","last_undegraded":"2024-03-18T01:35:02.123556+0000","last_fullsized":"2024-03-18T01:35:02.123556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:59.503277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583806+0000","last_change":"2024-03-18T01:35:01.136186+0000","last_active":"2024-03-18T01:35:03.583806+0000","last_peered":"2024-03-18T01:35:03.583806+0000","last_clean":"2024-03-18T01:35:03.583806+0000","last_became_active":"2024-03-18T01:35:01.135803+0000","last_became_peered":"2024-03-18T01:35:01.135803+0000","last_unstale":"2024-03-18T01:35:03.583806+0000","last_undegraded":"2024-03-18T01:35:03.583806+0000","last_fullsized":"2024-03-18T01:35:03.583806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:20.027448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.341363+0000","last_change":"2024-03-18T01:34:55.663090+0000","last_active":"2024-03-18T01:35:27.341363+0000","last_peered":"2024-03-18T01:35:27.341363+0000","last_clean":"2024-03-18T01:35:27.341363+0000","last_became_active":"2024-03-18T01:34:55.662774+0000","last_became_peered":"2024-03-18T01:34:55.662774+0000","last_unstale":"2024-03-18T01:35:27.341363+0000","last_undegraded":"2024-03-18T01:35:27.341363+0000","last_fullsized":"2024-03-18T01:35:27.341363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:30:35.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280200+0000","last_change":"2024-03-18T01:34:56.639509+0000","last_active":"2024-03-18T01:35:02.280200+0000","last_peered":"2024-03-18T01:35:02.280200+0000","last_clean":"2024-03-18T01:35:02.280200+0000","last_became_active":"2024-03-18T01:34:56.639144+0000","last_became_peered":"2024-03-18T01:34:56.639144+0000","last_unstale":"2024-03-18T01:35:02.280200+0000","last_undegraded":"2024-03-18T01:35:02.280200+0000","last_fullsized":"2024-03-18T01:35:02.280200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:12:36.220793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587941+0000","last_change":"2024-03-18T01:35:01.141762+0000","last_active":"2024-03-18T01:35:03.587941+0000","last_peered":"2024-03-18T01:35:03.587941+0000","last_clean":"2024-03-18T01:35:03.587941+0000","last_became_active":"2024-03-18T01:35:01.141219+0000","last_became_peered":"2024-03-18T01:35:01.141219+0000","last_unstale":"2024-03-18T01:35:03.587941+0000","last_undegraded":"2024-03-18T01:35:03.587941+0000","last_fullsized":"2024-03-18T01:35:03.587941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:55.839120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.373964+0000","last_change":"2024-03-18T01:34:55.662530+0000","last_active":"2024-03-18T01:35:30.373964+0000","last_peered":"2024-03-18T01:35:30.373964+0000","last_clean":"2024-03-18T01:35:30.373964+0000","last_became_active":"2024-03-18T01:34:55.661178+0000","last_became_peered":"2024-03-18T01:34:55.661178+0000","last_unstale":"2024-03-18T01:35:30.373964+0000","last_undegraded":"2024-03-18T01:35:30.373964+0000","last_fullsized":"2024-03-18T01:35:30.373964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:47:16.678553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538541+0000","last_change":"2024-03-18T01:34:56.660936+0000","last_active":"2024-03-18T01:35:02.538541+0000","last_peered":"2024-03-18T01:35:02.538541+0000","last_clean":"2024-03-18T01:35:02.538541+0000","last_became_active":"2024-03-18T01:34:56.660513+0000","last_became_peered":"2024-03-18T01:34:56.660513+0000","last_unstale":"2024-03-18T01:35:02.538541+0000","last_undegraded":"2024-03-18T01:35:02.538541+0000","last_fullsized":"2024-03-18T01:35:02.538541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:35:21.037366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583793+0000","last_change":"2024-03-18T01:35:01.136914+0000","last_active":"2024-03-18T01:35:03.583793+0000","last_peered":"2024-03-18T01:35:03.583793+0000","last_clean":"2024-03-18T01:35:03.583793+0000","last_became_active":"2024-03-18T01:35:01.136480+0000","last_became_peered":"2024-03-18T01:35:01.136480+0000","last_unstale":"2024-03-18T01:35:03.583793+0000","last_undegraded":"2024-03-18T01:35:03.583793+0000","last_fullsized":"2024-03-18T01:35:03.583793+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:49.262255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.364701+0000","last_change":"2024-03-18T01:34:55.654938+0000","last_active":"2024-03-18T01:35:29.364701+0000","last_peered":"2024-03-18T01:35:29.364701+0000","last_clean":"2024-03-18T01:35:29.364701+0000","last_became_active":"2024-03-18T01:34:55.654611+0000","last_became_peered":"2024-03-18T01:34:55.654611+0000","last_unstale":"2024-03-18T01:35:29.364701+0000","last_undegraded":"2024-03-18T01:35:29.364701+0000","last_fullsized":"2024-03-18T01:35:29.364701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:23.210205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585405+0000","last_change":"2024-03-18T01:34:56.650731+0000","last_active":"2024-03-18T01:35:03.585405+0000","last_peered":"2024-03-18T01:35:03.585405+0000","last_clean":"2024-03-18T01:35:03.585405+0000","last_became_active":"2024-03-18T01:34:56.650363+0000","last_became_peered":"2024-03-18T01:34:56.650363+0000","last_unstale":"2024-03-18T01:35:03.585405+0000","last_undegraded":"2024-03-18T01:35:03.585405+0000","last_fullsized":"2024-03-18T01:35:03.585405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:09:50.305019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583792+0000","last_change":"2024-03-18T01:35:01.135101+0000","last_active":"2024-03-18T01:35:03.583792+0000","last_peered":"2024-03-18T01:35:03.583792+0000","last_clean":"2024-03-18T01:35:03.583792+0000","last_became_active":"2024-03-18T01:35:01.134690+0000","last_became_peered":"2024-03-18T01:35:01.134690+0000","last_unstale":"2024-03-18T01:35:03.583792+0000","last_undegraded":"2024-03-18T01:35:03.583792+0000","last_fullsized":"2024-03-18T01:35:03.583792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:33.381629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.339513+0000","last_change":"2024-03-18T01:34:55.648108+0000","last_active":"2024-03-18T01:35:27.339513+0000","last_peered":"2024-03-18T01:35:27.339513+0000","last_clean":"2024-03-18T01:35:27.339513+0000","last_became_active":"2024-03-18T01:34:55.647754+0000","last_became_peered":"2024-03-18T01:34:55.647754+0000","last_unstale":"2024-03-18T01:35:27.339513+0000","last_undegraded":"2024-03-18T01:35:27.339513+0000","last_fullsized":"2024-03-18T01:35:27.339513+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:05:48.508015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119098+0000","last_change":"2024-03-18T01:34:56.651589+0000","last_active":"2024-03-18T01:35:02.119098+0000","last_peered":"2024-03-18T01:35:02.119098+0000","last_clean":"2024-03-18T01:35:02.119098+0000","last_became_active":"2024-03-18T01:34:56.650929+0000","last_became_peered":"2024-03-18T01:34:56.650929+0000","last_unstale":"2024-03-18T01:35:02.119098+0000","last_undegraded":"2024-03-18T01:35:02.119098+0000","last_fullsized":"2024-03-18T01:35:02.119098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:43.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083112+0000","last_change":"2024-03-18T01:35:01.148347+0000","last_active":"2024-03-18T01:35:03.083112+0000","last_peered":"2024-03-18T01:35:03.083112+0000","last_clean":"2024-03-18T01:35:03.083112+0000","last_became_active":"2024-03-18T01:35:01.147936+0000","last_became_peered":"2024-03-18T01:35:01.147936+0000","last_unstale":"2024-03-18T01:35:03.083112+0000","last_undegraded":"2024-03-18T01:35:03.083112+0000","last_fullsized":"2024-03-18T01:35:03.083112+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:12.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.645660+0000","last_change":"2024-03-18T01:34:55.653625+0000","last_active":"2024-03-18T01:35:30.645660+0000","last_peered":"2024-03-18T01:35:30.645660+0000","last_clean":"2024-03-18T01:35:30.645660+0000","last_became_active":"2024-03-18T01:34:55.653186+0000","last_became_peered":"2024-03-18T01:34:55.653186+0000","last_unstale":"2024-03-18T01:35:30.645660+0000","last_undegraded":"2024-03-18T01:35:30.645660+0000","last_fullsized":"2024-03-18T01:35:30.645660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:10:51.193296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119569+0000","last_change":"2024-03-18T01:34:56.652703+0000","last_active":"2024-03-18T01:35:02.119569+0000","last_peered":"2024-03-18T01:35:02.119569+0000","last_clean":"2024-03-18T01:35:02.119569+0000","last_became_active":"2024-03-18T01:34:56.651768+0000","last_became_peered":"2024-03-18T01:34:56.651768+0000","last_unstale":"2024-03-18T01:35:02.119569+0000","last_undegraded":"2024-03-18T01:35:02.119569+0000","last_fullsized":"2024-03-18T01:35:02.119569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:59.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122297+0000","last_change":"2024-03-18T01:35:01.144900+0000","last_active":"2024-03-18T01:35:02.122297+0000","last_peered":"2024-03-18T01:35:02.122297+0000","last_clean":"2024-03-18T01:35:02.122297+0000","last_became_active":"2024-03-18T01:35:01.144431+0000","last_became_peered":"2024-03-18T01:35:01.144431+0000","last_unstale":"2024-03-18T01:35:02.122297+0000","last_undegraded":"2024-03-18T01:35:02.122297+0000","last_fullsized":"2024-03-18T01:35:02.122297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:36.199904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543786+0000","last_change":"2024-03-18T01:34:55.661259+0000","last_active":"2024-03-18T01:35:03.543786+0000","last_peered":"2024-03-18T01:35:03.543786+0000","last_clean":"2024-03-18T01:35:03.543786+0000","last_became_active":"2024-03-18T01:34:55.660964+0000","last_became_peered":"2024-03-18T01:34:55.660964+0000","last_unstale":"2024-03-18T01:35:03.543786+0000","last_undegraded":"2024-03-18T01:35:03.543786+0000","last_fullsized":"2024-03-18T01:35:03.543786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:14.723905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119306+0000","last_change":"2024-03-18T01:34:56.663075+0000","last_active":"2024-03-18T01:35:02.119306+0000","last_peered":"2024-03-18T01:35:02.119306+0000","last_clean":"2024-03-18T01:35:02.119306+0000","last_became_active":"2024-03-18T01:34:56.662458+0000","last_became_peered":"2024-03-18T01:34:56.662458+0000","last_unstale":"2024-03-18T01:35:02.119306+0000","last_undegraded":"2024-03-18T01:35:02.119306+0000","last_fullsized":"2024-03-18T01:35:02.119306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:31:31.837493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543170+0000","last_change":"2024-03-18T01:35:01.133317+0000","last_active":"2024-03-18T01:35:03.543170+0000","last_peered":"2024-03-18T01:35:03.543170+0000","last_clean":"2024-03-18T01:35:03.543170+0000","last_became_active":"2024-03-18T01:35:01.131798+0000","last_became_peered":"2024-03-18T01:35:01.131798+0000","last_unstale":"2024-03-18T01:35:03.543170+0000","last_undegraded":"2024-03-18T01:35:03.543170+0000","last_fullsized":"2024-03-18T01:35:03.543170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:55.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588762+0000","last_change":"2024-03-18T01:34:55.660887+0000","last_active":"2024-03-18T01:35:03.588762+0000","last_peered":"2024-03-18T01:35:03.588762+0000","last_clean":"2024-03-18T01:35:03.588762+0000","last_became_active":"2024-03-18T01:34:55.660524+0000","last_became_peered":"2024-03-18T01:34:55.660524+0000","last_unstale":"2024-03-18T01:35:03.588762+0000","last_undegraded":"2024-03-18T01:35:03.588762+0000","last_fullsized":"2024-03-18T01:35:03.588762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:24.816951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119483+0000","last_change":"2024-03-18T01:34:56.654107+0000","last_active":"2024-03-18T01:35:02.119483+0000","last_peered":"2024-03-18T01:35:02.119483+0000","last_clean":"2024-03-18T01:35:02.119483+0000","last_became_active":"2024-03-18T01:34:56.653742+0000","last_became_peered":"2024-03-18T01:34:56.653742+0000","last_unstale":"2024-03-18T01:35:02.119483+0000","last_undegraded":"2024-03-18T01:35:02.119483+0000","last_fullsized":"2024-03-18T01:35:02.119483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:58:46.093589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538656+0000","last_change":"2024-03-18T01:34:56.647818+0000","last_active":"2024-03-18T01:35:02.538656+0000","last_peered":"2024-03-18T01:35:02.538656+0000","last_clean":"2024-03-18T01:35:02.538656+0000","last_became_active":"2024-03-18T01:34:56.647174+0000","last_became_peered":"2024-03-18T01:34:56.647174+0000","last_unstale":"2024-03-18T01:35:02.538656+0000","last_undegraded":"2024-03-18T01:35:02.538656+0000","last_fullsized":"2024-03-18T01:35:02.538656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:55.656083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584569+0000","last_change":"2024-03-18T01:34:56.643221+0000","last_active":"2024-03-18T01:35:03.584569+0000","last_peered":"2024-03-18T01:35:03.584569+0000","last_clean":"2024-03-18T01:35:03.584569+0000","last_became_active":"2024-03-18T01:34:56.642871+0000","last_became_peered":"2024-03-18T01:34:56.642871+0000","last_unstale":"2024-03-18T01:35:03.584569+0000","last_undegraded":"2024-03-18T01:35:03.584569+0000","last_fullsized":"2024-03-18T01:35:03.584569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:23:25.106060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584067+0000","last_change":"2024-03-18T01:34:56.656227+0000","last_active":"2024-03-18T01:35:03.584067+0000","last_peered":"2024-03-18T01:35:03.584067+0000","last_clean":"2024-03-18T01:35:03.584067+0000","last_became_active":"2024-03-18T01:34:56.655859+0000","last_became_peered":"2024-03-18T01:34:56.655859+0000","last_unstale":"2024-03-18T01:35:03.584067+0000","last_undegraded":"2024-03-18T01:35:03.584067+0000","last_fullsized":"2024-03-18T01:35:03.584067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:51:54.424342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122708+0000","last_change":"2024-03-18T01:34:56.664655+0000","last_active":"2024-03-18T01:35:02.122708+0000","last_peered":"2024-03-18T01:35:02.122708+0000","last_clean":"2024-03-18T01:35:02.122708+0000","last_became_active":"2024-03-18T01:34:56.662526+0000","last_became_peered":"2024-03-18T01:34:56.662526+0000","last_unstale":"2024-03-18T01:35:02.122708+0000","last_undegraded":"2024-03-18T01:35:02.122708+0000","last_fullsized":"2024-03-18T01:35:02.122708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:45:06.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588782+0000","last_change":"2024-03-18T01:34:56.657045+0000","last_active":"2024-03-18T01:35:03.588782+0000","last_peered":"2024-03-18T01:35:03.588782+0000","last_clean":"2024-03-18T01:35:03.588782+0000","last_became_active":"2024-03-18T01:34:56.656339+0000","last_became_peered":"2024-03-18T01:34:56.656339+0000","last_unstale":"2024-03-18T01:35:03.588782+0000","last_undegraded":"2024-03-18T01:35:03.588782+0000","last_fullsized":"2024-03-18T01:35:03.588782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:41.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538976+0000","last_change":"2024-03-18T01:34:56.647626+0000","last_active":"2024-03-18T01:35:02.538976+0000","last_peered":"2024-03-18T01:35:02.538976+0000","last_clean":"2024-03-18T01:35:02.538976+0000","last_became_active":"2024-03-18T01:34:56.647216+0000","last_became_peered":"2024-03-18T01:34:56.647216+0000","last_unstale":"2024-03-18T01:35:02.538976+0000","last_undegraded":"2024-03-18T01:35:02.538976+0000","last_fullsized":"2024-03-18T01:35:02.538976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:12:33.330925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584700+0000","last_change":"2024-03-18T01:34:56.645091+0000","last_active":"2024-03-18T01:35:03.584700+0000","last_peered":"2024-03-18T01:35:03.584700+0000","last_clean":"2024-03-18T01:35:03.584700+0000","last_became_active":"2024-03-18T01:34:56.644725+0000","last_became_peered":"2024-03-18T01:34:56.644725+0000","last_unstale":"2024-03-18T01:35:03.584700+0000","last_undegraded":"2024-03-18T01:35:03.584700+0000","last_fullsized":"2024-03-18T01:35:03.584700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585949+0000","last_change":"2024-03-18T01:34:56.651282+0000","last_active":"2024-03-18T01:35:03.585949+0000","last_peered":"2024-03-18T01:35:03.585949+0000","last_clean":"2024-03-18T01:35:03.585949+0000","last_became_active":"2024-03-18T01:34:56.650872+0000","last_became_peered":"2024-03-18T01:34:56.650872+0000","last_unstale":"2024-03-18T01:35:03.585949+0000","last_undegraded":"2024-03-18T01:35:03.585949+0000","last_fullsized":"2024-03-18T01:35:03.585949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:14:45.106848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543292+0000","last_change":"2024-03-18T01:34:56.651347+0000","last_active":"2024-03-18T01:35:03.543292+0000","last_peered":"2024-03-18T01:35:03.543292+0000","last_clean":"2024-03-18T01:35:03.543292+0000","last_became_active":"2024-03-18T01:34:56.650935+0000","last_became_peered":"2024-03-18T01:34:56.650935+0000","last_unstale":"2024-03-18T01:35:03.543292+0000","last_undegraded":"2024-03-18T01:35:03.543292+0000","last_fullsized":"2024-03-18T01:35:03.543292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:41.849883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584729+0000","last_change":"2024-03-18T01:34:56.667333+0000","last_active":"2024-03-18T01:35:03.584729+0000","last_peered":"2024-03-18T01:35:03.584729+0000","last_clean":"2024-03-18T01:35:03.584729+0000","last_became_active":"2024-03-18T01:34:56.666845+0000","last_became_peered":"2024-03-18T01:34:56.666845+0000","last_unstale":"2024-03-18T01:35:03.584729+0000","last_undegraded":"2024-03-18T01:35:03.584729+0000","last_fullsized":"2024-03-18T01:35:03.584729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:49:23.813455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083235+0000","last_change":"2024-03-18T01:34:56.662677+0000","last_active":"2024-03-18T01:35:03.083235+0000","last_peered":"2024-03-18T01:35:03.083235+0000","last_clean":"2024-03-18T01:35:03.083235+0000","last_became_active":"2024-03-18T01:34:56.662306+0000","last_became_peered":"2024-03-18T01:34:56.662306+0000","last_unstale":"2024-03-18T01:35:03.083235+0000","last_undegraded":"2024-03-18T01:35:03.083235+0000","last_fullsized":"2024-03-18T01:35:03.083235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:53:56.868984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587744+0000","last_change":"2024-03-18T01:34:56.657721+0000","last_active":"2024-03-18T01:35:03.587744+0000","last_peered":"2024-03-18T01:35:03.587744+0000","last_clean":"2024-03-18T01:35:03.587744+0000","last_became_active":"2024-03-18T01:34:56.657072+0000","last_became_peered":"2024-03-18T01:34:56.657072+0000","last_unstale":"2024-03-18T01:35:03.587744+0000","last_undegraded":"2024-03-18T01:35:03.587744+0000","last_fullsized":"2024-03-18T01:35:03.587744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:21.120391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538849+0000","last_change":"2024-03-18T01:34:56.647327+0000","last_active":"2024-03-18T01:35:02.538849+0000","last_peered":"2024-03-18T01:35:02.538849+0000","last_clean":"2024-03-18T01:35:02.538849+0000","last_became_active":"2024-03-18T01:34:56.646910+0000","last_became_peered":"2024-03-18T01:34:56.646910+0000","last_unstale":"2024-03-18T01:35:02.538849+0000","last_undegraded":"2024-03-18T01:35:02.538849+0000","last_fullsized":"2024-03-18T01:35:02.538849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:37:25.970272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":47991,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":24,"num_read_kb":58,"num_write":208,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":124772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28652,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716500,"statfs":{"total":95995035648,"available":95965696000,"internally_reserved":0,"allocated":1675264,"data_stored":1111616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":482433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070502,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1075212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":520873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":479393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":502328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1011712,"data_stored":477645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":477881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1605632,"data_stored":1069059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":43658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":25113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":44033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:35:33.612 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.863+0000 7fe625000640 1 -- 172.21.15.7:0/3756766854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe620103800 msgr2=0x7fe620103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.863+0000 7fe625000640 1 --2- 172.21.15.7:0/3756766854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe620103800 0x7fe620103c20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe610009900 tx=0x7fe61002bc40 comp rx=0 tx=0).stop 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.863+0000 7fe625000640 1 -- 172.21.15.7:0/3756766854 shutdown_connections 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.863+0000 7fe625000640 1 --2- 172.21.15.7:0/3756766854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6201053b0 0x7fe62010bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.864+0000 7fe625000640 1 --2- 172.21.15.7:0/3756766854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6201049f0 0x7fe620104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.864+0000 7fe625000640 1 --2- 172.21.15.7:0/3756766854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe620103800 0x7fe620103c20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.864+0000 7fe625000640 1 -- 172.21.15.7:0/3756766854 >> 172.21.15.7:0/3756766854 conn(0x7fe6200fed70 msgr2=0x7fe6201011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.864+0000 7fe625000640 1 -- 172.21.15.7:0/3756766854 shutdown_connections 2024-03-18T01:35:33.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.864+0000 7fe625000640 1 -- 172.21.15.7:0/3756766854 wait complete. 2024-03-18T01:35:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.865+0000 7fe625000640 1 Processor -- start 2024-03-18T01:35:33.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.865+0000 7fe625000640 1 -- start start 2024-03-18T01:35:33.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe625000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe620103800 0x7fe620199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:33.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe625000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6201049f0 0x7fe620199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:33.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe625000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201053b0 0x7fe6201a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:33.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe625000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe62010f4d0 con 0x7fe6201049f0 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe625000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe62010f350 con 0x7fe6201053b0 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe625000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe62010f650 con 0x7fe620103800 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe620103800 0x7fe620199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe620103800 0x7fe620199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52230/0 (socket says 172.21.15.7:52230) 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61f400640 1 -- 172.21.15.7:0/227354238 learned_addr learned my addr 172.21.15.7:0/227354238 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61fe00640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201053b0 0x7fe6201a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61ea00640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6201049f0 0x7fe620199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61f400640 1 -- 172.21.15.7:0/227354238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201053b0 msgr2=0x7fe6201a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61f400640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201053b0 0x7fe6201a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61f400640 1 -- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6201049f0 msgr2=0x7fe620199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61f400640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6201049f0 0x7fe620199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:33.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.866+0000 7fe61f400640 1 -- 172.21.15.7:0/227354238 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6201a1640 con 0x7fe620103800 2024-03-18T01:35:33.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.867+0000 7fe61f400640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe620103800 0x7fe620199a20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe61002efd0 tx=0x7fe61002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:33.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.867+0000 7fe617e00640 1 -- 172.21.15.7:0/227354238 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe610041070 con 0x7fe620103800 2024-03-18T01:35:33.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.867+0000 7fe617e00640 1 -- 172.21.15.7:0/227354238 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe610004450 con 0x7fe620103800 2024-03-18T01:35:33.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.867+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6201a18d0 con 0x7fe620103800 2024-03-18T01:35:33.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.867+0000 7fe617e00640 1 -- 172.21.15.7:0/227354238 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe610003e50 con 0x7fe620103800 2024-03-18T01:35:33.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.867+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6201a1d90 con 0x7fe620103800 2024-03-18T01:35:33.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.868+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe62010aa50 con 0x7fe620103800 2024-03-18T01:35:33.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.870+0000 7fe617e00640 1 -- 172.21.15.7:0/227354238 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fe610044070 con 0x7fe620103800 2024-03-18T01:35:33.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.871+0000 7fe617e00640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6040a2ae0 0x7fe6040a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:33.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.871+0000 7fe617e00640 1 -- 172.21.15.7:0/227354238 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe6100f0980 con 0x7fe620103800 2024-03-18T01:35:33.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.871+0000 7fe61ea00640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6040a2ae0 0x7fe6040a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:33.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.872+0000 7fe61ea00640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6040a2ae0 0x7fe6040a4fa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe62019af30 tx=0x7fe60c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:33.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:33.880+0000 7fe617e00640 1 -- 172.21.15.7:0/227354238 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe6100bd730 con 0x7fe620103800 2024-03-18T01:35:34.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:33 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1287299226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:34.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:33 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1287299226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:34.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.100+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe620061c20 con 0x7fe6040a2ae0 2024-03-18T01:35:34.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.121+0000 7fe617e00640 1 -- 172.21.15.7:0/227354238 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340397 (secure 0 0 0) 0x7fe620061c20 con 0x7fe6040a2ae0 2024-03-18T01:35:34.126 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:34.128 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-18T01:35:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.127+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6040a2ae0 msgr2=0x7fe6040a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.127+0000 7fe625000640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6040a2ae0 0x7fe6040a4fa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe62019af30 tx=0x7fe60c005f90 comp rx=0 tx=0).stop 2024-03-18T01:35:34.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.127+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe620103800 msgr2=0x7fe620199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.127+0000 7fe625000640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe620103800 0x7fe620199a20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe61002efd0 tx=0x7fe61002c660 comp rx=0 tx=0).stop 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.128+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 shutdown_connections 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.128+0000 7fe625000640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201053b0 0x7fe6201a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.128+0000 7fe625000640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe6040a2ae0 0x7fe6040a4fa0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.128+0000 7fe625000640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe6201049f0 0x7fe620199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.128+0000 7fe625000640 1 --2- 172.21.15.7:0/227354238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe620103800 0x7fe620199a20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.128+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 >> 172.21.15.7:0/227354238 conn(0x7fe6200fed70 msgr2=0x7fe620107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.128+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 shutdown_connections 2024-03-18T01:35:34.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:34.128+0000 7fe625000640 1 -- 172.21.15.7:0/227354238 wait complete. 2024-03-18T01:35:34.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:33 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1287299226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:34.150 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-03-18T01:35:33.180013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":638476,"num_objects":91,"num_object_clones":0,"num_object_copies":274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":150,"num_read_kb":167,"num_write":390,"num_write_kb":3035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":319,"ondisk_log_size":319,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":362848,"kb_used_data":13696,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578976,"statfs":{"total":1151940427776,"available":1151568871424,"internally_reserved":0,"allocated":14024704,"data_stored":7628913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":43609,"num_objects":67,"num_object_clones":0,"num_object_copies":202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":24,"num_read_kb":58,"num_write":161,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005686"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.539107+0000","last_change":"2024-03-18T01:34:56.661330+0000","last_active":"2024-03-18T01:35:02.539107+0000","last_peered":"2024-03-18T01:35:02.539107+0000","last_clean":"2024-03-18T01:35:02.539107+0000","last_became_active":"2024-03-18T01:34:56.660949+0000","last_became_peered":"2024-03-18T01:34:56.660949+0000","last_unstale":"2024-03-18T01:35:02.539107+0000","last_undegraded":"2024-03-18T01:35:02.539107+0000","last_fullsized":"2024-03-18T01:35:02.539107+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:40.962487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123262+0000","last_change":"2024-03-18T01:34:56.652417+0000","last_active":"2024-03-18T01:35:02.123262+0000","last_peered":"2024-03-18T01:35:02.123262+0000","last_clean":"2024-03-18T01:35:02.123262+0000","last_became_active":"2024-03-18T01:34:56.651851+0000","last_became_peered":"2024-03-18T01:34:56.651851+0000","last_unstale":"2024-03-18T01:35:02.123262+0000","last_undegraded":"2024-03-18T01:35:02.123262+0000","last_fullsized":"2024-03-18T01:35:02.123262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:14:24.172177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122940+0000","last_change":"2024-03-18T01:34:56.647657+0000","last_active":"2024-03-18T01:35:02.122940+0000","last_peered":"2024-03-18T01:35:02.122940+0000","last_clean":"2024-03-18T01:35:02.122940+0000","last_became_active":"2024-03-18T01:34:56.647241+0000","last_became_peered":"2024-03-18T01:34:56.647241+0000","last_unstale":"2024-03-18T01:35:02.122940+0000","last_undegraded":"2024-03-18T01:35:02.122940+0000","last_fullsized":"2024-03-18T01:35:02.122940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:01.706351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583945+0000","last_change":"2024-03-18T01:34:56.649915+0000","last_active":"2024-03-18T01:35:03.583945+0000","last_peered":"2024-03-18T01:35:03.583945+0000","last_clean":"2024-03-18T01:35:03.583945+0000","last_became_active":"2024-03-18T01:34:56.648974+0000","last_became_peered":"2024-03-18T01:34:56.648974+0000","last_unstale":"2024-03-18T01:35:03.583945+0000","last_undegraded":"2024-03-18T01:35:03.583945+0000","last_fullsized":"2024-03-18T01:35:03.583945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:14:35.055677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280299+0000","last_change":"2024-03-18T01:34:56.643992+0000","last_active":"2024-03-18T01:35:02.280299+0000","last_peered":"2024-03-18T01:35:02.280299+0000","last_clean":"2024-03-18T01:35:02.280299+0000","last_became_active":"2024-03-18T01:34:56.643652+0000","last_became_peered":"2024-03-18T01:34:56.643652+0000","last_unstale":"2024-03-18T01:35:02.280299+0000","last_undegraded":"2024-03-18T01:35:02.280299+0000","last_fullsized":"2024-03-18T01:35:02.280299+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:12.005357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.509019+0000","last_change":"2024-03-18T01:34:56.649955+0000","last_active":"2024-03-18T01:35:02.509019+0000","last_peered":"2024-03-18T01:35:02.509019+0000","last_clean":"2024-03-18T01:35:02.509019+0000","last_became_active":"2024-03-18T01:34:56.649609+0000","last_became_peered":"2024-03-18T01:34:56.649609+0000","last_unstale":"2024-03-18T01:35:02.509019+0000","last_undegraded":"2024-03-18T01:35:02.509019+0000","last_fullsized":"2024-03-18T01:35:02.509019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:34:28.113733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.118910+0000","last_change":"2024-03-18T01:34:56.647561+0000","last_active":"2024-03-18T01:35:02.118910+0000","last_peered":"2024-03-18T01:35:02.118910+0000","last_clean":"2024-03-18T01:35:02.118910+0000","last_became_active":"2024-03-18T01:34:56.647040+0000","last_became_peered":"2024-03-18T01:34:56.647040+0000","last_unstale":"2024-03-18T01:35:02.118910+0000","last_undegraded":"2024-03-18T01:35:02.118910+0000","last_fullsized":"2024-03-18T01:35:02.118910+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:42.433751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584851+0000","last_change":"2024-03-18T01:34:56.652073+0000","last_active":"2024-03-18T01:35:03.584851+0000","last_peered":"2024-03-18T01:35:03.584851+0000","last_clean":"2024-03-18T01:35:03.584851+0000","last_became_active":"2024-03-18T01:34:56.651748+0000","last_became_peered":"2024-03-18T01:34:56.651748+0000","last_unstale":"2024-03-18T01:35:03.584851+0000","last_undegraded":"2024-03-18T01:35:03.584851+0000","last_fullsized":"2024-03-18T01:35:03.584851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:33.148823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083665+0000","last_change":"2024-03-18T01:34:56.645782+0000","last_active":"2024-03-18T01:35:03.083665+0000","last_peered":"2024-03-18T01:35:03.083665+0000","last_clean":"2024-03-18T01:35:03.083665+0000","last_became_active":"2024-03-18T01:34:56.645439+0000","last_became_peered":"2024-03-18T01:34:56.645439+0000","last_unstale":"2024-03-18T01:35:03.083665+0000","last_undegraded":"2024-03-18T01:35:03.083665+0000","last_fullsized":"2024-03-18T01:35:03.083665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:40.242348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.509354+0000","last_change":"2024-03-18T01:34:56.639697+0000","last_active":"2024-03-18T01:35:02.509354+0000","last_peered":"2024-03-18T01:35:02.509354+0000","last_clean":"2024-03-18T01:35:02.509354+0000","last_became_active":"2024-03-18T01:34:56.639339+0000","last_became_peered":"2024-03-18T01:34:56.639339+0000","last_unstale":"2024-03-18T01:35:02.509354+0000","last_undegraded":"2024-03-18T01:35:02.509354+0000","last_fullsized":"2024-03-18T01:35:02.509354+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:25:22.741437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119467+0000","last_change":"2024-03-18T01:34:56.655341+0000","last_active":"2024-03-18T01:35:02.119467+0000","last_peered":"2024-03-18T01:35:02.119467+0000","last_clean":"2024-03-18T01:35:02.119467+0000","last_became_active":"2024-03-18T01:34:56.655015+0000","last_became_peered":"2024-03-18T01:34:56.655015+0000","last_unstale":"2024-03-18T01:35:02.119467+0000","last_undegraded":"2024-03-18T01:35:02.119467+0000","last_fullsized":"2024-03-18T01:35:02.119467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:50.766000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583886+0000","last_change":"2024-03-18T01:34:56.648399+0000","last_active":"2024-03-18T01:35:03.583886+0000","last_peered":"2024-03-18T01:35:03.583886+0000","last_clean":"2024-03-18T01:35:03.583886+0000","last_became_active":"2024-03-18T01:34:56.648042+0000","last_became_peered":"2024-03-18T01:34:56.648042+0000","last_unstale":"2024-03-18T01:35:03.583886+0000","last_undegraded":"2024-03-18T01:35:03.583886+0000","last_fullsized":"2024-03-18T01:35:03.583886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:04.972230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588010+0000","last_change":"2024-03-18T01:34:56.653465+0000","last_active":"2024-03-18T01:35:03.588010+0000","last_peered":"2024-03-18T01:35:03.588010+0000","last_clean":"2024-03-18T01:35:03.588010+0000","last_became_active":"2024-03-18T01:34:56.651183+0000","last_became_peered":"2024-03-18T01:34:56.651183+0000","last_unstale":"2024-03-18T01:35:03.588010+0000","last_undegraded":"2024-03-18T01:35:03.588010+0000","last_fullsized":"2024-03-18T01:35:03.588010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:57.983330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122010+0000","last_change":"2024-03-18T01:34:56.658845+0000","last_active":"2024-03-18T01:35:02.122010+0000","last_peered":"2024-03-18T01:35:02.122010+0000","last_clean":"2024-03-18T01:35:02.122010+0000","last_became_active":"2024-03-18T01:34:56.658435+0000","last_became_peered":"2024-03-18T01:34:56.658435+0000","last_unstale":"2024-03-18T01:35:02.122010+0000","last_undegraded":"2024-03-18T01:35:02.122010+0000","last_fullsized":"2024-03-18T01:35:02.122010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:02.176325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122050+0000","last_change":"2024-03-18T01:34:56.660752+0000","last_active":"2024-03-18T01:35:02.122050+0000","last_peered":"2024-03-18T01:35:02.122050+0000","last_clean":"2024-03-18T01:35:02.122050+0000","last_became_active":"2024-03-18T01:34:56.660378+0000","last_became_peered":"2024-03-18T01:34:56.660378+0000","last_unstale":"2024-03-18T01:35:02.122050+0000","last_undegraded":"2024-03-18T01:35:02.122050+0000","last_fullsized":"2024-03-18T01:35:02.122050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:13.143481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083064+0000","last_change":"2024-03-18T01:34:56.656589+0000","last_active":"2024-03-18T01:35:03.083064+0000","last_peered":"2024-03-18T01:35:03.083064+0000","last_clean":"2024-03-18T01:35:03.083064+0000","last_became_active":"2024-03-18T01:34:56.656184+0000","last_became_peered":"2024-03-18T01:34:56.656184+0000","last_unstale":"2024-03-18T01:35:03.083064+0000","last_undegraded":"2024-03-18T01:35:03.083064+0000","last_fullsized":"2024-03-18T01:35:03.083064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:56.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538934+0000","last_change":"2024-03-18T01:34:56.661327+0000","last_active":"2024-03-18T01:35:02.538934+0000","last_peered":"2024-03-18T01:35:02.538934+0000","last_clean":"2024-03-18T01:35:02.538934+0000","last_became_active":"2024-03-18T01:34:56.660767+0000","last_became_peered":"2024-03-18T01:34:56.660767+0000","last_unstale":"2024-03-18T01:35:02.538934+0000","last_undegraded":"2024-03-18T01:35:02.538934+0000","last_fullsized":"2024-03-18T01:35:02.538934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:07:59.066028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587752+0000","last_change":"2024-03-18T01:34:56.650149+0000","last_active":"2024-03-18T01:35:03.587752+0000","last_peered":"2024-03-18T01:35:03.587752+0000","last_clean":"2024-03-18T01:35:03.587752+0000","last_became_active":"2024-03-18T01:34:56.649780+0000","last_became_peered":"2024-03-18T01:34:56.649780+0000","last_unstale":"2024-03-18T01:35:03.587752+0000","last_undegraded":"2024-03-18T01:35:03.587752+0000","last_fullsized":"2024-03-18T01:35:03.587752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:41.605563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584451+0000","last_change":"2024-03-18T01:34:56.653919+0000","last_active":"2024-03-18T01:35:03.584451+0000","last_peered":"2024-03-18T01:35:03.584451+0000","last_clean":"2024-03-18T01:35:03.584451+0000","last_became_active":"2024-03-18T01:34:56.653512+0000","last_became_peered":"2024-03-18T01:34:56.653512+0000","last_unstale":"2024-03-18T01:35:03.584451+0000","last_undegraded":"2024-03-18T01:35:03.584451+0000","last_fullsized":"2024-03-18T01:35:03.584451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:27.419518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584519+0000","last_change":"2024-03-18T01:35:01.149999+0000","last_active":"2024-03-18T01:35:03.584519+0000","last_peered":"2024-03-18T01:35:03.584519+0000","last_clean":"2024-03-18T01:35:03.584519+0000","last_became_active":"2024-03-18T01:35:01.149603+0000","last_became_peered":"2024-03-18T01:35:01.149603+0000","last_unstale":"2024-03-18T01:35:03.584519+0000","last_undegraded":"2024-03-18T01:35:03.584519+0000","last_fullsized":"2024-03-18T01:35:03.584519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:37.619517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.643907+0000","last_change":"2024-03-18T01:34:55.650404+0000","last_active":"2024-03-18T01:35:30.643907+0000","last_peered":"2024-03-18T01:35:30.643907+0000","last_clean":"2024-03-18T01:35:30.643907+0000","last_became_active":"2024-03-18T01:34:55.650068+0000","last_became_peered":"2024-03-18T01:34:55.650068+0000","last_unstale":"2024-03-18T01:35:30.643907+0000","last_undegraded":"2024-03-18T01:35:30.643907+0000","last_fullsized":"2024-03-18T01:35:30.643907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:11.396529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585520+0000","last_change":"2024-03-18T01:34:56.654265+0000","last_active":"2024-03-18T01:35:03.585520+0000","last_peered":"2024-03-18T01:35:03.585520+0000","last_clean":"2024-03-18T01:35:03.585520+0000","last_became_active":"2024-03-18T01:34:56.653886+0000","last_became_peered":"2024-03-18T01:34:56.653886+0000","last_unstale":"2024-03-18T01:35:03.585520+0000","last_undegraded":"2024-03-18T01:35:03.585520+0000","last_fullsized":"2024-03-18T01:35:03.585520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:01:44.069966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583483+0000","last_change":"2024-03-18T01:35:01.155838+0000","last_active":"2024-03-18T01:35:03.583483+0000","last_peered":"2024-03-18T01:35:03.583483+0000","last_clean":"2024-03-18T01:35:03.583483+0000","last_became_active":"2024-03-18T01:35:01.155391+0000","last_became_peered":"2024-03-18T01:35:01.155391+0000","last_unstale":"2024-03-18T01:35:03.583483+0000","last_undegraded":"2024-03-18T01:35:03.583483+0000","last_fullsized":"2024-03-18T01:35:03.583483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:31.885767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585557+0000","last_change":"2024-03-18T01:34:55.658830+0000","last_active":"2024-03-18T01:35:03.585557+0000","last_peered":"2024-03-18T01:35:03.585557+0000","last_clean":"2024-03-18T01:35:03.585557+0000","last_became_active":"2024-03-18T01:34:55.658395+0000","last_became_peered":"2024-03-18T01:34:55.658395+0000","last_unstale":"2024-03-18T01:35:03.585557+0000","last_undegraded":"2024-03-18T01:35:03.585557+0000","last_fullsized":"2024-03-18T01:35:03.585557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:07.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123797+0000","last_change":"2024-03-18T01:34:56.656874+0000","last_active":"2024-03-18T01:35:02.123797+0000","last_peered":"2024-03-18T01:35:02.123797+0000","last_clean":"2024-03-18T01:35:02.123797+0000","last_became_active":"2024-03-18T01:34:56.656525+0000","last_became_peered":"2024-03-18T01:34:56.656525+0000","last_unstale":"2024-03-18T01:35:02.123797+0000","last_undegraded":"2024-03-18T01:35:02.123797+0000","last_fullsized":"2024-03-18T01:35:02.123797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:35:19.363586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583620+0000","last_change":"2024-03-18T01:35:01.153190+0000","last_active":"2024-03-18T01:35:03.583620+0000","last_peered":"2024-03-18T01:35:03.583620+0000","last_clean":"2024-03-18T01:35:03.583620+0000","last_became_active":"2024-03-18T01:35:01.152439+0000","last_became_peered":"2024-03-18T01:35:01.152439+0000","last_unstale":"2024-03-18T01:35:03.583620+0000","last_undegraded":"2024-03-18T01:35:03.583620+0000","last_fullsized":"2024-03-18T01:35:03.583620+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:08:40.294721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'11","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.370316+0000","last_change":"2024-03-18T01:34:55.648218+0000","last_active":"2024-03-18T01:35:30.370316+0000","last_peered":"2024-03-18T01:35:30.370316+0000","last_clean":"2024-03-18T01:35:30.370316+0000","last_became_active":"2024-03-18T01:34:55.647779+0000","last_became_peered":"2024-03-18T01:34:55.647779+0000","last_unstale":"2024-03-18T01:35:30.370316+0000","last_undegraded":"2024-03-18T01:35:30.370316+0000","last_fullsized":"2024-03-18T01:35:30.370316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:22:08.136973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083384+0000","last_change":"2024-03-18T01:34:56.649900+0000","last_active":"2024-03-18T01:35:03.083384+0000","last_peered":"2024-03-18T01:35:03.083384+0000","last_clean":"2024-03-18T01:35:03.083384+0000","last_became_active":"2024-03-18T01:34:56.649564+0000","last_became_peered":"2024-03-18T01:34:56.649564+0000","last_unstale":"2024-03-18T01:35:03.083384+0000","last_undegraded":"2024-03-18T01:35:03.083384+0000","last_fullsized":"2024-03-18T01:35:03.083384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:29.861861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583496+0000","last_change":"2024-03-18T01:35:01.147062+0000","last_active":"2024-03-18T01:35:03.583496+0000","last_peered":"2024-03-18T01:35:03.583496+0000","last_clean":"2024-03-18T01:35:03.583496+0000","last_became_active":"2024-03-18T01:35:01.146666+0000","last_became_peered":"2024-03-18T01:35:01.146666+0000","last_unstale":"2024-03-18T01:35:03.583496+0000","last_undegraded":"2024-03-18T01:35:03.583496+0000","last_fullsized":"2024-03-18T01:35:03.583496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:15:21.433263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.369728+0000","last_change":"2024-03-18T01:34:55.645512+0000","last_active":"2024-03-18T01:35:30.369728+0000","last_peered":"2024-03-18T01:35:30.369728+0000","last_clean":"2024-03-18T01:35:30.369728+0000","last_became_active":"2024-03-18T01:34:55.645155+0000","last_became_peered":"2024-03-18T01:34:55.645155+0000","last_unstale":"2024-03-18T01:35:30.369728+0000","last_undegraded":"2024-03-18T01:35:30.369728+0000","last_fullsized":"2024-03-18T01:35:30.369728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:32:17.049289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543742+0000","last_change":"2024-03-18T01:34:56.650449+0000","last_active":"2024-03-18T01:35:03.543742+0000","last_peered":"2024-03-18T01:35:03.543742+0000","last_clean":"2024-03-18T01:35:03.543742+0000","last_became_active":"2024-03-18T01:34:56.650103+0000","last_became_peered":"2024-03-18T01:34:56.650103+0000","last_unstale":"2024-03-18T01:35:03.543742+0000","last_undegraded":"2024-03-18T01:35:03.543742+0000","last_fullsized":"2024-03-18T01:35:03.543742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:07:55.003592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584327+0000","last_change":"2024-03-18T01:35:01.132227+0000","last_active":"2024-03-18T01:35:03.584327+0000","last_peered":"2024-03-18T01:35:03.584327+0000","last_clean":"2024-03-18T01:35:03.584327+0000","last_became_active":"2024-03-18T01:35:01.131767+0000","last_became_peered":"2024-03-18T01:35:01.131767+0000","last_unstale":"2024-03-18T01:35:03.584327+0000","last_undegraded":"2024-03-18T01:35:03.584327+0000","last_fullsized":"2024-03-18T01:35:03.584327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:54.335482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.340349+0000","last_change":"2024-03-18T01:34:55.649277+0000","last_active":"2024-03-18T01:35:27.340349+0000","last_peered":"2024-03-18T01:35:27.340349+0000","last_clean":"2024-03-18T01:35:27.340349+0000","last_became_active":"2024-03-18T01:34:55.648917+0000","last_became_peered":"2024-03-18T01:34:55.648917+0000","last_unstale":"2024-03-18T01:35:27.340349+0000","last_undegraded":"2024-03-18T01:35:27.340349+0000","last_fullsized":"2024-03-18T01:35:27.340349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:00.117647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280348+0000","last_change":"2024-03-18T01:34:56.648394+0000","last_active":"2024-03-18T01:35:02.280348+0000","last_peered":"2024-03-18T01:35:02.280348+0000","last_clean":"2024-03-18T01:35:02.280348+0000","last_became_active":"2024-03-18T01:34:56.648043+0000","last_became_peered":"2024-03-18T01:34:56.648043+0000","last_unstale":"2024-03-18T01:35:02.280348+0000","last_undegraded":"2024-03-18T01:35:02.280348+0000","last_fullsized":"2024-03-18T01:35:02.280348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:54:52.531851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584358+0000","last_change":"2024-03-18T01:35:01.151346+0000","last_active":"2024-03-18T01:35:03.584358+0000","last_peered":"2024-03-18T01:35:03.584358+0000","last_clean":"2024-03-18T01:35:03.584358+0000","last_became_active":"2024-03-18T01:35:01.150916+0000","last_became_peered":"2024-03-18T01:35:01.150916+0000","last_unstale":"2024-03-18T01:35:03.584358+0000","last_undegraded":"2024-03-18T01:35:03.584358+0000","last_fullsized":"2024-03-18T01:35:03.584358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:44:36.000983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.364785+0000","last_change":"2024-03-18T01:34:55.648498+0000","last_active":"2024-03-18T01:35:29.364785+0000","last_peered":"2024-03-18T01:35:29.364785+0000","last_clean":"2024-03-18T01:35:29.364785+0000","last_became_active":"2024-03-18T01:34:55.648152+0000","last_became_peered":"2024-03-18T01:34:55.648152+0000","last_unstale":"2024-03-18T01:35:29.364785+0000","last_undegraded":"2024-03-18T01:35:29.364785+0000","last_fullsized":"2024-03-18T01:35:29.364785+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:33.138620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543666+0000","last_change":"2024-03-18T01:34:56.651420+0000","last_active":"2024-03-18T01:35:03.543666+0000","last_peered":"2024-03-18T01:35:03.543666+0000","last_clean":"2024-03-18T01:35:03.543666+0000","last_became_active":"2024-03-18T01:34:56.650909+0000","last_became_peered":"2024-03-18T01:34:56.650909+0000","last_unstale":"2024-03-18T01:35:03.543666+0000","last_undegraded":"2024-03-18T01:35:03.543666+0000","last_fullsized":"2024-03-18T01:35:03.543666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:43:40.548970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.539037+0000","last_change":"2024-03-18T01:35:01.157759+0000","last_active":"2024-03-18T01:35:02.539037+0000","last_peered":"2024-03-18T01:35:02.539037+0000","last_clean":"2024-03-18T01:35:02.539037+0000","last_became_active":"2024-03-18T01:35:01.157288+0000","last_became_peered":"2024-03-18T01:35:01.157288+0000","last_unstale":"2024-03-18T01:35:02.539037+0000","last_undegraded":"2024-03-18T01:35:02.539037+0000","last_fullsized":"2024-03-18T01:35:02.539037+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:05:41.272727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:28.353189+0000","last_change":"2024-03-18T01:34:55.650388+0000","last_active":"2024-03-18T01:35:28.353189+0000","last_peered":"2024-03-18T01:35:28.353189+0000","last_clean":"2024-03-18T01:35:28.353189+0000","last_became_active":"2024-03-18T01:34:55.650055+0000","last_became_peered":"2024-03-18T01:34:55.650055+0000","last_unstale":"2024-03-18T01:35:28.353189+0000","last_undegraded":"2024-03-18T01:35:28.353189+0000","last_fullsized":"2024-03-18T01:35:28.353189+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:29:40.758561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584675+0000","last_change":"2024-03-18T01:34:56.658634+0000","last_active":"2024-03-18T01:35:03.584675+0000","last_peered":"2024-03-18T01:35:03.584675+0000","last_clean":"2024-03-18T01:35:03.584675+0000","last_became_active":"2024-03-18T01:34:56.657196+0000","last_became_peered":"2024-03-18T01:34:56.657196+0000","last_unstale":"2024-03-18T01:35:03.584675+0000","last_undegraded":"2024-03-18T01:35:03.584675+0000","last_fullsized":"2024-03-18T01:35:03.584675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:12:36.265949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123736+0000","last_change":"2024-03-18T01:35:01.150431+0000","last_active":"2024-03-18T01:35:02.123736+0000","last_peered":"2024-03-18T01:35:02.123736+0000","last_clean":"2024-03-18T01:35:02.123736+0000","last_became_active":"2024-03-18T01:35:01.149981+0000","last_became_peered":"2024-03-18T01:35:01.149981+0000","last_unstale":"2024-03-18T01:35:02.123736+0000","last_undegraded":"2024-03-18T01:35:02.123736+0000","last_fullsized":"2024-03-18T01:35:02.123736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:34:55.371860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543316+0000","last_change":"2024-03-18T01:34:55.649346+0000","last_active":"2024-03-18T01:35:03.543316+0000","last_peered":"2024-03-18T01:35:03.543316+0000","last_clean":"2024-03-18T01:35:03.543316+0000","last_became_active":"2024-03-18T01:34:55.648958+0000","last_became_peered":"2024-03-18T01:34:55.648958+0000","last_unstale":"2024-03-18T01:35:03.543316+0000","last_undegraded":"2024-03-18T01:35:03.543316+0000","last_fullsized":"2024-03-18T01:35:03.543316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:53.788081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.586327+0000","last_change":"2024-03-18T01:34:56.653525+0000","last_active":"2024-03-18T01:35:03.586327+0000","last_peered":"2024-03-18T01:35:03.586327+0000","last_clean":"2024-03-18T01:35:03.586327+0000","last_became_active":"2024-03-18T01:34:56.653218+0000","last_became_peered":"2024-03-18T01:34:56.653218+0000","last_unstale":"2024-03-18T01:35:03.586327+0000","last_undegraded":"2024-03-18T01:35:03.586327+0000","last_fullsized":"2024-03-18T01:35:03.586327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:32:37.479782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584927+0000","last_change":"2024-03-18T01:35:01.135505+0000","last_active":"2024-03-18T01:35:03.584927+0000","last_peered":"2024-03-18T01:35:03.584927+0000","last_clean":"2024-03-18T01:35:03.584927+0000","last_became_active":"2024-03-18T01:35:01.135055+0000","last_became_peered":"2024-03-18T01:35:01.135055+0000","last_unstale":"2024-03-18T01:35:03.584927+0000","last_undegraded":"2024-03-18T01:35:03.584927+0000","last_fullsized":"2024-03-18T01:35:03.584927+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:52:35.069313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:32.646768+0000","last_change":"2024-03-18T01:34:55.659389+0000","last_active":"2024-03-18T01:35:32.646768+0000","last_peered":"2024-03-18T01:35:32.646768+0000","last_clean":"2024-03-18T01:35:32.646768+0000","last_became_active":"2024-03-18T01:34:55.659090+0000","last_became_peered":"2024-03-18T01:34:55.659090+0000","last_unstale":"2024-03-18T01:35:32.646768+0000","last_undegraded":"2024-03-18T01:35:32.646768+0000","last_fullsized":"2024-03-18T01:35:32.646768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:55:51.731788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083673+0000","last_change":"2024-03-18T01:34:56.650411+0000","last_active":"2024-03-18T01:35:03.083673+0000","last_peered":"2024-03-18T01:35:03.083673+0000","last_clean":"2024-03-18T01:35:03.083673+0000","last_became_active":"2024-03-18T01:34:56.650036+0000","last_became_peered":"2024-03-18T01:34:56.650036+0000","last_unstale":"2024-03-18T01:35:03.083673+0000","last_undegraded":"2024-03-18T01:35:03.083673+0000","last_fullsized":"2024-03-18T01:35:03.083673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:59.912207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.586383+0000","last_change":"2024-03-18T01:35:01.149186+0000","last_active":"2024-03-18T01:35:03.586383+0000","last_peered":"2024-03-18T01:35:03.586383+0000","last_clean":"2024-03-18T01:35:03.586383+0000","last_became_active":"2024-03-18T01:35:01.148721+0000","last_became_peered":"2024-03-18T01:35:01.148721+0000","last_unstale":"2024-03-18T01:35:03.586383+0000","last_undegraded":"2024-03-18T01:35:03.586383+0000","last_fullsized":"2024-03-18T01:35:03.586383+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:09:35.509348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.369730+0000","last_change":"2024-03-18T01:34:55.647879+0000","last_active":"2024-03-18T01:35:30.369730+0000","last_peered":"2024-03-18T01:35:30.369730+0000","last_clean":"2024-03-18T01:35:30.369730+0000","last_became_active":"2024-03-18T01:34:55.647549+0000","last_became_peered":"2024-03-18T01:34:55.647549+0000","last_unstale":"2024-03-18T01:35:30.369730+0000","last_undegraded":"2024-03-18T01:35:30.369730+0000","last_fullsized":"2024-03-18T01:35:30.369730+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:12.261729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123791+0000","last_change":"2024-03-18T01:34:56.664303+0000","last_active":"2024-03-18T01:35:02.123791+0000","last_peered":"2024-03-18T01:35:02.123791+0000","last_clean":"2024-03-18T01:35:02.123791+0000","last_became_active":"2024-03-18T01:34:56.662597+0000","last_became_peered":"2024-03-18T01:34:56.662597+0000","last_unstale":"2024-03-18T01:35:02.123791+0000","last_undegraded":"2024-03-18T01:35:02.123791+0000","last_fullsized":"2024-03-18T01:35:02.123791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:54:19.581671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583776+0000","last_change":"2024-03-18T01:35:01.153448+0000","last_active":"2024-03-18T01:35:03.583776+0000","last_peered":"2024-03-18T01:35:03.583776+0000","last_clean":"2024-03-18T01:35:03.583776+0000","last_became_active":"2024-03-18T01:35:01.152593+0000","last_became_peered":"2024-03-18T01:35:01.152593+0000","last_unstale":"2024-03-18T01:35:03.583776+0000","last_undegraded":"2024-03-18T01:35:03.583776+0000","last_fullsized":"2024-03-18T01:35:03.583776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:46.363662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543662+0000","last_change":"2024-03-18T01:34:55.655079+0000","last_active":"2024-03-18T01:35:03.543662+0000","last_peered":"2024-03-18T01:35:03.543662+0000","last_clean":"2024-03-18T01:35:03.543662+0000","last_became_active":"2024-03-18T01:34:55.654702+0000","last_became_peered":"2024-03-18T01:34:55.654702+0000","last_unstale":"2024-03-18T01:35:03.543662+0000","last_undegraded":"2024-03-18T01:35:03.543662+0000","last_fullsized":"2024-03-18T01:35:03.543662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:50:46.524614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083490+0000","last_change":"2024-03-18T01:34:56.644441+0000","last_active":"2024-03-18T01:35:03.083490+0000","last_peered":"2024-03-18T01:35:03.083490+0000","last_clean":"2024-03-18T01:35:03.083490+0000","last_became_active":"2024-03-18T01:34:56.643924+0000","last_became_peered":"2024-03-18T01:34:56.643924+0000","last_unstale":"2024-03-18T01:35:03.083490+0000","last_undegraded":"2024-03-18T01:35:03.083490+0000","last_fullsized":"2024-03-18T01:35:03.083490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:44.260300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.586491+0000","last_change":"2024-03-18T01:35:01.150022+0000","last_active":"2024-03-18T01:35:03.586491+0000","last_peered":"2024-03-18T01:35:03.586491+0000","last_clean":"2024-03-18T01:35:03.586491+0000","last_became_active":"2024-03-18T01:35:01.149510+0000","last_became_peered":"2024-03-18T01:35:01.149510+0000","last_unstale":"2024-03-18T01:35:03.586491+0000","last_undegraded":"2024-03-18T01:35:03.586491+0000","last_fullsized":"2024-03-18T01:35:03.586491+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:40:36.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'26","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:32.649982+0000","last_change":"2024-03-18T01:34:55.642397+0000","last_active":"2024-03-18T01:35:32.649982+0000","last_peered":"2024-03-18T01:35:32.649982+0000","last_clean":"2024-03-18T01:35:32.649982+0000","last_became_active":"2024-03-18T01:34:55.642046+0000","last_became_peered":"2024-03-18T01:34:55.642046+0000","last_unstale":"2024-03-18T01:35:32.649982+0000","last_undegraded":"2024-03-18T01:35:32.649982+0000","last_fullsized":"2024-03-18T01:35:32.649982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:48:31.631414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41906,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":45,"num_write":28,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280585+0000","last_change":"2024-03-18T01:34:56.644388+0000","last_active":"2024-03-18T01:35:02.280585+0000","last_peered":"2024-03-18T01:35:02.280585+0000","last_clean":"2024-03-18T01:35:02.280585+0000","last_became_active":"2024-03-18T01:34:56.644014+0000","last_became_peered":"2024-03-18T01:34:56.644014+0000","last_unstale":"2024-03-18T01:35:02.280585+0000","last_undegraded":"2024-03-18T01:35:02.280585+0000","last_fullsized":"2024-03-18T01:35:02.280585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:06:18.193600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.544060+0000","last_change":"2024-03-18T01:35:01.148796+0000","last_active":"2024-03-18T01:35:03.544060+0000","last_peered":"2024-03-18T01:35:03.544060+0000","last_clean":"2024-03-18T01:35:03.544060+0000","last_became_active":"2024-03-18T01:35:01.148329+0000","last_became_peered":"2024-03-18T01:35:01.148329+0000","last_unstale":"2024-03-18T01:35:03.544060+0000","last_undegraded":"2024-03-18T01:35:03.544060+0000","last_fullsized":"2024-03-18T01:35:03.544060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:14:36.688878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.642536+0000","last_change":"2024-03-18T01:34:55.649600+0000","last_active":"2024-03-18T01:35:30.642536+0000","last_peered":"2024-03-18T01:35:30.642536+0000","last_clean":"2024-03-18T01:35:30.642536+0000","last_became_active":"2024-03-18T01:34:55.649192+0000","last_became_peered":"2024-03-18T01:34:55.649192+0000","last_unstale":"2024-03-18T01:35:30.642536+0000","last_undegraded":"2024-03-18T01:35:30.642536+0000","last_fullsized":"2024-03-18T01:35:30.642536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:55.294501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588223+0000","last_change":"2024-03-18T01:34:56.657786+0000","last_active":"2024-03-18T01:35:03.588223+0000","last_peered":"2024-03-18T01:35:03.588223+0000","last_clean":"2024-03-18T01:35:03.588223+0000","last_became_active":"2024-03-18T01:34:56.657432+0000","last_became_peered":"2024-03-18T01:34:56.657432+0000","last_unstale":"2024-03-18T01:35:03.588223+0000","last_undegraded":"2024-03-18T01:35:03.588223+0000","last_fullsized":"2024-03-18T01:35:03.588223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:13:39.498562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538899+0000","last_change":"2024-03-18T01:35:01.150495+0000","last_active":"2024-03-18T01:35:02.538899+0000","last_peered":"2024-03-18T01:35:02.538899+0000","last_clean":"2024-03-18T01:35:02.538899+0000","last_became_active":"2024-03-18T01:35:01.150033+0000","last_became_peered":"2024-03-18T01:35:01.150033+0000","last_unstale":"2024-03-18T01:35:02.538899+0000","last_undegraded":"2024-03-18T01:35:02.538899+0000","last_fullsized":"2024-03-18T01:35:02.538899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:38.433827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.340026+0000","last_change":"2024-03-18T01:34:55.658820+0000","last_active":"2024-03-18T01:35:27.340026+0000","last_peered":"2024-03-18T01:35:27.340026+0000","last_clean":"2024-03-18T01:35:27.340026+0000","last_became_active":"2024-03-18T01:34:55.658446+0000","last_became_peered":"2024-03-18T01:34:55.658446+0000","last_unstale":"2024-03-18T01:35:27.340026+0000","last_undegraded":"2024-03-18T01:35:27.340026+0000","last_fullsized":"2024-03-18T01:35:27.340026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:19:21.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119186+0000","last_change":"2024-03-18T01:34:56.652890+0000","last_active":"2024-03-18T01:35:02.119186+0000","last_peered":"2024-03-18T01:35:02.119186+0000","last_clean":"2024-03-18T01:35:02.119186+0000","last_became_active":"2024-03-18T01:34:56.652444+0000","last_became_peered":"2024-03-18T01:34:56.652444+0000","last_unstale":"2024-03-18T01:35:02.119186+0000","last_undegraded":"2024-03-18T01:35:02.119186+0000","last_fullsized":"2024-03-18T01:35:02.119186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:58:47.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587675+0000","last_change":"2024-03-18T01:35:01.154700+0000","last_active":"2024-03-18T01:35:03.587675+0000","last_peered":"2024-03-18T01:35:03.587675+0000","last_clean":"2024-03-18T01:35:03.587675+0000","last_became_active":"2024-03-18T01:35:01.148453+0000","last_became_peered":"2024-03-18T01:35:01.148453+0000","last_unstale":"2024-03-18T01:35:03.587675+0000","last_undegraded":"2024-03-18T01:35:03.587675+0000","last_fullsized":"2024-03-18T01:35:03.587675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:08:40.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.359311+0000","last_change":"2024-03-18T01:34:55.643861+0000","last_active":"2024-03-18T01:35:29.359311+0000","last_peered":"2024-03-18T01:35:29.359311+0000","last_clean":"2024-03-18T01:35:29.359311+0000","last_became_active":"2024-03-18T01:34:55.643352+0000","last_became_peered":"2024-03-18T01:34:55.643352+0000","last_unstale":"2024-03-18T01:35:29.359311+0000","last_undegraded":"2024-03-18T01:35:29.359311+0000","last_fullsized":"2024-03-18T01:35:29.359311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:56:53.685571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.118990+0000","last_change":"2024-03-18T01:34:56.646533+0000","last_active":"2024-03-18T01:35:02.118990+0000","last_peered":"2024-03-18T01:35:02.118990+0000","last_clean":"2024-03-18T01:35:02.118990+0000","last_became_active":"2024-03-18T01:34:56.646180+0000","last_became_peered":"2024-03-18T01:34:56.646180+0000","last_unstale":"2024-03-18T01:35:02.118990+0000","last_undegraded":"2024-03-18T01:35:02.118990+0000","last_fullsized":"2024-03-18T01:35:02.118990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:43:58.847258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588967+0000","last_change":"2024-03-18T01:35:01.158762+0000","last_active":"2024-03-18T01:35:03.588967+0000","last_peered":"2024-03-18T01:35:03.588967+0000","last_clean":"2024-03-18T01:35:03.588967+0000","last_became_active":"2024-03-18T01:35:01.158281+0000","last_became_peered":"2024-03-18T01:35:01.158281+0000","last_unstale":"2024-03-18T01:35:03.588967+0000","last_undegraded":"2024-03-18T01:35:03.588967+0000","last_fullsized":"2024-03-18T01:35:03.588967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:32:19.681361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.362929+0000","last_change":"2024-03-18T01:34:55.658245+0000","last_active":"2024-03-18T01:35:29.362929+0000","last_peered":"2024-03-18T01:35:29.362929+0000","last_clean":"2024-03-18T01:35:29.362929+0000","last_became_active":"2024-03-18T01:34:55.657916+0000","last_became_peered":"2024-03-18T01:34:55.657916+0000","last_unstale":"2024-03-18T01:35:29.362929+0000","last_undegraded":"2024-03-18T01:35:29.362929+0000","last_fullsized":"2024-03-18T01:35:29.362929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:18:49.207721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543317+0000","last_change":"2024-03-18T01:34:56.648646+0000","last_active":"2024-03-18T01:35:03.543317+0000","last_peered":"2024-03-18T01:35:03.543317+0000","last_clean":"2024-03-18T01:35:03.543317+0000","last_became_active":"2024-03-18T01:34:56.648257+0000","last_became_peered":"2024-03-18T01:34:56.648257+0000","last_unstale":"2024-03-18T01:35:03.543317+0000","last_undegraded":"2024-03-18T01:35:03.543317+0000","last_fullsized":"2024-03-18T01:35:03.543317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:53.788445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543374+0000","last_change":"2024-03-18T01:35:01.136912+0000","last_active":"2024-03-18T01:35:03.543374+0000","last_peered":"2024-03-18T01:35:03.543374+0000","last_clean":"2024-03-18T01:35:03.543374+0000","last_became_active":"2024-03-18T01:35:01.136611+0000","last_became_peered":"2024-03-18T01:35:01.136611+0000","last_unstale":"2024-03-18T01:35:03.543374+0000","last_undegraded":"2024-03-18T01:35:03.543374+0000","last_fullsized":"2024-03-18T01:35:03.543374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:31.089605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280837+0000","last_change":"2024-03-18T01:34:55.648345+0000","last_active":"2024-03-18T01:35:02.280837+0000","last_peered":"2024-03-18T01:35:02.280837+0000","last_clean":"2024-03-18T01:35:02.280837+0000","last_became_active":"2024-03-18T01:34:55.647113+0000","last_became_peered":"2024-03-18T01:34:55.647113+0000","last_unstale":"2024-03-18T01:35:02.280837+0000","last_undegraded":"2024-03-18T01:35:02.280837+0000","last_fullsized":"2024-03-18T01:35:02.280837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:20.904129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538825+0000","last_change":"2024-03-18T01:34:56.649147+0000","last_active":"2024-03-18T01:35:02.538825+0000","last_peered":"2024-03-18T01:35:02.538825+0000","last_clean":"2024-03-18T01:35:02.538825+0000","last_became_active":"2024-03-18T01:34:56.648815+0000","last_became_peered":"2024-03-18T01:34:56.648815+0000","last_unstale":"2024-03-18T01:35:02.538825+0000","last_undegraded":"2024-03-18T01:35:02.538825+0000","last_fullsized":"2024-03-18T01:35:02.538825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:53.560679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280436+0000","last_change":"2024-03-18T01:35:01.156778+0000","last_active":"2024-03-18T01:35:02.280436+0000","last_peered":"2024-03-18T01:35:02.280436+0000","last_clean":"2024-03-18T01:35:02.280436+0000","last_became_active":"2024-03-18T01:35:01.156367+0000","last_became_peered":"2024-03-18T01:35:01.156367+0000","last_unstale":"2024-03-18T01:35:02.280436+0000","last_undegraded":"2024-03-18T01:35:02.280436+0000","last_fullsized":"2024-03-18T01:35:02.280436+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:46.223915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587230+0000","last_change":"2024-03-18T01:34:55.658722+0000","last_active":"2024-03-18T01:35:03.587230+0000","last_peered":"2024-03-18T01:35:03.587230+0000","last_clean":"2024-03-18T01:35:03.587230+0000","last_became_active":"2024-03-18T01:34:55.658321+0000","last_became_peered":"2024-03-18T01:34:55.658321+0000","last_unstale":"2024-03-18T01:35:03.587230+0000","last_undegraded":"2024-03-18T01:35:03.587230+0000","last_fullsized":"2024-03-18T01:35:03.587230+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:05.753226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.509200+0000","last_change":"2024-03-18T01:34:56.649415+0000","last_active":"2024-03-18T01:35:02.509200+0000","last_peered":"2024-03-18T01:35:02.509200+0000","last_clean":"2024-03-18T01:35:02.509200+0000","last_became_active":"2024-03-18T01:34:56.649019+0000","last_became_peered":"2024-03-18T01:34:56.649019+0000","last_unstale":"2024-03-18T01:35:02.509200+0000","last_undegraded":"2024-03-18T01:35:02.509200+0000","last_fullsized":"2024-03-18T01:35:02.509200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:28.473683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588723+0000","last_change":"2024-03-18T01:35:01.148937+0000","last_active":"2024-03-18T01:35:03.588723+0000","last_peered":"2024-03-18T01:35:03.588723+0000","last_clean":"2024-03-18T01:35:03.588723+0000","last_became_active":"2024-03-18T01:35:01.148415+0000","last_became_peered":"2024-03-18T01:35:01.148415+0000","last_unstale":"2024-03-18T01:35:03.588723+0000","last_undegraded":"2024-03-18T01:35:03.588723+0000","last_fullsized":"2024-03-18T01:35:03.588723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:45.926970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.645164+0000","last_change":"2024-03-18T01:34:55.653443+0000","last_active":"2024-03-18T01:35:30.645164+0000","last_peered":"2024-03-18T01:35:30.645164+0000","last_clean":"2024-03-18T01:35:30.645164+0000","last_became_active":"2024-03-18T01:34:55.653072+0000","last_became_peered":"2024-03-18T01:34:55.653072+0000","last_unstale":"2024-03-18T01:35:30.645164+0000","last_undegraded":"2024-03-18T01:35:30.645164+0000","last_fullsized":"2024-03-18T01:35:30.645164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:26.376636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.509045+0000","last_change":"2024-03-18T01:34:56.647373+0000","last_active":"2024-03-18T01:35:02.509045+0000","last_peered":"2024-03-18T01:35:02.509045+0000","last_clean":"2024-03-18T01:35:02.509045+0000","last_became_active":"2024-03-18T01:34:56.646986+0000","last_became_peered":"2024-03-18T01:34:56.646986+0000","last_unstale":"2024-03-18T01:35:02.509045+0000","last_undegraded":"2024-03-18T01:35:02.509045+0000","last_fullsized":"2024-03-18T01:35:02.509045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:09:00.829251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588905+0000","last_change":"2024-03-18T01:35:01.152767+0000","last_active":"2024-03-18T01:35:03.588905+0000","last_peered":"2024-03-18T01:35:03.588905+0000","last_clean":"2024-03-18T01:35:03.588905+0000","last_became_active":"2024-03-18T01:35:01.150619+0000","last_became_peered":"2024-03-18T01:35:01.150619+0000","last_unstale":"2024-03-18T01:35:03.588905+0000","last_undegraded":"2024-03-18T01:35:03.588905+0000","last_fullsized":"2024-03-18T01:35:03.588905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:57:28.258251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543419+0000","last_change":"2024-03-18T01:34:56.650541+0000","last_active":"2024-03-18T01:35:03.543419+0000","last_peered":"2024-03-18T01:35:03.543419+0000","last_clean":"2024-03-18T01:35:03.543419+0000","last_became_active":"2024-03-18T01:34:56.650072+0000","last_became_peered":"2024-03-18T01:34:56.650072+0000","last_unstale":"2024-03-18T01:35:03.543419+0000","last_undegraded":"2024-03-18T01:35:03.543419+0000","last_fullsized":"2024-03-18T01:35:03.543419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:51.224883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:28.355820+0000","last_change":"2024-03-18T01:34:55.664297+0000","last_active":"2024-03-18T01:35:28.355820+0000","last_peered":"2024-03-18T01:35:28.355820+0000","last_clean":"2024-03-18T01:35:28.355820+0000","last_became_active":"2024-03-18T01:34:55.663938+0000","last_became_peered":"2024-03-18T01:34:55.663938+0000","last_unstale":"2024-03-18T01:35:28.355820+0000","last_undegraded":"2024-03-18T01:35:28.355820+0000","last_fullsized":"2024-03-18T01:35:28.355820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:51.771987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119459+0000","last_change":"2024-03-18T01:35:01.141736+0000","last_active":"2024-03-18T01:35:02.119459+0000","last_peered":"2024-03-18T01:35:02.119459+0000","last_clean":"2024-03-18T01:35:02.119459+0000","last_became_active":"2024-03-18T01:35:01.141381+0000","last_became_peered":"2024-03-18T01:35:01.141381+0000","last_unstale":"2024-03-18T01:35:02.119459+0000","last_undegraded":"2024-03-18T01:35:02.119459+0000","last_fullsized":"2024-03-18T01:35:02.119459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:25:04.190250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584951+0000","last_change":"2024-03-18T01:34:56.666439+0000","last_active":"2024-03-18T01:35:03.584951+0000","last_peered":"2024-03-18T01:35:03.584951+0000","last_clean":"2024-03-18T01:35:03.584951+0000","last_became_active":"2024-03-18T01:34:56.664556+0000","last_became_peered":"2024-03-18T01:34:56.664556+0000","last_unstale":"2024-03-18T01:35:03.584951+0000","last_undegraded":"2024-03-18T01:35:03.584951+0000","last_fullsized":"2024-03-18T01:35:03.584951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:08:48.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.358418+0000","last_change":"2024-03-18T01:34:55.656484+0000","last_active":"2024-03-18T01:35:29.358418+0000","last_peered":"2024-03-18T01:35:29.358418+0000","last_clean":"2024-03-18T01:35:29.358418+0000","last_became_active":"2024-03-18T01:34:55.656125+0000","last_became_peered":"2024-03-18T01:34:55.656125+0000","last_unstale":"2024-03-18T01:35:29.358418+0000","last_undegraded":"2024-03-18T01:35:29.358418+0000","last_fullsized":"2024-03-18T01:35:29.358418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:58:25.944403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":345,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.118836+0000","last_change":"2024-03-18T01:33:35.824651+0000","last_active":"2024-03-18T01:35:02.118836+0000","last_peered":"2024-03-18T01:35:02.118836+0000","last_clean":"2024-03-18T01:35:02.118836+0000","last_became_active":"2024-03-18T01:33:35.824106+0000","last_became_peered":"2024-03-18T01:33:35.824106+0000","last_unstale":"2024-03-18T01:35:02.118836+0000","last_undegraded":"2024-03-18T01:35:02.118836+0000","last_fullsized":"2024-03-18T01:35:02.118836+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_clean_scrub_stamp":"2024-03-18T01:29:41.724134+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:06.422316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543884+0000","last_change":"2024-03-18T01:35:01.157735+0000","last_active":"2024-03-18T01:35:03.543884+0000","last_peered":"2024-03-18T01:35:03.543884+0000","last_clean":"2024-03-18T01:35:03.543884+0000","last_became_active":"2024-03-18T01:35:01.156954+0000","last_became_peered":"2024-03-18T01:35:01.156954+0000","last_unstale":"2024-03-18T01:35:03.543884+0000","last_undegraded":"2024-03-18T01:35:03.543884+0000","last_fullsized":"2024-03-18T01:35:03.543884+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:00.106809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.586697+0000","last_change":"2024-03-18T01:34:56.646808+0000","last_active":"2024-03-18T01:35:03.586697+0000","last_peered":"2024-03-18T01:35:03.586697+0000","last_clean":"2024-03-18T01:35:03.586697+0000","last_became_active":"2024-03-18T01:34:56.646437+0000","last_became_peered":"2024-03-18T01:34:56.646437+0000","last_unstale":"2024-03-18T01:35:03.586697+0000","last_undegraded":"2024-03-18T01:35:03.586697+0000","last_fullsized":"2024-03-18T01:35:03.586697+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:46:09.703889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:28.349311+0000","last_change":"2024-03-18T01:34:55.645365+0000","last_active":"2024-03-18T01:35:28.349311+0000","last_peered":"2024-03-18T01:35:28.349311+0000","last_clean":"2024-03-18T01:35:28.349311+0000","last_became_active":"2024-03-18T01:34:55.645018+0000","last_became_peered":"2024-03-18T01:34:55.645018+0000","last_unstale":"2024-03-18T01:35:28.349311+0000","last_undegraded":"2024-03-18T01:35:28.349311+0000","last_fullsized":"2024-03-18T01:35:28.349311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:08:31.869328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.539005+0000","last_change":"2024-03-18T01:35:01.147294+0000","last_active":"2024-03-18T01:35:02.539005+0000","last_peered":"2024-03-18T01:35:02.539005+0000","last_clean":"2024-03-18T01:35:02.539005+0000","last_became_active":"2024-03-18T01:35:01.146858+0000","last_became_peered":"2024-03-18T01:35:01.146858+0000","last_unstale":"2024-03-18T01:35:02.539005+0000","last_undegraded":"2024-03-18T01:35:02.539005+0000","last_fullsized":"2024-03-18T01:35:02.539005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:45:24.010746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.364151+0000","last_change":"2024-03-18T01:34:55.658110+0000","last_active":"2024-03-18T01:35:29.364151+0000","last_peered":"2024-03-18T01:35:29.364151+0000","last_clean":"2024-03-18T01:35:29.364151+0000","last_became_active":"2024-03-18T01:34:55.657802+0000","last_became_peered":"2024-03-18T01:34:55.657802+0000","last_unstale":"2024-03-18T01:35:29.364151+0000","last_undegraded":"2024-03-18T01:35:29.364151+0000","last_fullsized":"2024-03-18T01:35:29.364151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:34.264115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588758+0000","last_change":"2024-03-18T01:34:56.656689+0000","last_active":"2024-03-18T01:35:03.588758+0000","last_peered":"2024-03-18T01:35:03.588758+0000","last_clean":"2024-03-18T01:35:03.588758+0000","last_became_active":"2024-03-18T01:34:56.655653+0000","last_became_peered":"2024-03-18T01:34:56.655653+0000","last_unstale":"2024-03-18T01:35:03.588758+0000","last_undegraded":"2024-03-18T01:35:03.588758+0000","last_fullsized":"2024-03-18T01:35:03.588758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:58.370904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584710+0000","last_change":"2024-03-18T01:35:01.137419+0000","last_active":"2024-03-18T01:35:03.584710+0000","last_peered":"2024-03-18T01:35:03.584710+0000","last_clean":"2024-03-18T01:35:03.584710+0000","last_became_active":"2024-03-18T01:35:01.137033+0000","last_became_peered":"2024-03-18T01:35:01.137033+0000","last_unstale":"2024-03-18T01:35:03.584710+0000","last_undegraded":"2024-03-18T01:35:03.584710+0000","last_fullsized":"2024-03-18T01:35:03.584710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:13:48.382329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.365780+0000","last_change":"2024-03-18T01:34:55.663134+0000","last_active":"2024-03-18T01:35:29.365780+0000","last_peered":"2024-03-18T01:35:29.365780+0000","last_clean":"2024-03-18T01:35:29.365780+0000","last_became_active":"2024-03-18T01:34:55.662083+0000","last_became_peered":"2024-03-18T01:34:55.662083+0000","last_unstale":"2024-03-18T01:35:29.365780+0000","last_undegraded":"2024-03-18T01:35:29.365780+0000","last_fullsized":"2024-03-18T01:35:29.365780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:01:09.186599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585802+0000","last_change":"2024-03-18T01:34:56.664761+0000","last_active":"2024-03-18T01:35:03.585802+0000","last_peered":"2024-03-18T01:35:03.585802+0000","last_clean":"2024-03-18T01:35:03.585802+0000","last_became_active":"2024-03-18T01:34:56.663657+0000","last_became_peered":"2024-03-18T01:34:56.663657+0000","last_unstale":"2024-03-18T01:35:03.585802+0000","last_undegraded":"2024-03-18T01:35:03.585802+0000","last_fullsized":"2024-03-18T01:35:03.585802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:26:53.075000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584043+0000","last_change":"2024-03-18T01:35:01.133002+0000","last_active":"2024-03-18T01:35:03.584043+0000","last_peered":"2024-03-18T01:35:03.584043+0000","last_clean":"2024-03-18T01:35:03.584043+0000","last_became_active":"2024-03-18T01:35:01.132542+0000","last_became_peered":"2024-03-18T01:35:01.132542+0000","last_unstale":"2024-03-18T01:35:03.584043+0000","last_undegraded":"2024-03-18T01:35:03.584043+0000","last_fullsized":"2024-03-18T01:35:03.584043+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:23.823409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.342714+0000","last_change":"2024-03-18T01:34:55.648924+0000","last_active":"2024-03-18T01:35:27.342714+0000","last_peered":"2024-03-18T01:35:27.342714+0000","last_clean":"2024-03-18T01:35:27.342714+0000","last_became_active":"2024-03-18T01:34:55.648548+0000","last_became_peered":"2024-03-18T01:34:55.648548+0000","last_unstale":"2024-03-18T01:35:27.342714+0000","last_undegraded":"2024-03-18T01:35:27.342714+0000","last_fullsized":"2024-03-18T01:35:27.342714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:26:38.916856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.123556+0000","last_change":"2024-03-18T01:34:56.663142+0000","last_active":"2024-03-18T01:35:02.123556+0000","last_peered":"2024-03-18T01:35:02.123556+0000","last_clean":"2024-03-18T01:35:02.123556+0000","last_became_active":"2024-03-18T01:34:56.660256+0000","last_became_peered":"2024-03-18T01:34:56.660256+0000","last_unstale":"2024-03-18T01:35:02.123556+0000","last_undegraded":"2024-03-18T01:35:02.123556+0000","last_fullsized":"2024-03-18T01:35:02.123556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:59.503277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583806+0000","last_change":"2024-03-18T01:35:01.136186+0000","last_active":"2024-03-18T01:35:03.583806+0000","last_peered":"2024-03-18T01:35:03.583806+0000","last_clean":"2024-03-18T01:35:03.583806+0000","last_became_active":"2024-03-18T01:35:01.135803+0000","last_became_peered":"2024-03-18T01:35:01.135803+0000","last_unstale":"2024-03-18T01:35:03.583806+0000","last_undegraded":"2024-03-18T01:35:03.583806+0000","last_fullsized":"2024-03-18T01:35:03.583806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:20.027448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.341363+0000","last_change":"2024-03-18T01:34:55.663090+0000","last_active":"2024-03-18T01:35:27.341363+0000","last_peered":"2024-03-18T01:35:27.341363+0000","last_clean":"2024-03-18T01:35:27.341363+0000","last_became_active":"2024-03-18T01:34:55.662774+0000","last_became_peered":"2024-03-18T01:34:55.662774+0000","last_unstale":"2024-03-18T01:35:27.341363+0000","last_undegraded":"2024-03-18T01:35:27.341363+0000","last_fullsized":"2024-03-18T01:35:27.341363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:30:35.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.280200+0000","last_change":"2024-03-18T01:34:56.639509+0000","last_active":"2024-03-18T01:35:02.280200+0000","last_peered":"2024-03-18T01:35:02.280200+0000","last_clean":"2024-03-18T01:35:02.280200+0000","last_became_active":"2024-03-18T01:34:56.639144+0000","last_became_peered":"2024-03-18T01:34:56.639144+0000","last_unstale":"2024-03-18T01:35:02.280200+0000","last_undegraded":"2024-03-18T01:35:02.280200+0000","last_fullsized":"2024-03-18T01:35:02.280200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:12:36.220793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587941+0000","last_change":"2024-03-18T01:35:01.141762+0000","last_active":"2024-03-18T01:35:03.587941+0000","last_peered":"2024-03-18T01:35:03.587941+0000","last_clean":"2024-03-18T01:35:03.587941+0000","last_became_active":"2024-03-18T01:35:01.141219+0000","last_became_peered":"2024-03-18T01:35:01.141219+0000","last_unstale":"2024-03-18T01:35:03.587941+0000","last_undegraded":"2024-03-18T01:35:03.587941+0000","last_fullsized":"2024-03-18T01:35:03.587941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:55.839120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.373964+0000","last_change":"2024-03-18T01:34:55.662530+0000","last_active":"2024-03-18T01:35:30.373964+0000","last_peered":"2024-03-18T01:35:30.373964+0000","last_clean":"2024-03-18T01:35:30.373964+0000","last_became_active":"2024-03-18T01:34:55.661178+0000","last_became_peered":"2024-03-18T01:34:55.661178+0000","last_unstale":"2024-03-18T01:35:30.373964+0000","last_undegraded":"2024-03-18T01:35:30.373964+0000","last_fullsized":"2024-03-18T01:35:30.373964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:47:16.678553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538541+0000","last_change":"2024-03-18T01:34:56.660936+0000","last_active":"2024-03-18T01:35:02.538541+0000","last_peered":"2024-03-18T01:35:02.538541+0000","last_clean":"2024-03-18T01:35:02.538541+0000","last_became_active":"2024-03-18T01:34:56.660513+0000","last_became_peered":"2024-03-18T01:34:56.660513+0000","last_unstale":"2024-03-18T01:35:02.538541+0000","last_undegraded":"2024-03-18T01:35:02.538541+0000","last_fullsized":"2024-03-18T01:35:02.538541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:35:21.037366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583793+0000","last_change":"2024-03-18T01:35:01.136914+0000","last_active":"2024-03-18T01:35:03.583793+0000","last_peered":"2024-03-18T01:35:03.583793+0000","last_clean":"2024-03-18T01:35:03.583793+0000","last_became_active":"2024-03-18T01:35:01.136480+0000","last_became_peered":"2024-03-18T01:35:01.136480+0000","last_unstale":"2024-03-18T01:35:03.583793+0000","last_undegraded":"2024-03-18T01:35:03.583793+0000","last_fullsized":"2024-03-18T01:35:03.583793+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:49.262255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:29.364701+0000","last_change":"2024-03-18T01:34:55.654938+0000","last_active":"2024-03-18T01:35:29.364701+0000","last_peered":"2024-03-18T01:35:29.364701+0000","last_clean":"2024-03-18T01:35:29.364701+0000","last_became_active":"2024-03-18T01:34:55.654611+0000","last_became_peered":"2024-03-18T01:34:55.654611+0000","last_unstale":"2024-03-18T01:35:29.364701+0000","last_undegraded":"2024-03-18T01:35:29.364701+0000","last_fullsized":"2024-03-18T01:35:29.364701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:23.210205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585405+0000","last_change":"2024-03-18T01:34:56.650731+0000","last_active":"2024-03-18T01:35:03.585405+0000","last_peered":"2024-03-18T01:35:03.585405+0000","last_clean":"2024-03-18T01:35:03.585405+0000","last_became_active":"2024-03-18T01:34:56.650363+0000","last_became_peered":"2024-03-18T01:34:56.650363+0000","last_unstale":"2024-03-18T01:35:03.585405+0000","last_undegraded":"2024-03-18T01:35:03.585405+0000","last_fullsized":"2024-03-18T01:35:03.585405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:09:50.305019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.583792+0000","last_change":"2024-03-18T01:35:01.135101+0000","last_active":"2024-03-18T01:35:03.583792+0000","last_peered":"2024-03-18T01:35:03.583792+0000","last_clean":"2024-03-18T01:35:03.583792+0000","last_became_active":"2024-03-18T01:35:01.134690+0000","last_became_peered":"2024-03-18T01:35:01.134690+0000","last_unstale":"2024-03-18T01:35:03.583792+0000","last_undegraded":"2024-03-18T01:35:03.583792+0000","last_fullsized":"2024-03-18T01:35:03.583792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:33.381629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:27.339513+0000","last_change":"2024-03-18T01:34:55.648108+0000","last_active":"2024-03-18T01:35:27.339513+0000","last_peered":"2024-03-18T01:35:27.339513+0000","last_clean":"2024-03-18T01:35:27.339513+0000","last_became_active":"2024-03-18T01:34:55.647754+0000","last_became_peered":"2024-03-18T01:34:55.647754+0000","last_unstale":"2024-03-18T01:35:27.339513+0000","last_undegraded":"2024-03-18T01:35:27.339513+0000","last_fullsized":"2024-03-18T01:35:27.339513+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:05:48.508015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119098+0000","last_change":"2024-03-18T01:34:56.651589+0000","last_active":"2024-03-18T01:35:02.119098+0000","last_peered":"2024-03-18T01:35:02.119098+0000","last_clean":"2024-03-18T01:35:02.119098+0000","last_became_active":"2024-03-18T01:34:56.650929+0000","last_became_peered":"2024-03-18T01:34:56.650929+0000","last_unstale":"2024-03-18T01:35:02.119098+0000","last_undegraded":"2024-03-18T01:35:02.119098+0000","last_fullsized":"2024-03-18T01:35:02.119098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:43.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083112+0000","last_change":"2024-03-18T01:35:01.148347+0000","last_active":"2024-03-18T01:35:03.083112+0000","last_peered":"2024-03-18T01:35:03.083112+0000","last_clean":"2024-03-18T01:35:03.083112+0000","last_became_active":"2024-03-18T01:35:01.147936+0000","last_became_peered":"2024-03-18T01:35:01.147936+0000","last_unstale":"2024-03-18T01:35:03.083112+0000","last_undegraded":"2024-03-18T01:35:03.083112+0000","last_fullsized":"2024-03-18T01:35:03.083112+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:12.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:30.645660+0000","last_change":"2024-03-18T01:34:55.653625+0000","last_active":"2024-03-18T01:35:30.645660+0000","last_peered":"2024-03-18T01:35:30.645660+0000","last_clean":"2024-03-18T01:35:30.645660+0000","last_became_active":"2024-03-18T01:34:55.653186+0000","last_became_peered":"2024-03-18T01:34:55.653186+0000","last_unstale":"2024-03-18T01:35:30.645660+0000","last_undegraded":"2024-03-18T01:35:30.645660+0000","last_fullsized":"2024-03-18T01:35:30.645660+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:10:51.193296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119569+0000","last_change":"2024-03-18T01:34:56.652703+0000","last_active":"2024-03-18T01:35:02.119569+0000","last_peered":"2024-03-18T01:35:02.119569+0000","last_clean":"2024-03-18T01:35:02.119569+0000","last_became_active":"2024-03-18T01:34:56.651768+0000","last_became_peered":"2024-03-18T01:34:56.651768+0000","last_unstale":"2024-03-18T01:35:02.119569+0000","last_undegraded":"2024-03-18T01:35:02.119569+0000","last_fullsized":"2024-03-18T01:35:02.119569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:59.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122297+0000","last_change":"2024-03-18T01:35:01.144900+0000","last_active":"2024-03-18T01:35:02.122297+0000","last_peered":"2024-03-18T01:35:02.122297+0000","last_clean":"2024-03-18T01:35:02.122297+0000","last_became_active":"2024-03-18T01:35:01.144431+0000","last_became_peered":"2024-03-18T01:35:01.144431+0000","last_unstale":"2024-03-18T01:35:02.122297+0000","last_undegraded":"2024-03-18T01:35:02.122297+0000","last_fullsized":"2024-03-18T01:35:02.122297+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:36.199904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543786+0000","last_change":"2024-03-18T01:34:55.661259+0000","last_active":"2024-03-18T01:35:03.543786+0000","last_peered":"2024-03-18T01:35:03.543786+0000","last_clean":"2024-03-18T01:35:03.543786+0000","last_became_active":"2024-03-18T01:34:55.660964+0000","last_became_peered":"2024-03-18T01:34:55.660964+0000","last_unstale":"2024-03-18T01:35:03.543786+0000","last_undegraded":"2024-03-18T01:35:03.543786+0000","last_fullsized":"2024-03-18T01:35:03.543786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:14.723905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119306+0000","last_change":"2024-03-18T01:34:56.663075+0000","last_active":"2024-03-18T01:35:02.119306+0000","last_peered":"2024-03-18T01:35:02.119306+0000","last_clean":"2024-03-18T01:35:02.119306+0000","last_became_active":"2024-03-18T01:34:56.662458+0000","last_became_peered":"2024-03-18T01:34:56.662458+0000","last_unstale":"2024-03-18T01:35:02.119306+0000","last_undegraded":"2024-03-18T01:35:02.119306+0000","last_fullsized":"2024-03-18T01:35:02.119306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:31:31.837493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543170+0000","last_change":"2024-03-18T01:35:01.133317+0000","last_active":"2024-03-18T01:35:03.543170+0000","last_peered":"2024-03-18T01:35:03.543170+0000","last_clean":"2024-03-18T01:35:03.543170+0000","last_became_active":"2024-03-18T01:35:01.131798+0000","last_became_peered":"2024-03-18T01:35:01.131798+0000","last_unstale":"2024-03-18T01:35:03.543170+0000","last_undegraded":"2024-03-18T01:35:03.543170+0000","last_fullsized":"2024-03-18T01:35:03.543170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:55.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588762+0000","last_change":"2024-03-18T01:34:55.660887+0000","last_active":"2024-03-18T01:35:03.588762+0000","last_peered":"2024-03-18T01:35:03.588762+0000","last_clean":"2024-03-18T01:35:03.588762+0000","last_became_active":"2024-03-18T01:34:55.660524+0000","last_became_peered":"2024-03-18T01:34:55.660524+0000","last_unstale":"2024-03-18T01:35:03.588762+0000","last_undegraded":"2024-03-18T01:35:03.588762+0000","last_fullsized":"2024-03-18T01:35:03.588762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:24.816951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.119483+0000","last_change":"2024-03-18T01:34:56.654107+0000","last_active":"2024-03-18T01:35:02.119483+0000","last_peered":"2024-03-18T01:35:02.119483+0000","last_clean":"2024-03-18T01:35:02.119483+0000","last_became_active":"2024-03-18T01:34:56.653742+0000","last_became_peered":"2024-03-18T01:34:56.653742+0000","last_unstale":"2024-03-18T01:35:02.119483+0000","last_undegraded":"2024-03-18T01:35:02.119483+0000","last_fullsized":"2024-03-18T01:35:02.119483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:58:46.093589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538656+0000","last_change":"2024-03-18T01:34:56.647818+0000","last_active":"2024-03-18T01:35:02.538656+0000","last_peered":"2024-03-18T01:35:02.538656+0000","last_clean":"2024-03-18T01:35:02.538656+0000","last_became_active":"2024-03-18T01:34:56.647174+0000","last_became_peered":"2024-03-18T01:34:56.647174+0000","last_unstale":"2024-03-18T01:35:02.538656+0000","last_undegraded":"2024-03-18T01:35:02.538656+0000","last_fullsized":"2024-03-18T01:35:02.538656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:55.656083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584569+0000","last_change":"2024-03-18T01:34:56.643221+0000","last_active":"2024-03-18T01:35:03.584569+0000","last_peered":"2024-03-18T01:35:03.584569+0000","last_clean":"2024-03-18T01:35:03.584569+0000","last_became_active":"2024-03-18T01:34:56.642871+0000","last_became_peered":"2024-03-18T01:34:56.642871+0000","last_unstale":"2024-03-18T01:35:03.584569+0000","last_undegraded":"2024-03-18T01:35:03.584569+0000","last_fullsized":"2024-03-18T01:35:03.584569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:23:25.106060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584067+0000","last_change":"2024-03-18T01:34:56.656227+0000","last_active":"2024-03-18T01:35:03.584067+0000","last_peered":"2024-03-18T01:35:03.584067+0000","last_clean":"2024-03-18T01:35:03.584067+0000","last_became_active":"2024-03-18T01:34:56.655859+0000","last_became_peered":"2024-03-18T01:34:56.655859+0000","last_unstale":"2024-03-18T01:35:03.584067+0000","last_undegraded":"2024-03-18T01:35:03.584067+0000","last_fullsized":"2024-03-18T01:35:03.584067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:51:54.424342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.122708+0000","last_change":"2024-03-18T01:34:56.664655+0000","last_active":"2024-03-18T01:35:02.122708+0000","last_peered":"2024-03-18T01:35:02.122708+0000","last_clean":"2024-03-18T01:35:02.122708+0000","last_became_active":"2024-03-18T01:34:56.662526+0000","last_became_peered":"2024-03-18T01:34:56.662526+0000","last_unstale":"2024-03-18T01:35:02.122708+0000","last_undegraded":"2024-03-18T01:35:02.122708+0000","last_fullsized":"2024-03-18T01:35:02.122708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:45:06.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.588782+0000","last_change":"2024-03-18T01:34:56.657045+0000","last_active":"2024-03-18T01:35:03.588782+0000","last_peered":"2024-03-18T01:35:03.588782+0000","last_clean":"2024-03-18T01:35:03.588782+0000","last_became_active":"2024-03-18T01:34:56.656339+0000","last_became_peered":"2024-03-18T01:34:56.656339+0000","last_unstale":"2024-03-18T01:35:03.588782+0000","last_undegraded":"2024-03-18T01:35:03.588782+0000","last_fullsized":"2024-03-18T01:35:03.588782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:41.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538976+0000","last_change":"2024-03-18T01:34:56.647626+0000","last_active":"2024-03-18T01:35:02.538976+0000","last_peered":"2024-03-18T01:35:02.538976+0000","last_clean":"2024-03-18T01:35:02.538976+0000","last_became_active":"2024-03-18T01:34:56.647216+0000","last_became_peered":"2024-03-18T01:34:56.647216+0000","last_unstale":"2024-03-18T01:35:02.538976+0000","last_undegraded":"2024-03-18T01:35:02.538976+0000","last_fullsized":"2024-03-18T01:35:02.538976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:12:33.330925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584700+0000","last_change":"2024-03-18T01:34:56.645091+0000","last_active":"2024-03-18T01:35:03.584700+0000","last_peered":"2024-03-18T01:35:03.584700+0000","last_clean":"2024-03-18T01:35:03.584700+0000","last_became_active":"2024-03-18T01:34:56.644725+0000","last_became_peered":"2024-03-18T01:34:56.644725+0000","last_unstale":"2024-03-18T01:35:03.584700+0000","last_undegraded":"2024-03-18T01:35:03.584700+0000","last_fullsized":"2024-03-18T01:35:03.584700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.585949+0000","last_change":"2024-03-18T01:34:56.651282+0000","last_active":"2024-03-18T01:35:03.585949+0000","last_peered":"2024-03-18T01:35:03.585949+0000","last_clean":"2024-03-18T01:35:03.585949+0000","last_became_active":"2024-03-18T01:34:56.650872+0000","last_became_peered":"2024-03-18T01:34:56.650872+0000","last_unstale":"2024-03-18T01:35:03.585949+0000","last_undegraded":"2024-03-18T01:35:03.585949+0000","last_fullsized":"2024-03-18T01:35:03.585949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:14:45.106848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.543292+0000","last_change":"2024-03-18T01:34:56.651347+0000","last_active":"2024-03-18T01:35:03.543292+0000","last_peered":"2024-03-18T01:35:03.543292+0000","last_clean":"2024-03-18T01:35:03.543292+0000","last_became_active":"2024-03-18T01:34:56.650935+0000","last_became_peered":"2024-03-18T01:34:56.650935+0000","last_unstale":"2024-03-18T01:35:03.543292+0000","last_undegraded":"2024-03-18T01:35:03.543292+0000","last_fullsized":"2024-03-18T01:35:03.543292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:41.849883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.584729+0000","last_change":"2024-03-18T01:34:56.667333+0000","last_active":"2024-03-18T01:35:03.584729+0000","last_peered":"2024-03-18T01:35:03.584729+0000","last_clean":"2024-03-18T01:35:03.584729+0000","last_became_active":"2024-03-18T01:34:56.666845+0000","last_became_peered":"2024-03-18T01:34:56.666845+0000","last_unstale":"2024-03-18T01:35:03.584729+0000","last_undegraded":"2024-03-18T01:35:03.584729+0000","last_fullsized":"2024-03-18T01:35:03.584729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:49:23.813455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.083235+0000","last_change":"2024-03-18T01:34:56.662677+0000","last_active":"2024-03-18T01:35:03.083235+0000","last_peered":"2024-03-18T01:35:03.083235+0000","last_clean":"2024-03-18T01:35:03.083235+0000","last_became_active":"2024-03-18T01:34:56.662306+0000","last_became_peered":"2024-03-18T01:34:56.662306+0000","last_unstale":"2024-03-18T01:35:03.083235+0000","last_undegraded":"2024-03-18T01:35:03.083235+0000","last_fullsized":"2024-03-18T01:35:03.083235+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:53:56.868984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:03.587744+0000","last_change":"2024-03-18T01:34:56.657721+0000","last_active":"2024-03-18T01:35:03.587744+0000","last_peered":"2024-03-18T01:35:03.587744+0000","last_clean":"2024-03-18T01:35:03.587744+0000","last_became_active":"2024-03-18T01:34:56.657072+0000","last_became_peered":"2024-03-18T01:34:56.657072+0000","last_unstale":"2024-03-18T01:35:03.587744+0000","last_undegraded":"2024-03-18T01:35:03.587744+0000","last_fullsized":"2024-03-18T01:35:03.587744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:21.120391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T01:35:02.538849+0000","last_change":"2024-03-18T01:34:56.647327+0000","last_active":"2024-03-18T01:35:02.538849+0000","last_peered":"2024-03-18T01:35:02.538849+0000","last_clean":"2024-03-18T01:35:02.538849+0000","last_became_active":"2024-03-18T01:34:56.646910+0000","last_became_peered":"2024-03-18T01:34:56.646910+0000","last_unstale":"2024-03-18T01:35:02.538849+0000","last_undegraded":"2024-03-18T01:35:02.538849+0000","last_fullsized":"2024-03-18T01:35:02.538849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:37:25.970272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":47991,"num_objects":88,"num_object_clones":0,"num_object_copies":264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":24,"num_read_kb":58,"num_write":208,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":124772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28652,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716500,"statfs":{"total":95995035648,"available":95965696000,"internally_reserved":0,"allocated":1675264,"data_stored":1111616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":482433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070502,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":477953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1613824,"data_stored":1075212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36360,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708792,"statfs":{"total":95995035648,"available":95957803008,"internally_reserved":0,"allocated":1048576,"data_stored":520873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":479393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":502328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36324,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708828,"statfs":{"total":95995035648,"available":95957839872,"internally_reserved":0,"allocated":1011712,"data_stored":477645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":477881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1605632,"data_stored":1069059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":43658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":25113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":44033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:35:34.156 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T01:35:34.157 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T01:35:34.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T01:35:34.655 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:34.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:34 smithi007 ceph-mon[31996]: pgmap v337: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-03-18T01:35:34.984 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:34 smithi007 ceph-mon[31996]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:35:35.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:34 smithi046 ceph-mon[36108]: pgmap v337: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-03-18T01:35:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:34 smithi046 ceph-mon[36108]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:35:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:34 smithi203 ceph-mon[36345]: pgmap v337: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-03-18T01:35:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:34 smithi203 ceph-mon[36345]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:35:36.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:35 smithi046 ceph-mon[36108]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:35:36.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:35 smithi007 ceph-mon[31996]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:35:36.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:35 smithi203 ceph-mon[36345]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:35:37.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:36 smithi007 ceph-mon[31996]: pgmap v338: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-03-18T01:35:37.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:36 smithi046 ceph-mon[36108]: pgmap v338: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-03-18T01:35:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.087+0000 7fc1a6800700 1 -- 172.21.15.7:0/3720649930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a01057a0 msgr2=0x7fc1a0107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:37.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.087+0000 7fc1a6800700 1 --2- 172.21.15.7:0/3720649930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a01057a0 0x7fc1a0107b90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009990 tx=0x7fc18c009ca0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.087+0000 7fc1a6800700 1 -- 172.21.15.7:0/3720649930 shutdown_connections 2024-03-18T01:35:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.087+0000 7fc1a6800700 1 --2- 172.21.15.7:0/3720649930 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1a01080d0 0x7fc1a010a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.087+0000 7fc1a6800700 1 --2- 172.21.15.7:0/3720649930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a0102e70 0x7fc1a0105260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.087+0000 7fc1a6800700 1 --2- 172.21.15.7:0/3720649930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a01057a0 0x7fc1a0107b90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.088+0000 7fc1a6800700 1 -- 172.21.15.7:0/3720649930 >> 172.21.15.7:0/3720649930 conn(0x7fc1a00fc7c0 msgr2=0x7fc1a00fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.088+0000 7fc1a6800700 1 -- 172.21.15.7:0/3720649930 shutdown_connections 2024-03-18T01:35:37.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.088+0000 7fc1a6800700 1 -- 172.21.15.7:0/3720649930 wait complete. 2024-03-18T01:35:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.089+0000 7fc1a6800700 1 Processor -- start 2024-03-18T01:35:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.089+0000 7fc1a6800700 1 -- start start 2024-03-18T01:35:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc1a6800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a0102e70 0x7fc1a019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:37.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc1a6800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1a01057a0 0x7fc1a019abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc1a6800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a01080d0 0x7fc1a01a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc1a6800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a010df70 con 0x7fc1a01057a0 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc1a6800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a010ddf0 con 0x7fc1a01080d0 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc1a6800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a010e0f0 con 0x7fc1a0102e70 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc19f400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a0102e70 0x7fc1a019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc19f400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a0102e70 0x7fc1a019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:46786/0 (socket says 172.21.15.7:46786) 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc19f400700 1 -- 172.21.15.7:0/1859209440 learned_addr learned my addr 172.21.15.7:0/1859209440 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc19ea00700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1a01057a0 0x7fc1a019abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.090+0000 7fc19fe00700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a01080d0 0x7fc1a01a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc19f400700 1 -- 172.21.15.7:0/1859209440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a01080d0 msgr2=0x7fc1a01a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc19f400700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a01080d0 0x7fc1a01a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc19f400700 1 -- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1a01057a0 msgr2=0x7fc1a019abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc19f400700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1a01057a0 0x7fc1a019abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc19f400700 1 -- 172.21.15.7:0/1859209440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a01a2310 con 0x7fc1a0102e70 2024-03-18T01:35:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc19fe00700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a01080d0 0x7fc1a01a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc19f400700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a0102e70 0x7fc1a019a6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc19000e9d0 tx=0x7fc19000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:37.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc197e00700 1 -- 172.21.15.7:0/1859209440 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19000cbd0 con 0x7fc1a0102e70 2024-03-18T01:35:37.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc197e00700 1 -- 172.21.15.7:0/1859209440 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc19000ee70 con 0x7fc1a0102e70 2024-03-18T01:35:37.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc197e00700 1 -- 172.21.15.7:0/1859209440 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190018660 con 0x7fc1a0102e70 2024-03-18T01:35:37.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.091+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a01a2600 con 0x7fc1a0102e70 2024-03-18T01:35:37.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.092+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a01a2b90 con 0x7fc1a0102e70 2024-03-18T01:35:37.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.093+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1a00507e0 con 0x7fc1a0102e70 2024-03-18T01:35:37.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.095+0000 7fc197e00700 1 -- 172.21.15.7:0/1859209440 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fc190010430 con 0x7fc1a0102e70 2024-03-18T01:35:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.096+0000 7fc197e00700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1840a2a50 0x7fc1840a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.096+0000 7fc197e00700 1 -- 172.21.15.7:0/1859209440 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc1900ceff0 con 0x7fc1a0102e70 2024-03-18T01:35:37.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.096+0000 7fc19ea00700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1840a2a50 0x7fc1840a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:37.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.097+0000 7fc19ea00700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1840a2a50 0x7fc1840a4f10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00b2e0 tx=0x7fc18c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:37.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.104+0000 7fc197e00700 1 -- 172.21.15.7:0/1859209440 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc19009bcd0 con 0x7fc1a0102e70 2024-03-18T01:35:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:36 smithi203 ceph-mon[36345]: pgmap v338: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.9 KiB/s wr, 19 op/s 2024-03-18T01:35:37.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.422+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fc1a002df90 con 0x7fc1a0102e70 2024-03-18T01:35:37.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.430+0000 7fc197e00700 1 -- 172.21.15.7:0/1859209440 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fc19000cd70 con 0x7fc1a0102e70 2024-03-18T01:35:37.432 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-03-18T01:35:37.432 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBpmvdlUEJWGRAAn65hjv0JEoE01XdjmpI0pw== 2024-03-18T01:35:37.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.433+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1840a2a50 msgr2=0x7fc1840a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:37.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.433+0000 7fc1a6800700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1840a2a50 0x7fc1840a4f10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc18c00b2e0 tx=0x7fc18c005fd0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.433+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a0102e70 msgr2=0x7fc1a019a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:37.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.433+0000 7fc1a6800700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a0102e70 0x7fc1a019a6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc19000e9d0 tx=0x7fc19000ece0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.434+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 shutdown_connections 2024-03-18T01:35:37.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.434+0000 7fc1a6800700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1840a2a50 0x7fc1840a4f10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.434+0000 7fc1a6800700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1a01080d0 0x7fc1a01a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.434+0000 7fc1a6800700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1a0102e70 0x7fc1a019a6b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.434+0000 7fc1a6800700 1 --2- 172.21.15.7:0/1859209440 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc1a01057a0 0x7fc1a019abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:37.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.434+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 >> 172.21.15.7:0/1859209440 conn(0x7fc1a00fc7c0 msgr2=0x7fc1a0102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:37.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.434+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 shutdown_connections 2024-03-18T01:35:37.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:37.434+0000 7fc1a6800700 1 -- 172.21.15.7:0/1859209440 wait complete. 2024-03-18T01:35:38.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:37 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1859209440' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:35:38.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:37 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:35:38.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:37 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T01:35:38.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:37 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1859209440' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:35:38.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:37 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:35:38.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:37 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T01:35:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:37 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1859209440' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:35:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:37 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T01:35:38.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:37 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T01:35:38.373 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:35:38.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T01:35:38.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T01:35:38.541 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T01:35:38.541 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T01:35:38.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T01:35:39.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:38 smithi046 ceph-mon[36108]: pgmap v339: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.1 KiB/s wr, 21 op/s 2024-03-18T01:35:39.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:38 smithi007 ceph-mon[31996]: pgmap v339: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.1 KiB/s wr, 21 op/s 2024-03-18T01:35:39.083 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:38 smithi203 ceph-mon[36345]: pgmap v339: 129 pgs: 129 active+clean; 624 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.1 KiB/s wr, 21 op/s 2024-03-18T01:35:40.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:40 smithi007 ceph-mon[31996]: pgmap v340: 129 pgs: 129 active+clean; 630 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-18T01:35:40.943 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:40 smithi203 ceph-mon[36345]: pgmap v340: 129 pgs: 129 active+clean; 630 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-18T01:35:41.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:40 smithi046 ceph-mon[36108]: pgmap v340: 129 pgs: 129 active+clean; 630 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-18T01:35:41.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.376+0000 7fdf43200700 1 -- 172.21.15.7:0/1055169510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf441074f0 msgr2=0x7fdf4410f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:41.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.376+0000 7fdf43200700 1 --2- 172.21.15.7:0/1055169510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf441074f0 0x7fdf4410f0c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00b0d0 tx=0x7fdf3c00b3e0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.377+0000 7fdf43200700 1 -- 172.21.15.7:0/1055169510 shutdown_connections 2024-03-18T01:35:41.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.377+0000 7fdf43200700 1 --2- 172.21.15.7:0/1055169510 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf441074f0 0x7fdf4410f0c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.377+0000 7fdf43200700 1 --2- 172.21.15.7:0/1055169510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf44102410 0x7fdf441027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.377+0000 7fdf43200700 1 --2- 172.21.15.7:0/1055169510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf44102dc0 0x7fdf44106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.377+0000 7fdf43200700 1 -- 172.21.15.7:0/1055169510 >> 172.21.15.7:0/1055169510 conn(0x7fdf440fdca0 msgr2=0x7fdf441000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.377+0000 7fdf43200700 1 -- 172.21.15.7:0/1055169510 shutdown_connections 2024-03-18T01:35:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.377+0000 7fdf43200700 1 -- 172.21.15.7:0/1055169510 wait complete. 2024-03-18T01:35:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.378+0000 7fdf43200700 1 Processor -- start 2024-03-18T01:35:41.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.378+0000 7fdf43200700 1 -- start start 2024-03-18T01:35:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf43200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf44102410 0x7fdf441a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf43200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf44102dc0 0x7fdf441a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf43200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf441074f0 0x7fdf4419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf43200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdf44112530 con 0x7fdf44102dc0 2024-03-18T01:35:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf43200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf441123b0 con 0x7fdf44102410 2024-03-18T01:35:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf43200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdf441126b0 con 0x7fdf441074f0 2024-03-18T01:35:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf44102410 0x7fdf441a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:41.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf44102410 0x7fdf441a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:33290/0 (socket says 172.21.15.7:33290) 2024-03-18T01:35:41.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41e00700 1 -- 172.21.15.7:0/668909004 learned_addr learned my addr 172.21.15.7:0/668909004 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:41.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf42800700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf441074f0 0x7fdf4419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:41.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41400700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf44102dc0 0x7fdf441a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:41.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41e00700 1 -- 172.21.15.7:0/668909004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf441074f0 msgr2=0x7fdf4419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41e00700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf441074f0 0x7fdf4419a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41e00700 1 -- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf44102dc0 msgr2=0x7fdf441a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41e00700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf44102dc0 0x7fdf441a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf41e00700 1 -- 172.21.15.7:0/668909004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4419b0b0 con 0x7fdf44102410 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.379+0000 7fdf42800700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf441074f0 0x7fdf4419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.380+0000 7fdf41e00700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf44102410 0x7fdf441a0790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400e9a0 tx=0x7fdf3400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.380+0000 7fdf3aa00700 1 -- 172.21.15.7:0/668909004 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3400ca10 con 0x7fdf44102410 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.380+0000 7fdf3aa00700 1 -- 172.21.15.7:0/668909004 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf3400cbb0 con 0x7fdf44102410 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.380+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4419b3a0 con 0x7fdf44102410 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.380+0000 7fdf3aa00700 1 -- 172.21.15.7:0/668909004 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf34018680 con 0x7fdf44102410 2024-03-18T01:35:41.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.380+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf441a7660 con 0x7fdf44102410 2024-03-18T01:35:41.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.382+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf44068f90 con 0x7fdf44102410 2024-03-18T01:35:41.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.384+0000 7fdf3aa00700 1 -- 172.21.15.7:0/668909004 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fdf34010430 con 0x7fdf44102410 2024-03-18T01:35:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.385+0000 7fdf3aa00700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdf280a29a0 0x7fdf280a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:41.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.386+0000 7fdf3aa00700 1 -- 172.21.15.7:0/668909004 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdf340cefe0 con 0x7fdf44102410 2024-03-18T01:35:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.386+0000 7fdf41400700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdf280a29a0 0x7fdf280a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:41.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.386+0000 7fdf41400700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdf280a29a0 0x7fdf280a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009680 tx=0x7fdf30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:41.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.390+0000 7fdf3aa00700 1 -- 172.21.15.7:0/668909004 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf3409bce0 con 0x7fdf44102410 2024-03-18T01:35:41.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.681+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fdf441a7890 con 0x7fdf44102410 2024-03-18T01:35:41.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.692+0000 7fdf3aa00700 1 -- 172.21.15.7:0/668909004 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262650 (secure 0 0 0) 0x7fdf3409b5e0 con 0x7fdf44102410 2024-03-18T01:35:41.694 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.697+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdf280a29a0 msgr2=0x7fdf280a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.697+0000 7fdf43200700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdf280a29a0 0x7fdf280a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009680 tx=0x7fdf30009260 comp rx=0 tx=0).stop 2024-03-18T01:35:41.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.697+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf44102410 msgr2=0x7fdf441a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.697+0000 7fdf43200700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf44102410 0x7fdf441a0790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400e9a0 tx=0x7fdf3400ecb0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.698+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 shutdown_connections 2024-03-18T01:35:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.698+0000 7fdf43200700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdf280a29a0 0x7fdf280a4e60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.698+0000 7fdf43200700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf441074f0 0x7fdf4419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.698+0000 7fdf43200700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf44102410 0x7fdf441a0790 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.698+0000 7fdf43200700 1 --2- 172.21.15.7:0/668909004 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdf44102dc0 0x7fdf441a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:41.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.698+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 >> 172.21.15.7:0/668909004 conn(0x7fdf440fdca0 msgr2=0x7fdf4410edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:41.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.698+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 shutdown_connections 2024-03-18T01:35:41.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:41.698+0000 7fdf43200700 1 -- 172.21.15.7:0/668909004 wait complete. 2024-03-18T01:35:42.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:41 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/668909004' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T01:35:42.027 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:41 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/668909004' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T01:35:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:41 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/668909004' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T01:35:42.500 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":2743163723},{"type":"v1","addr":"172.21.15.7:6801","nonce":2743163723}]},"active_addr":"172.21.15.7:6801/2743163723","active_change":"2024-03-18T01:26:57.023376+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.7:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1840290075}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":217421055}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2840610426}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":930522449}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3070717075}]}]} 2024-03-18T01:35:42.505 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T01:35:42.506 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T01:35:42.506 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T01:35:43.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:42 smithi046 ceph-mon[36108]: pgmap v341: 129 pgs: 129 active+clean; 630 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-18T01:35:43.031 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:43.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:42 smithi007 ceph-mon[31996]: pgmap v341: 129 pgs: 129 active+clean; 630 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-18T01:35:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:42 smithi203 ceph-mon[36345]: pgmap v341: 129 pgs: 129 active+clean; 630 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-18T01:35:44.875 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:44 smithi007 ceph-mon[31996]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-18T01:35:45.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:44 smithi046 ceph-mon[36108]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-18T01:35:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:44 smithi203 ceph-mon[36345]: pgmap v342: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.5 KiB/s wr, 18 op/s 2024-03-18T01:35:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 -- 172.21.15.7:0/2106687232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c10e8e0 msgr2=0x7fb90c10ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 --2- 172.21.15.7:0/2106687232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c10e8e0 0x7fb90c10ecc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009990 tx=0x7fb8fc009ca0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 -- 172.21.15.7:0/2106687232 shutdown_connections 2024-03-18T01:35:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 --2- 172.21.15.7:0/2106687232 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb90c1092a0 0x7fb90c1157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 --2- 172.21.15.7:0/2106687232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c10e8e0 0x7fb90c10ecc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 --2- 172.21.15.7:0/2106687232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c1088e0 0x7fb90c108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 -- 172.21.15.7:0/2106687232 >> 172.21.15.7:0/2106687232 conn(0x7fb90c075aa0 msgr2=0x7fb90c077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:45.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 -- 172.21.15.7:0/2106687232 shutdown_connections 2024-03-18T01:35:45.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.419+0000 7fb912c00700 1 -- 172.21.15.7:0/2106687232 wait complete. 2024-03-18T01:35:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.420+0000 7fb912c00700 1 Processor -- start 2024-03-18T01:35:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb912c00700 1 -- start start 2024-03-18T01:35:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb912c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c1088e0 0x7fb90c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb912c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb90c1092a0 0x7fb90c19f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb912c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c10e8e0 0x7fb90c1a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb912c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb90c118c30 con 0x7fb90c1092a0 2024-03-18T01:35:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb912c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb90c118ab0 con 0x7fb90c10e8e0 2024-03-18T01:35:45.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb912c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb90c118db0 con 0x7fb90c1088e0 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb903e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb90c1092a0 0x7fb90c19f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb903e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb90c1092a0 0x7fb90c19f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48510/0 (socket says 172.21.15.7:48510) 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb903e00700 1 -- 172.21.15.7:0/3028564969 learned_addr learned my addr 172.21.15.7:0/3028564969 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.421+0000 7fb90be00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c1088e0 0x7fb90c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.422+0000 7fb910c00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c10e8e0 0x7fb90c1a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.422+0000 7fb903e00700 1 -- 172.21.15.7:0/3028564969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c1088e0 msgr2=0x7fb90c19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.422+0000 7fb903e00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c1088e0 0x7fb90c19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.422+0000 7fb903e00700 1 -- 172.21.15.7:0/3028564969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c10e8e0 msgr2=0x7fb90c1a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.422+0000 7fb903e00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c10e8e0 0x7fb90c1a3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.422+0000 7fb903e00700 1 -- 172.21.15.7:0/3028564969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90c1a4040 con 0x7fb90c1092a0 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.422+0000 7fb903e00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb90c1092a0 0x7fb90c19f5c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb8f400e9c0 tx=0x7fb8f400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.423+0000 7fb909600700 1 -- 172.21.15.7:0/3028564969 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f400cbd0 con 0x7fb90c1092a0 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.423+0000 7fb909600700 1 -- 172.21.15.7:0/3028564969 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8f400ee50 con 0x7fb90c1092a0 2024-03-18T01:35:45.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.423+0000 7fb909600700 1 -- 172.21.15.7:0/3028564969 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4005680 con 0x7fb90c1092a0 2024-03-18T01:35:45.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.423+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb90c1a4330 con 0x7fb90c1092a0 2024-03-18T01:35:45.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.423+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb90c1abbb0 con 0x7fb90c1092a0 2024-03-18T01:35:45.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.424+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb90c110e70 con 0x7fb90c1092a0 2024-03-18T01:35:45.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.428+0000 7fb909600700 1 -- 172.21.15.7:0/3028564969 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fb8f4004750 con 0x7fb90c1092a0 2024-03-18T01:35:45.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.429+0000 7fb909600700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb8f00a29a0 0x7fb8f00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:45.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.430+0000 7fb909600700 1 -- 172.21.15.7:0/3028564969 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb8f40cf090 con 0x7fb90c1092a0 2024-03-18T01:35:45.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.430+0000 7fb90be00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb8f00a29a0 0x7fb8f00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:45.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.431+0000 7fb90be00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb8f00a29a0 0x7fb8f00a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00b2e0 tx=0x7fb8fc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:45.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.434+0000 7fb909600700 1 -- 172.21.15.7:0/3028564969 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb8f409bd30 con 0x7fb90c1092a0 2024-03-18T01:35:45.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.661+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb90c110850 con 0x7fb90c1092a0 2024-03-18T01:35:45.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.664+0000 7fb909600700 1 -- 172.21.15.7:0/3028564969 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7fb8f409b630 con 0x7fb90c1092a0 2024-03-18T01:35:45.665 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:45.665 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:45.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb8f00a29a0 msgr2=0x7fb8f00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:45.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb8f00a29a0 0x7fb8f00a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc00b2e0 tx=0x7fb8fc005960 comp rx=0 tx=0).stop 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb90c1092a0 msgr2=0x7fb90c19f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb90c1092a0 0x7fb90c19f5c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb8f400e9c0 tx=0x7fb8f400ecd0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 shutdown_connections 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb8f00a29a0 0x7fb8f00a4e60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb90c10e8e0 0x7fb90c1a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb90c1088e0 0x7fb90c19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 --2- 172.21.15.7:0/3028564969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb90c1092a0 0x7fb90c19f5c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 >> 172.21.15.7:0/3028564969 conn(0x7fb90c075aa0 msgr2=0x7fb90c076170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 shutdown_connections 2024-03-18T01:35:45.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:45.668+0000 7fb912c00700 1 -- 172.21.15.7:0/3028564969 wait complete. 2024-03-18T01:35:46.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:45 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3028564969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:46.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:45 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3028564969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:45 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3028564969' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:46.616 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T01:35:46.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T01:35:47.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:46 smithi046 ceph-mon[36108]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 14 op/s 2024-03-18T01:35:47.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:46 smithi007 ceph-mon[31996]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 14 op/s 2024-03-18T01:35:47.093 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:46 smithi203 ceph-mon[36345]: pgmap v343: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.6 KiB/s wr, 14 op/s 2024-03-18T01:35:49.002 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:48 smithi007 ceph-mon[31996]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-18T01:35:49.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:48 smithi046 ceph-mon[36108]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-18T01:35:49.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:48 smithi203 ceph-mon[36345]: pgmap v344: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-18T01:35:49.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.511+0000 7fd194600700 1 -- 172.21.15.7:0/4026968287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c102f00 msgr2=0x7fd18c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:49.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.511+0000 7fd194600700 1 --2- 172.21.15.7:0/4026968287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c102f00 0x7fd18c106f50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd178009990 tx=0x7fd178009ca0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.512+0000 7fd194600700 1 -- 172.21.15.7:0/4026968287 shutdown_connections 2024-03-18T01:35:49.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.512+0000 7fd194600700 1 --2- 172.21.15.7:0/4026968287 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c107630 0x7fd18c10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.512+0000 7fd194600700 1 --2- 172.21.15.7:0/4026968287 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd18c102550 0x7fd18c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.512+0000 7fd194600700 1 --2- 172.21.15.7:0/4026968287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c102f00 0x7fd18c106f50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.512+0000 7fd194600700 1 -- 172.21.15.7:0/4026968287 >> 172.21.15.7:0/4026968287 conn(0x7fd18c0fdda0 msgr2=0x7fd18c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:49.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.512+0000 7fd194600700 1 -- 172.21.15.7:0/4026968287 shutdown_connections 2024-03-18T01:35:49.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.512+0000 7fd194600700 1 -- 172.21.15.7:0/4026968287 wait complete. 2024-03-18T01:35:49.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.513+0000 7fd194600700 1 Processor -- start 2024-03-18T01:35:49.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.513+0000 7fd194600700 1 -- start start 2024-03-18T01:35:49.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd194600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102550 0x7fd18c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:49.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd194600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd18c102f00 0x7fd18c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:49.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd194600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c107630 0x7fd18c198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:49.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd194600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd18c112660 con 0x7fd18c102f00 2024-03-18T01:35:49.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd194600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd18c1124e0 con 0x7fd18c107630 2024-03-18T01:35:49.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd194600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd18c1127e0 con 0x7fd18c102550 2024-03-18T01:35:49.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd191c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102550 0x7fd18c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:49.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd191200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd18c102f00 0x7fd18c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:49.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd191c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102550 0x7fd18c19e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:40712/0 (socket says 172.21.15.7:40712) 2024-03-18T01:35:49.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd191c00700 1 -- 172.21.15.7:0/3556571130 learned_addr learned my addr 172.21.15.7:0/3556571130 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:35:49.516 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.514+0000 7fd192600700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c107630 0x7fd18c198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:49.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd191c00700 1 -- 172.21.15.7:0/3556571130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c107630 msgr2=0x7fd18c198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:49.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd191c00700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c107630 0x7fd18c198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd191c00700 1 -- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd18c102f00 msgr2=0x7fd18c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:49.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd191c00700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd18c102f00 0x7fd18c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd191c00700 1 -- 172.21.15.7:0/3556571130 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18c1990a0 con 0x7fd18c102550 2024-03-18T01:35:49.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd192600700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c107630 0x7fd18c198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:35:49.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd191c00700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102550 0x7fd18c19e730 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd18000d950 tx=0x7fd18000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:49.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd18a000700 1 -- 172.21.15.7:0/3556571130 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd180016070 con 0x7fd18c102550 2024-03-18T01:35:49.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd18c199390 con 0x7fd18c102550 2024-03-18T01:35:49.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd18a000700 1 -- 172.21.15.7:0/3556571130 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd180009e20 con 0x7fd18c102550 2024-03-18T01:35:49.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd18a000700 1 -- 172.21.15.7:0/3556571130 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18000f620 con 0x7fd18c102550 2024-03-18T01:35:49.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.515+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd18c1a5600 con 0x7fd18c102550 2024-03-18T01:35:49.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.517+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd18c068f90 con 0x7fd18c102550 2024-03-18T01:35:49.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.519+0000 7fd18a000700 1 -- 172.21.15.7:0/3556571130 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fd180019070 con 0x7fd18c102550 2024-03-18T01:35:49.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.520+0000 7fd18a000700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1740a2920 0x7fd1740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:35:49.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.520+0000 7fd18a000700 1 -- 172.21.15.7:0/3556571130 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd1800cee60 con 0x7fd18c102550 2024-03-18T01:35:49.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.520+0000 7fd191200700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1740a2920 0x7fd1740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:35:49.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.521+0000 7fd191200700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1740a2920 0x7fd1740a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd18c101470 tx=0x7fd17800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:35:49.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.527+0000 7fd18a000700 1 -- 172.21.15.7:0/3556571130 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd18009bb00 con 0x7fd18c102550 2024-03-18T01:35:49.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.805+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd18c02db70 con 0x7fd18c102550 2024-03-18T01:35:49.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.806+0000 7fd18a000700 1 -- 172.21.15.7:0/3556571130 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24012 (secure 0 0 0) 0x7fd18009b400 con 0x7fd18c102550 2024-03-18T01:35:49.808 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:35:49.809 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":89,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:01.587936+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:35:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.810+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1740a2920 msgr2=0x7fd1740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.810+0000 7fd194600700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1740a2920 0x7fd1740a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd18c101470 tx=0x7fd17800b260 comp rx=0 tx=0).stop 2024-03-18T01:35:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.810+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102550 msgr2=0x7fd18c19e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:35:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.810+0000 7fd194600700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102550 0x7fd18c19e730 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd18000d950 tx=0x7fd18000dc60 comp rx=0 tx=0).stop 2024-03-18T01:35:49.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.811+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 shutdown_connections 2024-03-18T01:35:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.811+0000 7fd194600700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1740a2920 0x7fd1740a4de0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.811+0000 7fd194600700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd18c107630 0x7fd18c198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.811+0000 7fd194600700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd18c102550 0x7fd18c19e730 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.811+0000 7fd194600700 1 --2- 172.21.15.7:0/3556571130 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd18c102f00 0x7fd18c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:35:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.811+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 >> 172.21.15.7:0/3556571130 conn(0x7fd18c0fdda0 msgr2=0x7fd18c1000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:35:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.811+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 shutdown_connections 2024-03-18T01:35:49.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:35:49.811+0000 7fd194600700 1 -- 172.21.15.7:0/3556571130 wait complete. 2024-03-18T01:35:50.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T01:35:50.702 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T01:35:50.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T01:35:50.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T01:35:50.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T01:35:50.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T01:35:50.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-18T01:35:50.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-18T01:35:50.705 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-18T01:35:50.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-18T01:35:50.706 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-18T01:35:50.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-18T01:35:50.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:50 smithi007 ceph-mon[31996]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-18T01:35:50.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:50 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3556571130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:51.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:50 smithi046 ceph-mon[36108]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-18T01:35:51.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:50 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3556571130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:50 smithi203 ceph-mon[36345]: pgmap v345: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-03-18T01:35:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:50 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3556571130' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:35:51.831 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:51.898 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:51.986 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:52.207 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:53.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:52 smithi007 ceph-mon[31996]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-18T01:35:53.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:52 smithi203 ceph-mon[36345]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-18T01:35:53.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:52 smithi046 ceph-mon[36108]: pgmap v346: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-18T01:35:54.106 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:54.447 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:54.785 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:54.786 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:54.791 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:55.028 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:54 smithi007 ceph-mon[31996]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T01:35:55.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:54 smithi203 ceph-mon[36345]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T01:35:55.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:54 smithi046 ceph-mon[36108]: pgmap v347: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T01:35:57.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:56 smithi007 ceph-mon[31996]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:35:57.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:56 smithi203 ceph-mon[36345]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:35:57.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:56 smithi046 ceph-mon[36108]: pgmap v348: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:35:57.495 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-18T01:35:57.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:57 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:35:58.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-18T01:35:58.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-18T01:35:58.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-18T01:35:58.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T01:35:58.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-18T01:35:58.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-18T01:35:58.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-03-18T01:35:58.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-18T01:35:58.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-18T01:35:58.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-18T01:35:58.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:57 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-18T01:35:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:57 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:35:58.908 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-18T01:35:58.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-18T01:35:58.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:58 smithi007 ceph-mon[31996]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T01:35:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:35:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-18T01:35:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-18T01:35:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-18T01:35:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T01:35:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-18T01:35:59.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-18T01:35:59.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-03-18T01:35:59.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-18T01:35:59.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-18T01:35:59.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-18T01:35:59.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:58 smithi046 ceph-mon[36108]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T01:35:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: pgmap v349: 129 pgs: 129 active+clean; 644 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:35:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-18T01:35:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-18T01:35:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-18T01:35:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T01:35:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-18T01:35:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-18T01:35:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 10]}]': finished 2024-03-18T01:35:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-18T01:35:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-18T01:35:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-18T01:35:59.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:58 smithi203 ceph-mon[36345]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T01:35:59.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:59 smithi007 ceph-mon[31996]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T01:35:59.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:59 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:35:59.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:59 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:35:59.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:59 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:35:59.917 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:35:59 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:59 smithi203 ceph-mon[36345]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T01:36:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:59 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:59 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:00.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:59 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:00.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:35:59 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:00.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:59 smithi046 ceph-mon[36108]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T01:36:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:59 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:59 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:59 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:35:59 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:01.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:00 smithi007 ceph-mon[31996]: pgmap v352: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:00 smithi203 ceph-mon[36345]: pgmap v352: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:00 smithi046 ceph-mon[36108]: pgmap v352: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:02.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:02 smithi007 ceph-mon[31996]: pgmap v353: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:02 smithi203 ceph-mon[36345]: pgmap v353: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:03.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:02 smithi046 ceph-mon[36108]: pgmap v353: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:04.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.364+0000 7f2227800700 1 -- 172.21.15.7:0/298878154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2220104ec0 msgr2=0x7f2220111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.364+0000 7f2227800700 1 --2- 172.21.15.7:0/298878154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2220104ec0 0x7f2220111380 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f221800b0d0 tx=0x7f221800b3e0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.365+0000 7f2227800700 1 -- 172.21.15.7:0/298878154 shutdown_connections 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.365+0000 7f2227800700 1 --2- 172.21.15.7:0/298878154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2220104ec0 0x7f2220111380 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.365+0000 7f2227800700 1 --2- 172.21.15.7:0/298878154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f222010a5b0 0x7f222010a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.365+0000 7f2227800700 1 --2- 172.21.15.7:0/298878154 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2220104500 0x7f2220104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.365+0000 7f2227800700 1 -- 172.21.15.7:0/298878154 >> 172.21.15.7:0/298878154 conn(0x7f22200fffd0 msgr2=0x7f22201023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.365+0000 7f2227800700 1 -- 172.21.15.7:0/298878154 shutdown_connections 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.365+0000 7f2227800700 1 -- 172.21.15.7:0/298878154 wait complete. 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.366+0000 7f2227800700 1 Processor -- start 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2227800700 1 -- start start 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2227800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2220104500 0x7f22201a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2227800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2220104ec0 0x7f22201a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2227800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f222010a5b0 0x7f222019a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2227800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f22201147f0 con 0x7f222010a5b0 2024-03-18T01:36:04.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2227800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2220114670 con 0x7f2220104ec0 2024-03-18T01:36:04.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2227800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2220114970 con 0x7f2220104500 2024-03-18T01:36:04.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2224e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2220104500 0x7f22201a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2224e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2220104500 0x7f22201a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45542/0 (socket says 172.21.15.7:45542) 2024-03-18T01:36:04.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f2224e00700 1 -- 172.21.15.7:0/631732873 learned_addr learned my addr 172.21.15.7:0/631732873 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:04.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.367+0000 7f221fe00700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2220104ec0 0x7f22201a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f2224e00700 1 -- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2220104ec0 msgr2=0x7f22201a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f2224e00700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2220104ec0 0x7f22201a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f2224e00700 1 -- 172.21.15.7:0/631732873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f222010a5b0 msgr2=0x7f222019a9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f2224e00700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f222010a5b0 0x7f222019a9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f2224e00700 1 -- 172.21.15.7:0/631732873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222019b130 con 0x7f2220104500 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f221fe00700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2220104ec0 0x7f22201a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f2224e00700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2220104500 0x7f22201a07c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f221400eb30 tx=0x7f221400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221400cc90 con 0x7f2220104500 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2214004500 con 0x7f2220104500 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.368+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f222019b420 con 0x7f2220104500 2024-03-18T01:36:04.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.369+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22201a7690 con 0x7f2220104500 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.369+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214010430 con 0x7f2220104500 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.369+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2200001030 con 0x7f2220104500 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.372+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f2214003680 con 0x7f2220104500 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.374+0000 7f221d600700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f22080a2920 0x7f22080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.374+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f22140cf590 con 0x7f2220104500 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.374+0000 7f221fe00700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f22080a2920 0x7f22080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.374+0000 7f221d600700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] conn(0x7f22080ada80 0x7f22080aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.374+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 --> [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22140090b0 con 0x7f22080ada80 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.374+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f22140cf940 con 0x7f2220104500 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.375+0000 7f2225800700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] conn(0x7f22080ada80 0x7f22080aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.375+0000 7f221fe00700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f22080a2920 0x7f22080a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2220109b70 tx=0x7f220c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.375+0000 7f2225800700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] conn(0x7f22080ada80 0x7f22080aff20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:04.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.376+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 <== osd.10 v2:172.21.15.46:6824/511858712 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22140090b0 con 0x7f22080ada80 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.397+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 --> [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2200002e60 con 0x7f22080ada80 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.397+0000 7f221d600700 1 -- 172.21.15.7:0/631732873 <== osd.10 v2:172.21.15.46:6824/511858712 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2200002e60 con 0x7f22080ada80 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] conn(0x7f22080ada80 msgr2=0x7f22080aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2227800700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] conn(0x7f22080ada80 0x7f22080aff20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f22080a2920 msgr2=0x7f22080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2227800700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f22080a2920 0x7f22080a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2220109b70 tx=0x7f220c009260 comp rx=0 tx=0).stop 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2220104500 msgr2=0x7f22201a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2227800700 1 --2- 172.21.15.7:0/631732873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2220104500 0x7f22201a07c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f221400eb30 tx=0x7f221400ee40 comp rx=0 tx=0).stop 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2225800700 1 -- 172.21.15.7:0/631732873 reap_dead start 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 shutdown_connections 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.398+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 >> 172.21.15.7:0/631732873 conn(0x7f22200fffd0 msgr2=0x7f2220101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.399+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 shutdown_connections 2024-03-18T01:36:04.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.399+0000 7f2227800700 1 -- 172.21.15.7:0/631732873 wait complete. 2024-03-18T01:36:04.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 -- 172.21.15.7:0/1433924801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc41071c0 msgr2=0x7fbfc4069f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 --2- 172.21.15.7:0/1433924801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc41071c0 0x7fbfc4069f80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009990 tx=0x7fbfb8009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 -- 172.21.15.7:0/1433924801 shutdown_connections 2024-03-18T01:36:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 --2- 172.21.15.7:0/1433924801 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfc4104cd0 0x7fbfc4105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 --2- 172.21.15.7:0/1433924801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc41071c0 0x7fbfc4069f80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 --2- 172.21.15.7:0/1433924801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfc406a4c0 0x7fbfc406a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 -- 172.21.15.7:0/1433924801 >> 172.21.15.7:0/1433924801 conn(0x7fbfc4078100 msgr2=0x7fbfc4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 -- 172.21.15.7:0/1433924801 shutdown_connections 2024-03-18T01:36:04.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.959+0000 7fbfcbc00700 1 -- 172.21.15.7:0/1433924801 wait complete. 2024-03-18T01:36:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.960+0000 7fbfcbc00700 1 Processor -- start 2024-03-18T01:36:04.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.960+0000 7fbfcbc00700 1 -- start start 2024-03-18T01:36:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.961+0000 7fbfcbc00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc406a4c0 0x7fbfc419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.961+0000 7fbfcbc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfc4104cd0 0x7fbfc419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.961+0000 7fbfcbc00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfc41071c0 0x7fbfc41a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.961+0000 7fbfcbc00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc4118ba0 con 0x7fbfc4104cd0 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.961+0000 7fbfcbc00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc4118a20 con 0x7fbfc406a4c0 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.961+0000 7fbfcbc00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc4118d20 con 0x7fbfc41071c0 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc406a4c0 0x7fbfc419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc406a4c0 0x7fbfc419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:36224/0 (socket says 172.21.15.7:36224) 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9200700 1 -- 172.21.15.7:0/945289478 learned_addr learned my addr 172.21.15.7:0/945289478 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9c00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfc41071c0 0x7fbfc41a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc3e00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfc4104cd0 0x7fbfc419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9200700 1 -- 172.21.15.7:0/945289478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfc41071c0 msgr2=0x7fbfc41a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9200700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfc41071c0 0x7fbfc41a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9200700 1 -- 172.21.15.7:0/945289478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfc4104cd0 msgr2=0x7fbfc419f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9200700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbfc4104cd0 0x7fbfc419f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.962+0000 7fbfc9200700 1 -- 172.21.15.7:0/945289478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc41a4080 con 0x7fbfc406a4c0 2024-03-18T01:36:04.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.963+0000 7fbfc9200700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc406a4c0 0x7fbfc419f070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8000c00 tx=0x7fbfb800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.963+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8027070 con 0x7fbfc406a4c0 2024-03-18T01:36:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.963+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfb80056d0 con 0x7fbfc406a4c0 2024-03-18T01:36:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.963+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc41a4310 con 0x7fbfc406a4c0 2024-03-18T01:36:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.963+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8022e10 con 0x7fbfc406a4c0 2024-03-18T01:36:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.963+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc41abbf0 con 0x7fbfc406a4c0 2024-03-18T01:36:04.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.964+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbfc4103e20 con 0x7fbfc406a4c0 2024-03-18T01:36:04.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.966+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fbfb802a070 con 0x7fbfc406a4c0 2024-03-18T01:36:04.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.967+0000 7fbfc1600700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbfac0a29a0 0x7fbfac0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.968+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbfb80d03c0 con 0x7fbfc406a4c0 2024-03-18T01:36:04.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.968+0000 7fbfc3e00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbfac0a29a0 0x7fbfac0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.968+0000 7fbfc1600700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] conn(0x7fbfac0adb00 0x7fbfac0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.968+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 --> [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfb8001d30 con 0x7fbfac0adb00 2024-03-18T01:36:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.968+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fbfb80d0810 con 0x7fbfc406a4c0 2024-03-18T01:36:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.968+0000 7fbfc9c00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] conn(0x7fbfac0adb00 0x7fbfac0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.969+0000 7fbfc3e00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbfac0a29a0 0x7fbfac0a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009680 tx=0x7fbfb0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:04.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.969+0000 7fbfc9c00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] conn(0x7fbfac0adb00 0x7fbfac0affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:04.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.970+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 <== osd.2 v2:172.21.15.203:6800/3828545760 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbfb8001d30 con 0x7fbfac0adb00 2024-03-18T01:36:04.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.990+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 --> [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbfc41017b0 con 0x7fbfac0adb00 2024-03-18T01:36:04.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.990+0000 7fbfc1600700 1 -- 172.21.15.7:0/945289478 <== osd.2 v2:172.21.15.203:6800/3828545760 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbfc41017b0 con 0x7fbfac0adb00 2024-03-18T01:36:04.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 >> [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] conn(0x7fbfac0adb00 msgr2=0x7fbfac0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] conn(0x7fbfac0adb00 0x7fbfac0affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbfac0a29a0 msgr2=0x7fbfac0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbfac0a29a0 0x7fbfac0a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0009680 tx=0x7fbfb0009260 comp rx=0 tx=0).stop 2024-03-18T01:36:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc406a4c0 msgr2=0x7fbfc419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 --2- 172.21.15.7:0/945289478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfc406a4c0 0x7fbfc419f070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8000c00 tx=0x7fbfb800bab0 comp rx=0 tx=0).stop 2024-03-18T01:36:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfc9c00700 1 -- 172.21.15.7:0/945289478 reap_dead start 2024-03-18T01:36:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 shutdown_connections 2024-03-18T01:36:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 >> 172.21.15.7:0/945289478 conn(0x7fbfc4078100 msgr2=0x7fbfc41033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 shutdown_connections 2024-03-18T01:36:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.991+0000 7fbfcbc00700 1 -- 172.21.15.7:0/945289478 wait complete. 2024-03-18T01:36:05.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:04 smithi007 ceph-mon[31996]: pgmap v354: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 -- 172.21.15.7:0/3287215147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddc8075020 msgr2=0x7fddc810e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 --2- 172.21.15.7:0/3287215147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddc8075020 0x7fddc810e5b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009990 tx=0x7fddb4009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 -- 172.21.15.7:0/3287215147 shutdown_connections 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 --2- 172.21.15.7:0/3287215147 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddc810eaf0 0x7fddc8110f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 --2- 172.21.15.7:0/3287215147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddc8074700 0x7fddc8074ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 --2- 172.21.15.7:0/3287215147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddc8075020 0x7fddc810e5b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 -- 172.21.15.7:0/3287215147 >> 172.21.15.7:0/3287215147 conn(0x7fddc80fdd80 msgr2=0x7fddc81001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 -- 172.21.15.7:0/3287215147 shutdown_connections 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.986+0000 7fddcf800700 1 -- 172.21.15.7:0/3287215147 wait complete. 2024-03-18T01:36:05.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.988+0000 7fddcf800700 1 Processor -- start 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.988+0000 7fddcf800700 1 -- start start 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.988+0000 7fddcf800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddc8074700 0x7fddc81a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.988+0000 7fddcf800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddc8075020 0x7fddc81a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.988+0000 7fddcf800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddc810eaf0 0x7fddc819aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.988+0000 7fddcf800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fddc8114820 con 0x7fddc810eaf0 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.988+0000 7fddcf800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fddc81146a0 con 0x7fddc8075020 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.988+0000 7fddcf800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fddc81149a0 con 0x7fddc8074700 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddc8074700 0x7fddc81a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddc8074700 0x7fddc81a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45520/0 (socket says 172.21.15.7:45520) 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 -- 172.21.15.7:0/394371112 learned_addr learned my addr 172.21.15.7:0/394371112 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcd800700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddc810eaf0 0x7fddc819aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 -- 172.21.15.7:0/394371112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddc8075020 msgr2=0x7fddc81a0e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:36:05.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fddc8075020 0x7fddc81a0e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 -- 172.21.15.7:0/394371112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddc810eaf0 msgr2=0x7fddc819aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fddc810eaf0 0x7fddc819aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 -- 172.21.15.7:0/394371112 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc819b240 con 0x7fddc8074700 2024-03-18T01:36:05.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.989+0000 7fddcce00700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddc8074700 0x7fddc81a0920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fddbc00eb30 tx=0x7fddbc00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.990+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc00cc90 con 0x7fddc8074700 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.990+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddbc004500 con 0x7fddc8074700 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.990+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddbc010430 con 0x7fddc8074700 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.990+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc819b530 con 0x7fddc8074700 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.990+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc81a77f0 con 0x7fddc8074700 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.991+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fddc8062a20 con 0x7fddc8074700 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.993+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fddbc004020 con 0x7fddc8074700 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.994+0000 7fddc5600700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fddb00a29a0 0x7fddb00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.994+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fddbc0cf1f0 con 0x7fddc8074700 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.994+0000 7fddc7e00700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fddb00a29a0 0x7fddb00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.995+0000 7fddc5600700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] conn(0x7fddb00adb00 0x7fddb00affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.995+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 --> [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fddbc019e50 con 0x7fddb00adb00 2024-03-18T01:36:05.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.995+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fddbc0d6050 con 0x7fddc8074700 2024-03-18T01:36:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.995+0000 7fddcd800700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] conn(0x7fddb00adb00 0x7fddb00affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.995+0000 7fddc7e00700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fddb00a29a0 0x7fddb00a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fddb40098e0 tx=0x7fddb4005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.996+0000 7fddcd800700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] conn(0x7fddb00adb00 0x7fddb00affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:03.996+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 <== osd.1 v2:172.21.15.46:6800/868303069 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fddbc019e50 con 0x7fddb00adb00 2024-03-18T01:36:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.026+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 --> [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fddc8062e20 con 0x7fddb00adb00 2024-03-18T01:36:05.057 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.026+0000 7fddc5600700 1 -- 172.21.15.7:0/394371112 <== osd.1 v2:172.21.15.46:6800/868303069 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fddc8062e20 con 0x7fddb00adb00 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 >> [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] conn(0x7fddb00adb00 msgr2=0x7fddb00affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] conn(0x7fddb00adb00 0x7fddb00affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fddb00a29a0 msgr2=0x7fddb00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fddb00a29a0 0x7fddb00a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fddb40098e0 tx=0x7fddb4005e80 comp rx=0 tx=0).stop 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddc8074700 msgr2=0x7fddc81a0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 --2- 172.21.15.7:0/394371112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddc8074700 0x7fddc81a0920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fddbc00eb30 tx=0x7fddbc00ee40 comp rx=0 tx=0).stop 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcd800700 1 -- 172.21.15.7:0/394371112 reap_dead start 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 shutdown_connections 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 >> 172.21.15.7:0/394371112 conn(0x7fddc80fdd80 msgr2=0x7fddc80fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 shutdown_connections 2024-03-18T01:36:05.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:04.027+0000 7fddcf800700 1 -- 172.21.15.7:0/394371112 wait complete. 2024-03-18T01:36:05.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:04 smithi203 ceph-mon[36345]: pgmap v354: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:04 smithi046 ceph-mon[36108]: pgmap v354: 129 pgs: 1 active+recovering, 128 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 81 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.145+0000 7f0a2fa00700 1 -- 172.21.15.7:0/4195830011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a2806b420 msgr2=0x7f0a28106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.145+0000 7f0a2fa00700 1 --2- 172.21.15.7:0/4195830011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a2806b420 0x7f0a28106ed0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0a140098f0 tx=0x7f0a14009c00 comp rx=0 tx=0).stop 2024-03-18T01:36:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.146+0000 7f0a2fa00700 1 -- 172.21.15.7:0/4195830011 shutdown_connections 2024-03-18T01:36:06.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.146+0000 7f0a2fa00700 1 --2- 172.21.15.7:0/4195830011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a281075b0 0x7f0a2810f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.146+0000 7f0a2fa00700 1 --2- 172.21.15.7:0/4195830011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a2806aa70 0x7f0a2806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.146+0000 7f0a2fa00700 1 --2- 172.21.15.7:0/4195830011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a2806b420 0x7f0a28106ed0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.146+0000 7f0a2fa00700 1 -- 172.21.15.7:0/4195830011 >> 172.21.15.7:0/4195830011 conn(0x7f0a280772b0 msgr2=0x7f0a280776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.146+0000 7f0a2fa00700 1 -- 172.21.15.7:0/4195830011 shutdown_connections 2024-03-18T01:36:06.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.146+0000 7f0a2fa00700 1 -- 172.21.15.7:0/4195830011 wait complete. 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.147+0000 7f0a2fa00700 1 Processor -- start 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.147+0000 7f0a2fa00700 1 -- start start 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.147+0000 7f0a2fa00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2806aa70 0x7f0a281a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.147+0000 7f0a2fa00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a2806b420 0x7f0a281a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2fa00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281075b0 0x7f0a2819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2fa00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281125a0 con 0x7f0a2806b420 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2fa00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a28112420 con 0x7f0a281075b0 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2fa00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a28112720 con 0x7f0a2806aa70 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2d000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2806aa70 0x7f0a281a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a27e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a2806b420 0x7f0a281a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2d000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2806aa70 0x7f0a281a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:37274/0 (socket says 172.21.15.7:37274) 2024-03-18T01:36:06.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2d000700 1 -- 172.21.15.7:0/54946097 learned_addr learned my addr 172.21.15.7:0/54946097 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:06.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2da00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281075b0 0x7f0a2819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2d000700 1 -- 172.21.15.7:0/54946097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281075b0 msgr2=0x7f0a2819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2d000700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281075b0 0x7f0a2819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2d000700 1 -- 172.21.15.7:0/54946097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a2806b420 msgr2=0x7f0a281a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2d000700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a2806b420 0x7f0a281a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a2d000700 1 -- 172.21.15.7:0/54946097 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2819b230 con 0x7f0a2806aa70 2024-03-18T01:36:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.148+0000 7f0a27e00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a2806b420 0x7f0a281a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:36:06.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.149+0000 7f0a2d000700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2806aa70 0x7f0a281a0910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00e9c0 tx=0x7f0a1c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.149+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c004a30 con 0x7f0a2806aa70 2024-03-18T01:36:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.149+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a1c00cca0 con 0x7f0a2806aa70 2024-03-18T01:36:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.149+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c005680 con 0x7f0a2806aa70 2024-03-18T01:36:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.149+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2819b520 con 0x7f0a2806aa70 2024-03-18T01:36:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.149+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a281a77e0 con 0x7f0a2806aa70 2024-03-18T01:36:06.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.150+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0a28062a60 con 0x7f0a2806aa70 2024-03-18T01:36:06.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.152+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f0a1c00c7c0 con 0x7f0a2806aa70 2024-03-18T01:36:06.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.154+0000 7f0a25600700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a100a29a0 0x7f0a100a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.154+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0a1c0cf3c0 con 0x7f0a2806aa70 2024-03-18T01:36:06.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.154+0000 7f0a27e00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a100a29a0 0x7f0a100a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.154+0000 7f0a25600700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] conn(0x7f0a100adb00 0x7f0a100affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.154+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 --> [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a1c004500 con 0x7f0a100adb00 2024-03-18T01:36:06.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.154+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f0a1c0cf810 con 0x7f0a2806aa70 2024-03-18T01:36:06.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.155+0000 7f0a2da00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] conn(0x7f0a100adb00 0x7f0a100affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.155+0000 7f0a27e00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a100a29a0 0x7f0a100a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a140098c0 tx=0x7f0a14019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:06.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.155+0000 7f0a2da00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] conn(0x7f0a100adb00 0x7f0a100affa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:06.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.156+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 <== osd.11 v2:172.21.15.203:6824/2624070508 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0a1c004500 con 0x7f0a100adb00 2024-03-18T01:36:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.174+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 --> [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0a28062ee0 con 0x7f0a100adb00 2024-03-18T01:36:06.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.175+0000 7f0a25600700 1 -- 172.21.15.7:0/54946097 <== osd.11 v2:172.21.15.203:6824/2624070508 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0a28062ee0 con 0x7f0a100adb00 2024-03-18T01:36:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.175+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 >> [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] conn(0x7f0a100adb00 msgr2=0x7f0a100affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.175+0000 7f0a2fa00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] conn(0x7f0a100adb00 0x7f0a100affa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.175+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a100a29a0 msgr2=0x7f0a100a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.175+0000 7f0a2fa00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a100a29a0 0x7f0a100a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0a140098c0 tx=0x7f0a14019040 comp rx=0 tx=0).stop 2024-03-18T01:36:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.175+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2806aa70 msgr2=0x7f0a281a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.175+0000 7f0a2fa00700 1 --2- 172.21.15.7:0/54946097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a2806aa70 0x7f0a281a0910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00e9c0 tx=0x7f0a1c00ecd0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.176+0000 7f0a2da00700 1 -- 172.21.15.7:0/54946097 reap_dead start 2024-03-18T01:36:06.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.176+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 shutdown_connections 2024-03-18T01:36:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.176+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 >> 172.21.15.7:0/54946097 conn(0x7f0a280772b0 msgr2=0x7f0a2810ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.176+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 shutdown_connections 2024-03-18T01:36:06.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.176+0000 7f0a2fa00700 1 -- 172.21.15.7:0/54946097 wait complete. 2024-03-18T01:36:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 -- 172.21.15.7:0/2080213951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74fc074fc0 msgr2=0x7f74fc10e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 --2- 172.21.15.7:0/2080213951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74fc074fc0 0x7f74fc10e510 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f74e8009990 tx=0x7f74e8009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 -- 172.21.15.7:0/2080213951 shutdown_connections 2024-03-18T01:36:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 --2- 172.21.15.7:0/2080213951 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74fc10ea50 0x7f74fc110e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 --2- 172.21.15.7:0/2080213951 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74fc0746a0 0x7f74fc074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 --2- 172.21.15.7:0/2080213951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74fc074fc0 0x7f74fc10e510 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 -- 172.21.15.7:0/2080213951 >> 172.21.15.7:0/2080213951 conn(0x7f74fc0fdca0 msgr2=0x7f74fc1000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 -- 172.21.15.7:0/2080213951 shutdown_connections 2024-03-18T01:36:06.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.673+0000 7f7502c00700 1 -- 172.21.15.7:0/2080213951 wait complete. 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.674+0000 7f7502c00700 1 Processor -- start 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.674+0000 7f7502c00700 1 -- start start 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.674+0000 7f7502c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74fc0746a0 0x7f74fc1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.674+0000 7f7502c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74fc074fc0 0x7f74fc1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7502c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74fc10ea50 0x7f74fc19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7502c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc114740 con 0x7f74fc10ea50 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7502c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1145c0 con 0x7f74fc074fc0 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7502c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74fc1148c0 con 0x7f74fc0746a0 2024-03-18T01:36:06.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7501800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74fc0746a0 0x7f74fc1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7501800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74fc0746a0 0x7f74fc1a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:37286/0 (socket says 172.21.15.7:37286) 2024-03-18T01:36:06.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7501800700 1 -- 172.21.15.7:0/2890251490 learned_addr learned my addr 172.21.15.7:0/2890251490 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:06.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7500e00700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74fc074fc0 0x7f74fc1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7502200700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74fc10ea50 0x7f74fc19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7501800700 1 -- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74fc074fc0 msgr2=0x7f74fc1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7501800700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74fc074fc0 0x7f74fc1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7501800700 1 -- 172.21.15.7:0/2890251490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74fc10ea50 msgr2=0x7f74fc19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7501800700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74fc10ea50 0x7f74fc19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7501800700 1 -- 172.21.15.7:0/2890251490 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74fc19b180 con 0x7f74fc0746a0 2024-03-18T01:36:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.675+0000 7f7500e00700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74fc074fc0 0x7f74fc1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:36:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.676+0000 7f7501800700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74fc0746a0 0x7f74fc1a0860 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f74f000e9a0 tx=0x7f74f000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:06.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.676+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f000ca10 con 0x7f74fc0746a0 2024-03-18T01:36:06.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.676+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f000cbb0 con 0x7f74fc0746a0 2024-03-18T01:36:06.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.676+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0018660 con 0x7f74fc0746a0 2024-03-18T01:36:06.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.677+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74fc19b470 con 0x7f74fc0746a0 2024-03-18T01:36:06.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.677+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74fc1a7730 con 0x7f74fc0746a0 2024-03-18T01:36:06.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.677+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f74fc062a60 con 0x7f74fc0746a0 2024-03-18T01:36:06.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.680+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f74f001f020 con 0x7f74fc0746a0 2024-03-18T01:36:06.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.681+0000 7f74fa000700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e40a2920 0x7f74e40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.681+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74f00cfa80 con 0x7f74fc0746a0 2024-03-18T01:36:06.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.681+0000 7f7500e00700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e40a2920 0x7f74e40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.682+0000 7f74fa000700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] conn(0x7f74e40ada80 0x7f74e40aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.682+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 --> [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f74e40b05f0 con 0x7f74e40ada80 2024-03-18T01:36:06.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.682+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f74f00cfe30 con 0x7f74fc0746a0 2024-03-18T01:36:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.682+0000 7f7502200700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] conn(0x7f74e40ada80 0x7f74e40aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.682+0000 7f7500e00700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e40a2920 0x7f74e40a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f74fc10fd00 tx=0x7f74e800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.683+0000 7f7502200700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] conn(0x7f74e40ada80 0x7f74e40aff20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:06.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.683+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 <== osd.4 v2:172.21.15.46:6808/35412412 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f74e40b05f0 con 0x7f74e40ada80 2024-03-18T01:36:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.702+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 --> [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f74fc062ee0 con 0x7f74e40ada80 2024-03-18T01:36:06.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.702+0000 7f74fa000700 1 -- 172.21.15.7:0/2890251490 <== osd.4 v2:172.21.15.46:6808/35412412 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f74fc062ee0 con 0x7f74e40ada80 2024-03-18T01:36:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] conn(0x7f74e40ada80 msgr2=0x7f74e40aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] conn(0x7f74e40ada80 0x7f74e40aff20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e40a2920 msgr2=0x7f74e40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e40a2920 0x7f74e40a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f74fc10fd00 tx=0x7f74e800b260 comp rx=0 tx=0).stop 2024-03-18T01:36:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74fc0746a0 msgr2=0x7f74fc1a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 --2- 172.21.15.7:0/2890251490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74fc0746a0 0x7f74fc1a0860 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f74f000e9a0 tx=0x7f74f000ecb0 comp rx=0 tx=0).stop 2024-03-18T01:36:06.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502200700 1 -- 172.21.15.7:0/2890251490 reap_dead start 2024-03-18T01:36:06.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 shutdown_connections 2024-03-18T01:36:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 >> 172.21.15.7:0/2890251490 conn(0x7f74fc0fdca0 msgr2=0x7f74fc0fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 shutdown_connections 2024-03-18T01:36:06.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:06.703+0000 7f7502c00700 1 -- 172.21.15.7:0/2890251490 wait complete. 2024-03-18T01:36:06.846 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:06 smithi007 ceph-mon[31996]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T01:36:07.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:06 smithi203 ceph-mon[36345]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T01:36:07.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:06 smithi046 ceph-mon[36108]: pgmap v355: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s; 81 B/s, 1 keys/s, 1 objects/s recovering 2024-03-18T01:36:09.097 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:08 smithi007 ceph-mon[31996]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s; 69 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:08 smithi203 ceph-mon[36345]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s; 69 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:09.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:08 smithi046 ceph-mon[36108]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s; 69 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T01:36:09.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 -- 172.21.15.7:0/4021943231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101460 msgr2=0x7f1668101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:09.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 --2- 172.21.15.7:0/4021943231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101460 0x7f1668101840 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f16580099c0 tx=0x7f1658009cd0 comp rx=0 tx=0).stop 2024-03-18T01:36:09.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 -- 172.21.15.7:0/4021943231 shutdown_connections 2024-03-18T01:36:09.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 --2- 172.21.15.7:0/4021943231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1668106480 0x7f166810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:09.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 --2- 172.21.15.7:0/4021943231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101460 0x7f1668101840 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:09.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 --2- 172.21.15.7:0/4021943231 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1668101e10 0x7f1668105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:09.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 -- 172.21.15.7:0/4021943231 >> 172.21.15.7:0/4021943231 conn(0x7f16680772b0 msgr2=0x7f16680776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:09.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 -- 172.21.15.7:0/4021943231 shutdown_connections 2024-03-18T01:36:09.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.286+0000 7f166ce00700 1 -- 172.21.15.7:0/4021943231 wait complete. 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.287+0000 7f166ce00700 1 Processor -- start 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.287+0000 7f166ce00700 1 -- start start 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f166ce00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1668101460 0x7f1668102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f166ce00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101e10 0x7f1668103380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f166ce00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1668106480 0x7f16681038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f166ce00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f16681114b0 con 0x7f1668101460 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f166ce00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1668111330 con 0x7f1668106480 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f166ce00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1668111630 con 0x7f1668101e10 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f1665e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1668101460 0x7f1668102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:09.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f1665400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101e10 0x7f1668103380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f1665400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101e10 0x7f1668103380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:37304/0 (socket says 172.21.15.7:37304) 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f1665400700 1 -- 172.21.15.7:0/3384022176 learned_addr learned my addr 172.21.15.7:0/3384022176 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.288+0000 7f1666800700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1668106480 0x7f16681038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f1665400700 1 -- 172.21.15.7:0/3384022176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1668106480 msgr2=0x7f16681038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f1665400700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1668106480 0x7f16681038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f1665400700 1 -- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1668101460 msgr2=0x7f1668102e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f1665400700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1668101460 0x7f1668102e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f1665400700 1 -- 172.21.15.7:0/3384022176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16681a6f60 con 0x7f1668101e10 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f1665e00700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1668101460 0x7f1668102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:36:09.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f1665400700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101e10 0x7f1668103380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f165400e970 tx=0x7f165400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:09.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654004a30 con 0x7f1668101e10 2024-03-18T01:36:09.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f165400ccc0 con 0x7f1668101e10 2024-03-18T01:36:09.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16681a7250 con 0x7f1668101e10 2024-03-18T01:36:09.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1654005650 con 0x7f1668101e10 2024-03-18T01:36:09.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.289+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16681a7790 con 0x7f1668101e10 2024-03-18T01:36:09.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.290+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f16680629e0 con 0x7f1668101e10 2024-03-18T01:36:09.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.295+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f1654004bd0 con 0x7f1668101e10 2024-03-18T01:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.296+0000 7f165ea00700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f164c0a2920 0x7f164c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:09.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.296+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f16540cf320 con 0x7f1668101e10 2024-03-18T01:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.296+0000 7f1665e00700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f164c0a2920 0x7f164c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.296+0000 7f165ea00700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] conn(0x7f164c0ada80 0x7f164c0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.296+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 --> [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f164c0b05f0 con 0x7f164c0ada80 2024-03-18T01:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.296+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f16540cf6d0 con 0x7f1668101e10 2024-03-18T01:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.297+0000 7f1666800700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] conn(0x7f164c0ada80 0x7f164c0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:09.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.297+0000 7f1665e00700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f164c0a2920 0x7f164c0a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1658000c00 tx=0x7f1658005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:09.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.298+0000 7f1666800700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] conn(0x7f164c0ada80 0x7f164c0aff20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:09.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.299+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 <== osd.6 v2:172.21.15.7:6818/3071783622 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f164c0b05f0 con 0x7f164c0ada80 2024-03-18T01:36:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.317+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 --> [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1668062e60 con 0x7f164c0ada80 2024-03-18T01:36:09.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.318+0000 7f165ea00700 1 -- 172.21.15.7:0/3384022176 <== osd.6 v2:172.21.15.7:6818/3071783622 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1668062e60 con 0x7f164c0ada80 2024-03-18T01:36:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.318+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] conn(0x7f164c0ada80 msgr2=0x7f164c0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:09.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.318+0000 7f166ce00700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] conn(0x7f164c0ada80 0x7f164c0aff20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.318+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f164c0a2920 msgr2=0x7f164c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.318+0000 7f166ce00700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f164c0a2920 0x7f164c0a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1658000c00 tx=0x7f1658005fb0 comp rx=0 tx=0).stop 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.318+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101e10 msgr2=0x7f1668103380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.318+0000 7f166ce00700 1 --2- 172.21.15.7:0/3384022176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1668101e10 0x7f1668103380 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f165400e970 tx=0x7f165400ec80 comp rx=0 tx=0).stop 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.319+0000 7f1666800700 1 -- 172.21.15.7:0/3384022176 reap_dead start 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.319+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 shutdown_connections 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.319+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 >> 172.21.15.7:0/3384022176 conn(0x7f16680772b0 msgr2=0x7f1668105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.319+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 shutdown_connections 2024-03-18T01:36:09.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:09.319+0000 7f166ce00700 1 -- 172.21.15.7:0/3384022176 wait complete. 2024-03-18T01:36:09.875 INFO:teuthology.orchestra.run.smithi007.stdout:141733920835 2024-03-18T01:36:09.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T01:36:11.110 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:10 smithi007 ceph-mon[31996]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s; 62 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T01:36:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:10 smithi203 ceph-mon[36345]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s; 62 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T01:36:11.168 INFO:teuthology.orchestra.run.smithi007.stdout:309237645344 2024-03-18T01:36:11.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-18T01:36:11.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.184+0000 7f0ce3400700 1 -- 172.21.15.7:0/3851728101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cdc06a920 msgr2=0x7f0cdc10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.184+0000 7f0ce3400700 1 --2- 172.21.15.7:0/3851728101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cdc06a920 0x7f0cdc10f160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400b0a0 tx=0x7f0cd400b3b0 comp rx=0 tx=0).stop 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.185+0000 7f0ce3400700 1 -- 172.21.15.7:0/3851728101 shutdown_connections 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.185+0000 7f0ce3400700 1 --2- 172.21.15.7:0/3851728101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cdc06a920 0x7f0cdc10f160 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.185+0000 7f0ce3400700 1 --2- 172.21.15.7:0/3851728101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc108320 0x7f0cdc108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.185+0000 7f0ce3400700 1 --2- 172.21.15.7:0/3851728101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cdc069f80 0x7f0cdc06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.185+0000 7f0ce3400700 1 -- 172.21.15.7:0/3851728101 >> 172.21.15.7:0/3851728101 conn(0x7f0cdc078100 msgr2=0x7f0cdc078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.185+0000 7f0ce3400700 1 -- 172.21.15.7:0/3851728101 shutdown_connections 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.185+0000 7f0ce3400700 1 -- 172.21.15.7:0/3851728101 wait complete. 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.186+0000 7f0ce3400700 1 Processor -- start 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.186+0000 7f0ce3400700 1 -- start start 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce3400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc069f80 0x7f0cdc1012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:11.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce3400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cdc06a920 0x7f0cdc1017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce3400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cdc108320 0x7f0cdc101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce3400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc1125c0 con 0x7f0cdc06a920 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce3400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc112440 con 0x7f0cdc108320 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce3400700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc112740 con 0x7f0cdc069f80 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce0a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc069f80 0x7f0cdc1012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0cdbe00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cdc06a920 0x7f0cdc1017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce0a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc069f80 0x7f0cdc1012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:37316/0 (socket says 172.21.15.7:37316) 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce0a00700 1 -- 172.21.15.7:0/2545399439 learned_addr learned my addr 172.21.15.7:0/2545399439 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce1400700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cdc108320 0x7f0cdc101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce1400700 1 -- 172.21.15.7:0/2545399439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc069f80 msgr2=0x7f0cdc1012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce1400700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cdc069f80 0x7f0cdc1012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce1400700 1 -- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cdc06a920 msgr2=0x7f0cdc1017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce1400700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0cdc06a920 0x7f0cdc1017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:11.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.187+0000 7f0ce1400700 1 -- 172.21.15.7:0/2545399439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc1a6f20 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.188+0000 7f0ce1400700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cdc108320 0x7f0cdc101d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0cd40062a0 tx=0x7f0cd4004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.188+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4021070 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.188+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cdc1a71b0 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.188+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd4003c00 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.188+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd40045a0 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.188+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cdc1a76b0 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.189+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0cdc0629a0 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.193+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f0cd4004740 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.194+0000 7f0cd9600700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0cc40a29a0 0x7f0cc40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.195+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0cd400f020 con 0x7f0cdc108320 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.195+0000 7f0ce0a00700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0cc40a29a0 0x7f0cc40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.195+0000 7f0cd9600700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] conn(0x7f0cc40adb00 0x7f0cc40affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:11.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.195+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 --> [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0cd4009ce0 con 0x7f0cc40adb00 2024-03-18T01:36:11.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.195+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f0cd40d0810 con 0x7f0cdc108320 2024-03-18T01:36:11.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.195+0000 7f0cdbe00700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] conn(0x7f0cc40adb00 0x7f0cc40affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:11.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.196+0000 7f0ce0a00700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0cc40a29a0 0x7f0cc40a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009ca0 tx=0x7f0cd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:11.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.196+0000 7f0cdbe00700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] conn(0x7f0cc40adb00 0x7f0cc40affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.198+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 <== osd.9 v2:172.21.15.7:6826/4123652106 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0cd4009ce0 con 0x7f0cc40adb00 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.223+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 --> [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0cdc062e20 con 0x7f0cc40adb00 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.223+0000 7f0cd9600700 1 -- 172.21.15.7:0/2545399439 <== osd.9 v2:172.21.15.7:6826/4123652106 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0cdc062e20 con 0x7f0cc40adb00 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] conn(0x7f0cc40adb00 msgr2=0x7f0cc40affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce3400700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] conn(0x7f0cc40adb00 0x7f0cc40affa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0cc40a29a0 msgr2=0x7f0cc40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce3400700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0cc40a29a0 0x7f0cc40a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009ca0 tx=0x7f0cd0009260 comp rx=0 tx=0).stop 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cdc108320 msgr2=0x7f0cdc101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce3400700 1 --2- 172.21.15.7:0/2545399439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cdc108320 0x7f0cdc101d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0cd40062a0 tx=0x7f0cd4004c60 comp rx=0 tx=0).stop 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce1400700 1 -- 172.21.15.7:0/2545399439 reap_dead start 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 shutdown_connections 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.224+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 >> 172.21.15.7:0/2545399439 conn(0x7f0cdc078100 msgr2=0x7f0cdc0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.225+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 shutdown_connections 2024-03-18T01:36:11.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:11.225+0000 7f0ce3400700 1 -- 172.21.15.7:0/2545399439 wait complete. 2024-03-18T01:36:11.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:10 smithi046 ceph-mon[36108]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s; 62 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T01:36:13.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:12 smithi007 ceph-mon[31996]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:13.064 INFO:teuthology.orchestra.run.smithi007.stdout:55834574931 2024-03-18T01:36:13.065 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T01:36:13.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:12 smithi203 ceph-mon[36345]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:13.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:12 smithi046 ceph-mon[36108]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:13.409 INFO:teuthology.orchestra.run.smithi007.stdout:197568495672 2024-03-18T01:36:13.409 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-18T01:36:13.887 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:13.930 INFO:teuthology.orchestra.run.smithi007.stdout:287762808871 2024-03-18T01:36:13.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-18T01:36:14.954 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:14 smithi007 ceph-mon[31996]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:15.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:14 smithi203 ceph-mon[36345]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:15.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:14 smithi046 ceph-mon[36108]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 -- 172.21.15.7:0/2909810182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 msgr2=0x7f52941027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 --2- 172.21.15.7:0/2909810182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 0x7f52941027a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5280009920 tx=0x7f5280009c30 comp rx=0 tx=0).stop 2024-03-18T01:36:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 -- 172.21.15.7:0/2909810182 shutdown_connections 2024-03-18T01:36:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 --2- 172.21.15.7:0/2909810182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5294102ce0 0x7f529410f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 --2- 172.21.15.7:0/2909810182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5294108300 0x7f52941086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 --2- 172.21.15.7:0/2909810182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 0x7f52941027a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 -- 172.21.15.7:0/2909810182 >> 172.21.15.7:0/2909810182 conn(0x7f5294079400 msgr2=0x7f5294079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 -- 172.21.15.7:0/2909810182 shutdown_connections 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.806+0000 7f529c000700 1 -- 172.21.15.7:0/2909810182 wait complete. 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.808+0000 7f529c000700 1 Processor -- start 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.808+0000 7f529c000700 1 -- start start 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f529c000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 0x7f52941a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f529c000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5294102ce0 0x7f52941a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f529c000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5294108300 0x7f529419ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f529c000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f52941125a0 con 0x7f5294102320 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f529c000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5294112420 con 0x7f5294102ce0 2024-03-18T01:36:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f529c000700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5294112720 con 0x7f5294108300 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5299600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 0x7f52941a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5299600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 0x7f52941a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:52526/0 (socket says 172.21.15.7:52526) 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5299600700 1 -- 172.21.15.7:0/3356229917 learned_addr learned my addr 172.21.15.7:0/3356229917 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f529a000700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5294108300 0x7f529419ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5298c00700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5294102ce0 0x7f52941a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5299600700 1 -- 172.21.15.7:0/3356229917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5294108300 msgr2=0x7f529419ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5299600700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5294108300 0x7f529419ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5299600700 1 -- 172.21.15.7:0/3356229917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5294102ce0 msgr2=0x7f52941a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5299600700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5294102ce0 0x7f52941a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.809+0000 7f5299600700 1 -- 172.21.15.7:0/3356229917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529419b260 con 0x7f5294102320 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.810+0000 7f5298c00700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5294102ce0 0x7f52941a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:36:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.810+0000 7f5299600700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 0x7f52941a0940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f528800e9c0 tx=0x7f528800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.810+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528800cbd0 con 0x7f5294102320 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.810+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529419b550 con 0x7f5294102320 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.810+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52941a7810 con 0x7f5294102320 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.810+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528800cd70 con 0x7f5294102320 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.810+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288010430 con 0x7f5294102320 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.812+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5274001030 con 0x7f5294102320 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.813+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f5288004750 con 0x7f5294102320 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.815+0000 7f5292000700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f527c0a29a0 0x7f527c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.815+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f52880cf1a0 con 0x7f5294102320 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.815+0000 7f5298c00700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f527c0a29a0 0x7f527c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.815+0000 7f5292000700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] conn(0x7f527c0adb00 0x7f527c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.815+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 --> [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f527c0b0670 con 0x7f527c0adb00 2024-03-18T01:36:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.815+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f52880d6050 con 0x7f5294102320 2024-03-18T01:36:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.816+0000 7f529a000700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] conn(0x7f527c0adb00 0x7f527c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.816+0000 7f529a000700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] conn(0x7f527c0adb00 0x7f527c0affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.816+0000 7f5298c00700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f527c0a29a0 0x7f527c0a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5294079d10 tx=0x7f5280019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.817+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 <== osd.8 v2:172.21.15.203:6816/1251446815 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f527c0b0670 con 0x7f527c0adb00 2024-03-18T01:36:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.835+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 --> [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5274002e60 con 0x7f527c0adb00 2024-03-18T01:36:15.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.835+0000 7f5292000700 1 -- 172.21.15.7:0/3356229917 <== osd.8 v2:172.21.15.203:6816/1251446815 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5274002e60 con 0x7f527c0adb00 2024-03-18T01:36:15.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 >> [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] conn(0x7f527c0adb00 msgr2=0x7f527c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:15.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] conn(0x7f527c0adb00 0x7f527c0affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:15.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f527c0a29a0 msgr2=0x7f527c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:15.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f527c0a29a0 0x7f527c0a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5294079d10 tx=0x7f5280019040 comp rx=0 tx=0).stop 2024-03-18T01:36:15.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 msgr2=0x7f52941a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 --2- 172.21.15.7:0/3356229917 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5294102320 0x7f52941a0940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f528800e9c0 tx=0x7f528800ecd0 comp rx=0 tx=0).stop 2024-03-18T01:36:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529a000700 1 -- 172.21.15.7:0/3356229917 reap_dead start 2024-03-18T01:36:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 shutdown_connections 2024-03-18T01:36:15.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 >> 172.21.15.7:0/3356229917 conn(0x7f5294079400 msgr2=0x7f5294074810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:15.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 shutdown_connections 2024-03-18T01:36:15.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:14.836+0000 7f529c000700 1 -- 172.21.15.7:0/3356229917 wait complete. 2024-03-18T01:36:16.247 INFO:teuthology.orchestra.run.smithi007.stdout:77309411407 2024-03-18T01:36:16.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T01:36:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.596+0000 7f2d18e00700 1 -- 172.21.15.7:0/3551334580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14108320 msgr2=0x7f2d14108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.596+0000 7f2d18e00700 1 --2- 172.21.15.7:0/3551334580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14108320 0x7f2d14108700 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2d04009920 tx=0x7f2d04009c30 comp rx=0 tx=0).stop 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.596+0000 7f2d18e00700 1 -- 172.21.15.7:0/3551334580 shutdown_connections 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.596+0000 7f2d18e00700 1 --2- 172.21.15.7:0/3551334580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d1406a920 0x7f2d1410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.596+0000 7f2d18e00700 1 --2- 172.21.15.7:0/3551334580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14108320 0x7f2d14108700 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.596+0000 7f2d18e00700 1 --2- 172.21.15.7:0/3551334580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d14069f80 0x7f2d1406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.596+0000 7f2d18e00700 1 -- 172.21.15.7:0/3551334580 >> 172.21.15.7:0/3551334580 conn(0x7f2d14078100 msgr2=0x7f2d14078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.597+0000 7f2d18e00700 1 -- 172.21.15.7:0/3551334580 shutdown_connections 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.597+0000 7f2d18e00700 1 -- 172.21.15.7:0/3551334580 wait complete. 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.598+0000 7f2d18e00700 1 Processor -- start 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.598+0000 7f2d18e00700 1 -- start start 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d18e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14069f80 0x7f2d141a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d18e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d1406a920 0x7f2d141a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d18e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d14108320 0x7f2d1419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d18e00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2d141125b0 con 0x7f2d1406a920 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d18e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2d14112430 con 0x7f2d14108320 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d18e00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d14112730 con 0x7f2d14069f80 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d11e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14069f80 0x7f2d141a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d11400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d1406a920 0x7f2d141a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d11e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14069f80 0x7f2d141a0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36864/0 (socket says 172.21.15.7:36864) 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d11e00700 1 -- 172.21.15.7:0/380437074 learned_addr learned my addr 172.21.15.7:0/380437074 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d12800700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d14108320 0x7f2d1419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.599+0000 7f2d11e00700 1 -- 172.21.15.7:0/380437074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d14108320 msgr2=0x7f2d1419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.600+0000 7f2d11e00700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d14108320 0x7f2d1419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.600+0000 7f2d11e00700 1 -- 172.21.15.7:0/380437074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d1406a920 msgr2=0x7f2d141a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.600+0000 7f2d11e00700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2d1406a920 0x7f2d141a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.600+0000 7f2d11e00700 1 -- 172.21.15.7:0/380437074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1419b410 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.600+0000 7f2d11e00700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14069f80 0x7f2d141a0a60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2d04000c00 tx=0x7f2d0400fb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.600+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d04027070 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.600+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1419b6a0 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.600+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d040056d0 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.601+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d040049c0 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.601+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d141090a0 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.603+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d1410df10 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.604+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f2d0402a070 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.607+0000 7f2d0aa00700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2cf80a2a70 0x7f2cf80a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.607+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2d040cf930 con 0x7f2d14069f80 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.607+0000 7f2d11400700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2cf80a2a70 0x7f2cf80a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.608+0000 7f2d0aa00700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] conn(0x7f2cf80adaa0 0x7f2cf80aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.608+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 --> [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d0400bde0 con 0x7f2cf80adaa0 2024-03-18T01:36:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.608+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f2d040d42a0 con 0x7f2d14069f80 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.608+0000 7f2d12800700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] conn(0x7f2cf80adaa0 0x7f2cf80aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.608+0000 7f2d11400700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2cf80a2a70 0x7f2cf80a4f30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2d1419c280 tx=0x7f2d00016040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.609+0000 7f2d12800700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] conn(0x7f2cf80adaa0 0x7f2cf80aff40 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.610+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 <== osd.7 v2:172.21.15.46:6816/3445048558 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d0400bde0 con 0x7f2cf80adaa0 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.634+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 --> [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2d141039d0 con 0x7f2cf80adaa0 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.634+0000 7f2d0aa00700 1 -- 172.21.15.7:0/380437074 <== osd.7 v2:172.21.15.46:6816/3445048558 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2d141039d0 con 0x7f2cf80adaa0 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 >> [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] conn(0x7f2cf80adaa0 msgr2=0x7f2cf80aff40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] conn(0x7f2cf80adaa0 0x7f2cf80aff40 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:16.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2cf80a2a70 msgr2=0x7f2cf80a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2cf80a2a70 0x7f2cf80a4f30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2d1419c280 tx=0x7f2d00016040 comp rx=0 tx=0).stop 2024-03-18T01:36:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14069f80 msgr2=0x7f2d141a0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 --2- 172.21.15.7:0/380437074 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d14069f80 0x7f2d141a0a60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2d04000c00 tx=0x7f2d0400fb40 comp rx=0 tx=0).stop 2024-03-18T01:36:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d12800700 1 -- 172.21.15.7:0/380437074 reap_dead start 2024-03-18T01:36:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 shutdown_connections 2024-03-18T01:36:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 >> 172.21.15.7:0/380437074 conn(0x7f2d14078100 msgr2=0x7f2d1410d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 shutdown_connections 2024-03-18T01:36:16.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:16.635+0000 7f2d18e00700 1 -- 172.21.15.7:0/380437074 wait complete. 2024-03-18T01:36:17.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:16 smithi007 ceph-mon[31996]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:16 smithi203 ceph-mon[36345]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:17.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:16 smithi046 ceph-mon[36108]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-18T01:36:17.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.722+0000 7f7cd4600700 1 -- 172.21.15.7:0/2367690606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc0746a0 msgr2=0x7f7ccc074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.722+0000 7f7cd4600700 1 --2- 172.21.15.7:0/2367690606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc0746a0 0x7f7ccc074a80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009990 tx=0x7f7cc0009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.722+0000 7f7cd4600700 1 -- 172.21.15.7:0/2367690606 shutdown_connections 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.722+0000 7f7cd4600700 1 --2- 172.21.15.7:0/2367690606 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc10eb90 0x7f7ccc110fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.722+0000 7f7cd4600700 1 --2- 172.21.15.7:0/2367690606 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc0746a0 0x7f7ccc074a80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.722+0000 7f7cd4600700 1 --2- 172.21.15.7:0/2367690606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ccc074fc0 0x7f7ccc10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.722+0000 7f7cd4600700 1 -- 172.21.15.7:0/2367690606 >> 172.21.15.7:0/2367690606 conn(0x7f7ccc0fdda0 msgr2=0x7f7ccc1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.722+0000 7f7cd4600700 1 -- 172.21.15.7:0/2367690606 shutdown_connections 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.723+0000 7f7cd4600700 1 -- 172.21.15.7:0/2367690606 wait complete. 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.724+0000 7f7cd4600700 1 Processor -- start 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.724+0000 7f7cd4600700 1 -- start start 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd4600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0746a0 0x7f7ccc1a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:17.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd4600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc074fc0 0x7f7ccc1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd4600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ccc10eb90 0x7f7ccc19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd4600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc114880 con 0x7f7ccc074fc0 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd4600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc114700 con 0x7f7ccc0746a0 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd4600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc114a00 con 0x7f7ccc10eb90 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0746a0 0x7f7ccc1a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0746a0 0x7f7ccc1a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:50210/0 (socket says 172.21.15.7:50210) 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1c00700 1 -- 172.21.15.7:0/4017305477 learned_addr learned my addr 172.21.15.7:0/4017305477 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1200700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc074fc0 0x7f7ccc1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd2600700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ccc10eb90 0x7f7ccc19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1c00700 1 -- 172.21.15.7:0/4017305477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ccc10eb90 msgr2=0x7f7ccc19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1c00700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ccc10eb90 0x7f7ccc19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1c00700 1 -- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc074fc0 msgr2=0x7f7ccc1a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:17.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1c00700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ccc074fc0 0x7f7ccc1a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.725+0000 7f7cd1c00700 1 -- 172.21.15.7:0/4017305477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc19b400 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.726+0000 7f7cd1c00700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0746a0 0x7f7ccc1a0a50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0000c00 tx=0x7f7cc000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.726+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0027070 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.726+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc19b690 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.726+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cc0003d70 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.726+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000f460 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.726+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc1a7800 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.727+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ccc062a20 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.729+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f7cc002a070 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.730+0000 7f7cca000700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7cb40a2920 0x7f7cb40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.731+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7cc00cfe70 con 0x7f7ccc0746a0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.731+0000 7f7cd1200700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7cb40a2920 0x7f7cb40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.731+0000 7f7cca000700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] conn(0x7f7cb40ada80 0x7f7cb40aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:17.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.731+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 --> [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7cb40b05f0 con 0x7f7cb40ada80 2024-03-18T01:36:17.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.731+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7cc00d2490 con 0x7f7ccc0746a0 2024-03-18T01:36:17.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.731+0000 7f7cd2600700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] conn(0x7f7cb40ada80 0x7f7cb40aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:17.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.732+0000 7f7cd1200700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7cb40a2920 0x7f7cb40a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc19c270 tx=0x7f7cb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:17.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.732+0000 7f7cd2600700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] conn(0x7f7cb40ada80 0x7f7cb40aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.733+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 <== osd.0 v2:172.21.15.7:6802/2740971601 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7cb40b05f0 con 0x7f7cb40ada80 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.752+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 --> [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7ccc062ea0 con 0x7f7cb40ada80 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.752+0000 7f7cca000700 1 -- 172.21.15.7:0/4017305477 <== osd.0 v2:172.21.15.7:6802/2740971601 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7ccc062ea0 con 0x7f7cb40ada80 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] conn(0x7f7cb40ada80 msgr2=0x7f7cb40aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] conn(0x7f7cb40ada80 0x7f7cb40aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7cb40a2920 msgr2=0x7f7cb40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7cb40a2920 0x7f7cb40a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc19c270 tx=0x7f7cb8009260 comp rx=0 tx=0).stop 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0746a0 msgr2=0x7f7ccc1a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 --2- 172.21.15.7:0/4017305477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ccc0746a0 0x7f7ccc1a0a50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0000c00 tx=0x7f7cc000bab0 comp rx=0 tx=0).stop 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd2600700 1 -- 172.21.15.7:0/4017305477 reap_dead start 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 shutdown_connections 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 >> 172.21.15.7:0/4017305477 conn(0x7f7ccc0fdda0 msgr2=0x7f7ccc0fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:17.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 shutdown_connections 2024-03-18T01:36:17.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:17.753+0000 7f7cd4600700 1 -- 172.21.15.7:0/4017305477 wait complete. 2024-03-18T01:36:18.525 INFO:teuthology.orchestra.run.smithi007.stdout:253403070511 2024-03-18T01:36:18.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-18T01:36:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.759+0000 7f1b79400700 1 -- 172.21.15.7:0/458610704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74069f80 msgr2=0x7f1b7406a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.759+0000 7f1b79400700 1 --2- 172.21.15.7:0/458610704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74069f80 0x7f1b7406a3e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1b600099c0 tx=0x7f1b60009cd0 comp rx=0 tx=0).stop 2024-03-18T01:36:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.760+0000 7f1b79400700 1 -- 172.21.15.7:0/458610704 shutdown_connections 2024-03-18T01:36:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.760+0000 7f1b79400700 1 --2- 172.21.15.7:0/458610704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7406a920 0x7f1b7410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.760+0000 7f1b79400700 1 --2- 172.21.15.7:0/458610704 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b74108320 0x7f1b74108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.760+0000 7f1b79400700 1 --2- 172.21.15.7:0/458610704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74069f80 0x7f1b7406a3e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.760+0000 7f1b79400700 1 -- 172.21.15.7:0/458610704 >> 172.21.15.7:0/458610704 conn(0x7f1b74078100 msgr2=0x7f1b74078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.760+0000 7f1b79400700 1 -- 172.21.15.7:0/458610704 shutdown_connections 2024-03-18T01:36:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.760+0000 7f1b79400700 1 -- 172.21.15.7:0/458610704 wait complete. 2024-03-18T01:36:18.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.761+0000 7f1b79400700 1 Processor -- start 2024-03-18T01:36:18.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.761+0000 7f1b79400700 1 -- start start 2024-03-18T01:36:18.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b79400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b74069f80 0x7f1b741a3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:18.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b79400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7406a920 0x7f1b741a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:18.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b79400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74108320 0x7f1b7410e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:18.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b79400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1b741125d0 con 0x7f1b74069f80 2024-03-18T01:36:18.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b79400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b74112450 con 0x7f1b7406a920 2024-03-18T01:36:18.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b79400700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1b74112750 con 0x7f1b74108320 2024-03-18T01:36:18.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b73200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74108320 0x7f1b7410e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:18.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b73200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74108320 0x7f1b7410e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36898/0 (socket says 172.21.15.7:36898) 2024-03-18T01:36:18.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b73200700 1 -- 172.21.15.7:0/4155685427 learned_addr learned my addr 172.21.15.7:0/4155685427 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:18.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.762+0000 7f1b72800700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b74069f80 0x7f1b741a3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:18.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b73200700 1 -- 172.21.15.7:0/4155685427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7406a920 msgr2=0x7f1b741a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b71e00700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7406a920 0x7f1b741a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b73200700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b7406a920 0x7f1b741a41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b73200700 1 -- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b74069f80 msgr2=0x7f1b741a3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b73200700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b74069f80 0x7f1b741a3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b73200700 1 -- 172.21.15.7:0/4155685427 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7410ea50 con 0x7f1b74108320 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b72800700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b74069f80 0x7f1b741a3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b73200700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74108320 0x7f1b7410e2c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00ef30 tx=0x7f1b6c00c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c0092e0 con 0x7f1b74108320 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b6c004bb0 con 0x7f1b74108320 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c004280 con 0x7f1b74108320 2024-03-18T01:36:18.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.763+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7410ece0 con 0x7f1b74108320 2024-03-18T01:36:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.764+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b741abbd0 con 0x7f1b74108320 2024-03-18T01:36:18.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.765+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1b74100650 con 0x7f1b74108320 2024-03-18T01:36:18.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.767+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f1b6c01b030 con 0x7f1b74108320 2024-03-18T01:36:18.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.768+0000 7f1b6b400700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b580a29a0 0x7f1b580a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:18.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.768+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1b6c0ceb70 con 0x7f1b74108320 2024-03-18T01:36:18.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.769+0000 7f1b72800700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b580a29a0 0x7f1b580a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:18.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.769+0000 7f1b6b400700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] conn(0x7f1b580adb00 0x7f1b580affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:18.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.769+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 --> [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b580b0670 con 0x7f1b580adb00 2024-03-18T01:36:18.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.769+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f1b6c0d32a0 con 0x7f1b74108320 2024-03-18T01:36:18.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.769+0000 7f1b71e00700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] conn(0x7f1b580adb00 0x7f1b580affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:18.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.769+0000 7f1b72800700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b580a29a0 0x7f1b580a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009c90 tx=0x7f1b64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:18.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.770+0000 7f1b71e00700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] conn(0x7f1b580adb00 0x7f1b580affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:18.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.771+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 <== osd.5 v2:172.21.15.203:6808/2080514850 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b580b0670 con 0x7f1b580adb00 2024-03-18T01:36:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.790+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 --> [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1b741039d0 con 0x7f1b580adb00 2024-03-18T01:36:18.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.790+0000 7f1b6b400700 1 -- 172.21.15.7:0/4155685427 <== osd.5 v2:172.21.15.203:6808/2080514850 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1b741039d0 con 0x7f1b580adb00 2024-03-18T01:36:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 >> [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] conn(0x7f1b580adb00 msgr2=0x7f1b580affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b79400700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] conn(0x7f1b580adb00 0x7f1b580affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b580a29a0 msgr2=0x7f1b580a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b79400700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b580a29a0 0x7f1b580a4e60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009c90 tx=0x7f1b64009260 comp rx=0 tx=0).stop 2024-03-18T01:36:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74108320 msgr2=0x7f1b7410e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b79400700 1 --2- 172.21.15.7:0/4155685427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b74108320 0x7f1b7410e2c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00ef30 tx=0x7f1b6c00c450 comp rx=0 tx=0).stop 2024-03-18T01:36:18.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b73200700 1 -- 172.21.15.7:0/4155685427 reap_dead start 2024-03-18T01:36:18.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 shutdown_connections 2024-03-18T01:36:18.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.791+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 >> 172.21.15.7:0/4155685427 conn(0x7f1b74078100 msgr2=0x7f1b740ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:18.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.792+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 shutdown_connections 2024-03-18T01:36:18.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:18.792+0000 7f1b79400700 1 -- 172.21.15.7:0/4155685427 wait complete. 2024-03-18T01:36:19.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:18 smithi007 ceph-mon[31996]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-18T01:36:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:18 smithi203 ceph-mon[36345]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-18T01:36:19.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:18 smithi046 ceph-mon[36108]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-18T01:36:19.831 INFO:teuthology.orchestra.run.smithi007.stdout:343597383708 2024-03-18T01:36:19.832 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-18T01:36:20.240 INFO:teuthology.orchestra.run.smithi007.stdout:219043332149 2024-03-18T01:36:20.241 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-18T01:36:21.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:20 smithi203 ceph-mon[36345]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-18T01:36:21.247 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:20 smithi007 ceph-mon[31996]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-18T01:36:21.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:20 smithi046 ceph-mon[36108]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-18T01:36:21.303 INFO:teuthology.orchestra.run.smithi007.stdout:34359738461 2024-03-18T01:36:21.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T01:36:21.924 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:21.924 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:21.924 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:21.933 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:21.947 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:21.947 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 -- 172.21.15.7:0/3232081224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f800c06a4c0 msgr2=0x7f800c06a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 --2- 172.21.15.7:0/3232081224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f800c06a4c0 0x7f800c06a920 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0098f0 tx=0x7f7ffc009c00 comp rx=0 tx=0).stop 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 -- 172.21.15.7:0/3232081224 shutdown_connections 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 --2- 172.21.15.7:0/3232081224 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f800c104cd0 0x7f800c105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 --2- 172.21.15.7:0/3232081224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f800c1071c0 0x7f800c069f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 --2- 172.21.15.7:0/3232081224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f800c06a4c0 0x7f800c06a920 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 -- 172.21.15.7:0/3232081224 >> 172.21.15.7:0/3232081224 conn(0x7f800c078100 msgr2=0x7f800c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 -- 172.21.15.7:0/3232081224 shutdown_connections 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.980+0000 7f8012600700 1 -- 172.21.15.7:0/3232081224 wait complete. 2024-03-18T01:36:21.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.981+0000 7f8012600700 1 Processor -- start 2024-03-18T01:36:21.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.981+0000 7f8012600700 1 -- start start 2024-03-18T01:36:21.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f8012600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f800c06a4c0 0x7f800c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:21.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f8012600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f800c104cd0 0x7f800c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:21.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f8012600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f800c1071c0 0x7f800c1a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:21.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f8012600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f800c118bd0 con 0x7f800c104cd0 2024-03-18T01:36:21.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f8012600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f800c118a50 con 0x7f800c1071c0 2024-03-18T01:36:21.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f8012600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f800c118d50 con 0x7f800c06a4c0 2024-03-18T01:36:21.983 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f800b400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f800c06a4c0 0x7f800c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:21.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f800b400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f800c06a4c0 0x7f800c19f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36914/0 (socket says 172.21.15.7:36914) 2024-03-18T01:36:21.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.982+0000 7f800b400700 1 -- 172.21.15.7:0/91403036 learned_addr learned my addr 172.21.15.7:0/91403036 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:21.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800aa00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f800c104cd0 0x7f800c19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:21.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800b400700 1 -- 172.21.15.7:0/91403036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f800c1071c0 msgr2=0x7f800c1a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:21.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800be00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f800c1071c0 0x7f800c1a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:21.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800b400700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f800c1071c0 0x7f800c1a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:21.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800b400700 1 -- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f800c104cd0 msgr2=0x7f800c19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:21.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800b400700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f800c104cd0 0x7f800c19f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800b400700 1 -- 172.21.15.7:0/91403036 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800c1a40f0 con 0x7f800c06a4c0 2024-03-18T01:36:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800aa00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f800c104cd0 0x7f800c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:36:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.983+0000 7f800be00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f800c1071c0 0x7f800c1a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:36:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.984+0000 7f800b400700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f800c06a4c0 0x7f800c19f0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800ec10 tx=0x7f7ff800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.984+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800cde0 con 0x7f800c06a4c0 2024-03-18T01:36:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.984+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ff8004500 con 0x7f800c06a4c0 2024-03-18T01:36:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.984+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8010690 con 0x7f800c06a4c0 2024-03-18T01:36:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.985+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f800c1a43e0 con 0x7f800c06a4c0 2024-03-18T01:36:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.985+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f800c06bcf0 con 0x7f800c06a4c0 2024-03-18T01:36:21.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.986+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f800c104690 con 0x7f800c06a4c0 2024-03-18T01:36:21.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.988+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f7ff8003f70 con 0x7f800c06a4c0 2024-03-18T01:36:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.990+0000 7f8003e00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ff00a2a70 0x7f7ff00a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:21.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.990+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7ff80cf3b0 con 0x7f800c06a4c0 2024-03-18T01:36:21.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.990+0000 7f8003e00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] conn(0x7f7ff00adbd0 0x7f7ff00b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:21.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.990+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 --> [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ff8019e50 con 0x7f7ff00adbd0 2024-03-18T01:36:21.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.990+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7ff80cf800 con 0x7f800c06a4c0 2024-03-18T01:36:21.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.991+0000 7f800aa00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ff00a2a70 0x7f7ff00a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:21.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.991+0000 7f800be00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] conn(0x7f7ff00adbd0 0x7f7ff00b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:21.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.992+0000 7f800aa00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ff00a2a70 0x7f7ff00a4f30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc005bf0 tx=0x7f7ffc005b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:21.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.992+0000 7f800be00700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] conn(0x7f7ff00adbd0 0x7f7ff00b0070 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:21.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:21.993+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 <== osd.3 v2:172.21.15.7:6810/2660921725 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ff8019e50 con 0x7f7ff00adbd0 2024-03-18T01:36:22.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.025+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 --> [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f800c1017b0 con 0x7f7ff00adbd0 2024-03-18T01:36:22.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.026+0000 7f8003e00700 1 -- 172.21.15.7:0/91403036 <== osd.3 v2:172.21.15.7:6810/2660921725 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f800c1017b0 con 0x7f7ff00adbd0 2024-03-18T01:36:22.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.026+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] conn(0x7f7ff00adbd0 msgr2=0x7f7ff00b0070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.026+0000 7f8012600700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] conn(0x7f7ff00adbd0 0x7f7ff00b0070 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.027+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ff00a2a70 msgr2=0x7f7ff00a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.027+0000 7f8012600700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ff00a2a70 0x7f7ff00a4f30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc005bf0 tx=0x7f7ffc005b40 comp rx=0 tx=0).stop 2024-03-18T01:36:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.027+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f800c06a4c0 msgr2=0x7f800c19f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:22.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.027+0000 7f8012600700 1 --2- 172.21.15.7:0/91403036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f800c06a4c0 0x7f800c19f0e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800ec10 tx=0x7f7ff800c2d0 comp rx=0 tx=0).stop 2024-03-18T01:36:22.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.031+0000 7f800be00700 1 -- 172.21.15.7:0/91403036 reap_dead start 2024-03-18T01:36:22.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.031+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 shutdown_connections 2024-03-18T01:36:22.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.031+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 >> 172.21.15.7:0/91403036 conn(0x7f800c078100 msgr2=0x7f800c103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:22.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.031+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 shutdown_connections 2024-03-18T01:36:22.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:22.031+0000 7f8012600700 1 -- 172.21.15.7:0/91403036 wait complete. 2024-03-18T01:36:22.614 INFO:teuthology.orchestra.run.smithi007.stdout:163208757313 2024-03-18T01:36:22.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T01:36:23.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:22 smithi203 ceph-mon[36345]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:23.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:22 smithi046 ceph-mon[36108]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:23.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:22 smithi007 ceph-mon[31996]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:23.881 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:23.906 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:23.909 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:23.913 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:25.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:24 smithi007 ceph-mon[31996]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:25.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:24 smithi203 ceph-mon[36345]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:25.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:24 smithi046 ceph-mon[36108]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:26 smithi203 ceph-mon[36345]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:27.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:26 smithi046 ceph-mon[36108]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:27.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:26 smithi007 ceph-mon[31996]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:29.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:28 smithi007 ceph-mon[31996]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:28 smithi203 ceph-mon[36345]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:29.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:28 smithi046 ceph-mon[36108]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:31.134 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:30 smithi007 ceph-mon[31996]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:31.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:30 smithi203 ceph-mon[36345]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:31.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:30 smithi046 ceph-mon[36108]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:33.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:32 smithi007 ceph-mon[31996]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:32 smithi203 ceph-mon[36345]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:33.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:32 smithi046 ceph-mon[36108]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:34.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:33 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/796286622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T01:36:34.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:33 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/796286622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T01:36:34.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:33 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/796286622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T01:36:34.768 INFO:teuthology.orchestra.run.smithi007.stdout:107374182475 2024-03-18T01:36:34.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T01:36:35.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:34 smithi007 ceph-mon[31996]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.073+0000 7f3017200700 1 -- 172.21.15.7:0/3441233893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 msgr2=0x7f3010102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.073+0000 7f3017200700 1 --2- 172.21.15.7:0/3441233893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 0x7f3010102960 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3000009990 tx=0x7f3000009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.073+0000 7f3017200700 1 -- 172.21.15.7:0/3441233893 shutdown_connections 2024-03-18T01:36:35.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.073+0000 7f3017200700 1 --2- 172.21.15.7:0/3441233893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107650 0x7f301010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.073+0000 7f3017200700 1 --2- 172.21.15.7:0/3441233893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 0x7f3010102960 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.073+0000 7f3017200700 1 --2- 172.21.15.7:0/3441233893 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3010102f30 0x7f3010106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.073+0000 7f3017200700 1 -- 172.21.15.7:0/3441233893 >> 172.21.15.7:0/3441233893 conn(0x7f3010079430 msgr2=0x7f3010079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.074+0000 7f3017200700 1 -- 172.21.15.7:0/3441233893 shutdown_connections 2024-03-18T01:36:35.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.074+0000 7f3017200700 1 -- 172.21.15.7:0/3441233893 wait complete. 2024-03-18T01:36:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.075+0000 7f3017200700 1 Processor -- start 2024-03-18T01:36:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.075+0000 7f3017200700 1 -- start start 2024-03-18T01:36:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f3017200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 0x7f301019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f3017200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3010102f30 0x7f301019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f3017200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107650 0x7f301019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f3017200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3010112680 con 0x7f3010102f30 2024-03-18T01:36:35.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f3017200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3010112500 con 0x7f3010102580 2024-03-18T01:36:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f3017200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3010112800 con 0x7f3010107650 2024-03-18T01:36:35.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f300fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 0x7f301019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f300fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 0x7f301019add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:48698/0 (socket says 172.21.15.7:48698) 2024-03-18T01:36:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.076+0000 7f300fe00700 1 -- 172.21.15.7:0/385465258 learned_addr learned my addr 172.21.15.7:0/385465258 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.077+0000 7f3015200700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107650 0x7f301019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.077+0000 7f300f400700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3010102f30 0x7f301019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.077+0000 7f300fe00700 1 -- 172.21.15.7:0/385465258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107650 msgr2=0x7f301019f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.077+0000 7f300fe00700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107650 0x7f301019f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.077+0000 7f300fe00700 1 -- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3010102f30 msgr2=0x7f301019b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:35.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.077+0000 7f300fe00700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3010102f30 0x7f301019b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.077+0000 7f300fe00700 1 -- 172.21.15.7:0/385465258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301019fd90 con 0x7f3010102580 2024-03-18T01:36:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.078+0000 7f300fe00700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 0x7f301019add0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3000000c00 tx=0x7f3000005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.078+0000 7f300cc00700 1 -- 172.21.15.7:0/385465258 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000027070 con 0x7f3010102580 2024-03-18T01:36:35.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.078+0000 7f300cc00700 1 -- 172.21.15.7:0/385465258 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30000056b0 con 0x7f3010102580 2024-03-18T01:36:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.078+0000 7f300cc00700 1 -- 172.21.15.7:0/385465258 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3000017650 con 0x7f3010102580 2024-03-18T01:36:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.079+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30101a0020 con 0x7f3010102580 2024-03-18T01:36:35.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.079+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30101a0480 con 0x7f3010102580 2024-03-18T01:36:35.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.081+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3010068f90 con 0x7f3010102580 2024-03-18T01:36:35.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.083+0000 7f300cc00700 1 -- 172.21.15.7:0/385465258 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f3000009d10 con 0x7f3010102580 2024-03-18T01:36:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.084+0000 7f300cc00700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2ff40a29a0 0x7f2ff40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.084+0000 7f300cc00700 1 -- 172.21.15.7:0/385465258 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f30000cfb60 con 0x7f3010102580 2024-03-18T01:36:35.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.085+0000 7f300f400700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2ff40a29a0 0x7f2ff40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:35.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.086+0000 7f300f400700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2ff40a29a0 0x7f2ff40a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f30100765f0 tx=0x7f2ffc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:35.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.088+0000 7f300cc00700 1 -- 172.21.15.7:0/385465258 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f300005d740 con 0x7f3010102580 2024-03-18T01:36:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:34 smithi203 ceph-mon[36345]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:35.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:34 smithi046 ceph-mon[36108]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.358+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f30101a7900 con 0x7f3010102580 2024-03-18T01:36:35.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.359+0000 7f300cc00700 1 -- 172.21.15.7:0/385465258 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f30101a0480 con 0x7f3010102580 2024-03-18T01:36:35.360 INFO:teuthology.orchestra.run.smithi007.stdout:77309411413 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2ff40a29a0 msgr2=0x7f2ff40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2ff40a29a0 0x7f2ff40a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f30100765f0 tx=0x7f2ffc009260 comp rx=0 tx=0).stop 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 msgr2=0x7f301019add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 0x7f301019add0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3000000c00 tx=0x7f3000005590 comp rx=0 tx=0).stop 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 shutdown_connections 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2ff40a29a0 0x7f2ff40a4e60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3010107650 0x7f301019f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3010102580 0x7f301019add0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 --2- 172.21.15.7:0/385465258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3010102f30 0x7f301019b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.362+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 >> 172.21.15.7:0/385465258 conn(0x7f3010079430 msgr2=0x7f3010105f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.363+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 shutdown_connections 2024-03-18T01:36:35.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:35.363+0000 7f3017200700 1 -- 172.21.15.7:0/385465258 wait complete. 2024-03-18T01:36:36.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:35 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/385465258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T01:36:36.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:35 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/385465258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T01:36:36.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:35 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/385465258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.588+0000 7ff996c00700 1 -- 172.21.15.7:0/604144874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff99010ed00 msgr2=0x7ff9901110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.588+0000 7ff996c00700 1 --2- 172.21.15.7:0/604144874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff99010ed00 0x7ff9901110f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff98800b0a0 tx=0x7ff98800b3b0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.588+0000 7ff996c00700 1 -- 172.21.15.7:0/604144874 shutdown_connections 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.588+0000 7ff996c00700 1 --2- 172.21.15.7:0/604144874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff99010ed00 0x7ff9901110f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.588+0000 7ff996c00700 1 --2- 172.21.15.7:0/604144874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff990074500 0x7ff9900748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.588+0000 7ff996c00700 1 --2- 172.21.15.7:0/604144874 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff990074e20 0x7ff99010e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.588+0000 7ff996c00700 1 -- 172.21.15.7:0/604144874 >> 172.21.15.7:0/604144874 conn(0x7ff9900793a0 msgr2=0x7ff9900797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.589+0000 7ff996c00700 1 -- 172.21.15.7:0/604144874 shutdown_connections 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.589+0000 7ff996c00700 1 -- 172.21.15.7:0/604144874 wait complete. 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.590+0000 7ff996c00700 1 Processor -- start 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.590+0000 7ff996c00700 1 -- start start 2024-03-18T01:36:36.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff996c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990074500 0x7ff9901a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff996c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff990074e20 0x7ff9901a0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff996c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff99010ed00 0x7ff99019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff996c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff990114810 con 0x7ff990074e20 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff996c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff990114690 con 0x7ff990074500 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff996c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff990114990 con 0x7ff99010ed00 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff994c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff99010ed00 0x7ff99019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff98fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990074500 0x7ff9901a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff994c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff99010ed00 0x7ff99019aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34962/0 (socket says 172.21.15.7:34962) 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.591+0000 7ff994c00700 1 -- 172.21.15.7:0/796286622 learned_addr learned my addr 172.21.15.7:0/796286622 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98f400700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff990074e20 0x7ff9901a0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98fe00700 1 -- 172.21.15.7:0/796286622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff99010ed00 msgr2=0x7ff99019aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98fe00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff99010ed00 0x7ff99019aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98fe00700 1 -- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff990074e20 msgr2=0x7ff9901a0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98fe00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff990074e20 0x7ff9901a0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98fe00700 1 -- 172.21.15.7:0/796286622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff99019b250 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98f400700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff990074e20 0x7ff9901a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98fe00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990074500 0x7ff9901a0930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff98000e9f0 tx=0x7ff98000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98cc00700 1 -- 172.21.15.7:0/796286622 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98000cbd0 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff99019b540 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98cc00700 1 -- 172.21.15.7:0/796286622 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff980004500 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff98cc00700 1 -- 172.21.15.7:0/796286622 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980005280 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.592+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9901a7800 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.594+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff990068f90 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.596+0000 7ff98cc00700 1 -- 172.21.15.7:0/796286622 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7ff980005420 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.597+0000 7ff98cc00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff9740a2920 0x7ff9740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.597+0000 7ff98cc00700 1 -- 172.21.15.7:0/796286622 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff9800cef20 con 0x7ff990074500 2024-03-18T01:36:36.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.597+0000 7ff98f400700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff9740a2920 0x7ff9740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:36.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.598+0000 7ff98f400700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff9740a2920 0x7ff9740a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9901108e0 tx=0x7ff97c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:36.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.599+0000 7ff98cc00700 1 -- 172.21.15.7:0/796286622 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff98009baa0 con 0x7ff990074500 2024-03-18T01:36:36.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.847+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff99019bfe0 con 0x7ff990074500 2024-03-18T01:36:36.579 INFO:teuthology.orchestra.run.smithi007.stdout:55834574936 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.848+0000 7ff98cc00700 1 -- 172.21.15.7:0/796286622 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff98009b3a0 con 0x7ff990074500 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.850+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff9740a2920 msgr2=0x7ff9740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.850+0000 7ff996c00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff9740a2920 0x7ff9740a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff9901108e0 tx=0x7ff97c009260 comp rx=0 tx=0).stop 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990074500 msgr2=0x7ff9901a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990074500 0x7ff9901a0930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff98000e9f0 tx=0x7ff98000ed00 comp rx=0 tx=0).stop 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 shutdown_connections 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff9740a2920 0x7ff9740a4de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff99010ed00 0x7ff99019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff990074500 0x7ff9901a0930 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 --2- 172.21.15.7:0/796286622 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff990074e20 0x7ff9901a0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 >> 172.21.15.7:0/796286622 conn(0x7ff9900793a0 msgr2=0x7ff9900ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:36.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 shutdown_connections 2024-03-18T01:36:36.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:33.851+0000 7ff996c00700 1 -- 172.21.15.7:0/796286622 wait complete. 2024-03-18T01:36:37.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:36 smithi046 ceph-mon[36108]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:37.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:36 smithi007 ceph-mon[31996]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:36 smithi203 ceph-mon[36345]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:38.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.130+0000 7f1511e00700 1 -- 172.21.15.7:0/442713576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102f00 msgr2=0x7f150c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:38.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.130+0000 7f1511e00700 1 --2- 172.21.15.7:0/442713576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102f00 0x7f150c106f50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f14f8009990 tx=0x7f14f8009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.130+0000 7f1511e00700 1 -- 172.21.15.7:0/442713576 shutdown_connections 2024-03-18T01:36:38.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.130+0000 7f1511e00700 1 --2- 172.21.15.7:0/442713576 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f150c107630 0x7f150c10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.130+0000 7f1511e00700 1 --2- 172.21.15.7:0/442713576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f150c102550 0x7f150c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.130+0000 7f1511e00700 1 --2- 172.21.15.7:0/442713576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102f00 0x7f150c106f50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.130+0000 7f1511e00700 1 -- 172.21.15.7:0/442713576 >> 172.21.15.7:0/442713576 conn(0x7f150c0fdda0 msgr2=0x7f150c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:38.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.131+0000 7f1511e00700 1 -- 172.21.15.7:0/442713576 shutdown_connections 2024-03-18T01:36:38.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.131+0000 7f1511e00700 1 -- 172.21.15.7:0/442713576 wait complete. 2024-03-18T01:36:38.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.132+0000 7f1511e00700 1 Processor -- start 2024-03-18T01:36:38.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.132+0000 7f1511e00700 1 -- start start 2024-03-18T01:36:38.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f1511e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102550 0x7f150c0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:38.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f1511e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f150c102f00 0x7f150c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:38.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f1511e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f150c107630 0x7f150c075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:38.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f1511e00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f150c112640 con 0x7f150c107630 2024-03-18T01:36:38.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f1511e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f150c1124c0 con 0x7f150c102550 2024-03-18T01:36:38.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f1511e00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f150c1127c0 con 0x7f150c102f00 2024-03-18T01:36:38.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f150aa00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102550 0x7f150c0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f150a000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f150c102f00 0x7f150c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f150aa00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102550 0x7f150c0745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:37876/0 (socket says 172.21.15.7:37876) 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f150aa00700 1 -- 172.21.15.7:0/978326387 learned_addr learned my addr 172.21.15.7:0/978326387 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f150a000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f150c102f00 0x7f150c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:49122/0 (socket says 172.21.15.7:49122) 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.133+0000 7f150b400700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f150c107630 0x7f150c075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f150a000700 1 -- 172.21.15.7:0/978326387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102550 msgr2=0x7f150c0745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f150a000700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102550 0x7f150c0745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f150a000700 1 -- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f150c107630 msgr2=0x7f150c075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f150a000700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f150c107630 0x7f150c075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f150a000700 1 -- 172.21.15.7:0/978326387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c1a7020 con 0x7f150c102f00 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f150a000700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f150c102f00 0x7f150c074b10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f14f80098e0 tx=0x7f14f800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:38.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f1510a00700 1 -- 172.21.15.7:0/978326387 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f8027070 con 0x7f150c102f00 2024-03-18T01:36:38.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f1510a00700 1 -- 172.21.15.7:0/978326387 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14f800fd10 con 0x7f150c102f00 2024-03-18T01:36:38.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f1510a00700 1 -- 172.21.15.7:0/978326387 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f80173f0 con 0x7f150c102f00 2024-03-18T01:36:38.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.134+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150c1a72b0 con 0x7f150c102f00 2024-03-18T01:36:38.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.135+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150c1a7760 con 0x7f150c102f00 2024-03-18T01:36:38.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.136+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150c068f90 con 0x7f150c102f00 2024-03-18T01:36:38.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.137+0000 7f1510a00700 1 -- 172.21.15.7:0/978326387 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f14f802a070 con 0x7f150c102f00 2024-03-18T01:36:38.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.139+0000 7f1510a00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14f00a29a0 0x7f14f00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:38.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.139+0000 7f1510a00700 1 -- 172.21.15.7:0/978326387 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14f80cfb50 con 0x7f150c102f00 2024-03-18T01:36:38.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.139+0000 7f150aa00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14f00a29a0 0x7f14f00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:38.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.140+0000 7f150aa00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14f00a29a0 0x7f14f00a4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009d20 tx=0x7f14fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:38.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.143+0000 7f1510a00700 1 -- 172.21.15.7:0/978326387 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f14f8013070 con 0x7f150c102f00 2024-03-18T01:36:38.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.378+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f150c02db70 con 0x7f150c102f00 2024-03-18T01:36:38.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.379+0000 7f1510a00700 1 -- 172.21.15.7:0/978326387 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f14f809c760 con 0x7f150c102f00 2024-03-18T01:36:38.380 INFO:teuthology.orchestra.run.smithi007.stdout:343597383714 2024-03-18T01:36:38.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14f00a29a0 msgr2=0x7f14f00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:38.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14f00a29a0 0x7f14f00a4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009d20 tx=0x7f14fc009260 comp rx=0 tx=0).stop 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f150c102f00 msgr2=0x7f150c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f150c102f00 0x7f150c074b10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f14f80098e0 tx=0x7f14f800b840 comp rx=0 tx=0).stop 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 shutdown_connections 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14f00a29a0 0x7f14f00a4e60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f150c107630 0x7f150c075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f150c102550 0x7f150c0745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 --2- 172.21.15.7:0/978326387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f150c102f00 0x7f150c074b10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 >> 172.21.15.7:0/978326387 conn(0x7f150c0fdda0 msgr2=0x7f150c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:38.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.383+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 shutdown_connections 2024-03-18T01:36:38.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:38.384+0000 7f1511e00700 1 -- 172.21.15.7:0/978326387 wait complete. 2024-03-18T01:36:38.858 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574931 got 55834574936 for osd.1 2024-03-18T01:36:38.858 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:39.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:38 smithi007 ceph-mon[31996]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:38 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/978326387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T01:36:39.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:38 smithi046 ceph-mon[36108]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:38 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/978326387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T01:36:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:38 smithi203 ceph-mon[36345]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:38 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/978326387' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T01:36:39.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.622+0000 7fed41600700 1 -- 172.21.15.7:0/135606953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed3c102550 msgr2=0x7fed3c102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:39.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.622+0000 7fed41600700 1 --2- 172.21.15.7:0/135606953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed3c102550 0x7fed3c102930 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed240099c0 tx=0x7fed24009cd0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.622+0000 7fed41600700 1 -- 172.21.15.7:0/135606953 shutdown_connections 2024-03-18T01:36:39.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.622+0000 7fed41600700 1 --2- 172.21.15.7:0/135606953 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed3c107630 0x7fed3c10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.622+0000 7fed41600700 1 --2- 172.21.15.7:0/135606953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed3c102550 0x7fed3c102930 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.622+0000 7fed41600700 1 --2- 172.21.15.7:0/135606953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c102f00 0x7fed3c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.622+0000 7fed41600700 1 -- 172.21.15.7:0/135606953 >> 172.21.15.7:0/135606953 conn(0x7fed3c0fdda0 msgr2=0x7fed3c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.622+0000 7fed41600700 1 -- 172.21.15.7:0/135606953 shutdown_connections 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.623+0000 7fed41600700 1 -- 172.21.15.7:0/135606953 wait complete. 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.624+0000 7fed41600700 1 Processor -- start 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.624+0000 7fed41600700 1 -- start start 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed41600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed3c102550 0x7fed3c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed41600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c102f00 0x7fed3c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed41600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed3c107630 0x7fed3c198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed41600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c112680 con 0x7fed3c102550 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed41600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c112500 con 0x7fed3c102f00 2024-03-18T01:36:39.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed41600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c112800 con 0x7fed3c107630 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed3a000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c102f00 0x7fed3c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed3aa00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed3c102550 0x7fed3c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed3a000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c102f00 0x7fed3c19ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:37896/0 (socket says 172.21.15.7:37896) 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed3a000700 1 -- 172.21.15.7:0/3940583578 learned_addr learned my addr 172.21.15.7:0/3940583578 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.625+0000 7fed3b400700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed3c107630 0x7fed3c198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.626+0000 7fed3a000700 1 -- 172.21.15.7:0/3940583578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed3c107630 msgr2=0x7fed3c198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.626+0000 7fed3a000700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed3c107630 0x7fed3c198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.626+0000 7fed3a000700 1 -- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed3c102550 msgr2=0x7fed3c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.626+0000 7fed3a000700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed3c102550 0x7fed3c19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.626+0000 7fed3a000700 1 -- 172.21.15.7:0/3940583578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c1990a0 con 0x7fed3c102f00 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.626+0000 7fed3a000700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c102f00 0x7fed3c19ec70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00ef90 tx=0x7fed2c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.626+0000 7fed33400700 1 -- 172.21.15.7:0/3940583578 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2c0092e0 con 0x7fed3c102f00 2024-03-18T01:36:39.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.626+0000 7fed33400700 1 -- 172.21.15.7:0/3940583578 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed2c004bb0 con 0x7fed3c102f00 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.627+0000 7fed33400700 1 -- 172.21.15.7:0/3940583578 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed2c004280 con 0x7fed3c102f00 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.627+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3c199330 con 0x7fed3c102f00 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.627+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3c074500 con 0x7fed3c102f00 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.628+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3c103a30 con 0x7fed3c102f00 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.630+0000 7fed33400700 1 -- 172.21.15.7:0/3940583578 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fed2c01b030 con 0x7fed3c102f00 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.633+0000 7fed33400700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fed1c0a29a0 0x7fed1c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.633+0000 7fed33400700 1 -- 172.21.15.7:0/3940583578 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fed2c0cf4e0 con 0x7fed3c102f00 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.633+0000 7fed3aa00700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fed1c0a29a0 0x7fed1c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.633+0000 7fed33400700 1 -- 172.21.15.7:0/3940583578 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fed2c022020 con 0x7fed3c102f00 2024-03-18T01:36:39.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.634+0000 7fed3aa00700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fed1c0a29a0 0x7fed1c0a4e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fed24000c00 tx=0x7fed24005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:39.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.895+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fed3c02db70 con 0x7fed3c102f00 2024-03-18T01:36:39.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.895+0000 7fed33400700 1 -- 172.21.15.7:0/3940583578 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fed2c010040 con 0x7fed3c102f00 2024-03-18T01:36:39.897 INFO:teuthology.orchestra.run.smithi007.stdout:309237645350 2024-03-18T01:36:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.898+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fed1c0a29a0 msgr2=0x7fed1c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.898+0000 7fed41600700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fed1c0a29a0 0x7fed1c0a4e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fed24000c00 tx=0x7fed24005040 comp rx=0 tx=0).stop 2024-03-18T01:36:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.898+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c102f00 msgr2=0x7fed3c19ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.898+0000 7fed41600700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c102f00 0x7fed3c19ec70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fed2c00ef90 tx=0x7fed2c00c3a0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.899+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 shutdown_connections 2024-03-18T01:36:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.899+0000 7fed41600700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fed1c0a29a0 0x7fed1c0a4e60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.899+0000 7fed41600700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed3c107630 0x7fed3c198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.899+0000 7fed41600700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fed3c102550 0x7fed3c19e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.899+0000 7fed41600700 1 --2- 172.21.15.7:0/3940583578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed3c102f00 0x7fed3c19ec70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:39.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.899+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 >> 172.21.15.7:0/3940583578 conn(0x7fed3c0fdda0 msgr2=0x7fed3c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:39.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.899+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 shutdown_connections 2024-03-18T01:36:39.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:39.899+0000 7fed41600700 1 -- 172.21.15.7:0/3940583578 wait complete. 2024-03-18T01:36:40.009 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:39 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3940583578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T01:36:40.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:39 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3940583578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T01:36:40.291 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:39 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3940583578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T01:36:41.137 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:40 smithi007 ceph-mon[31996]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:41.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:41.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:40 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:41.140 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383708 got 343597383714 for osd.11 2024-03-18T01:36:41.140 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:41.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:40 smithi046 ceph-mon[36108]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:40 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:41.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:40 smithi203 ceph-mon[36345]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:41.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:41.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:40 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:43.010 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:42 smithi007 ceph-mon[31996]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:43.011 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:42 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/674015697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T01:36:43.011 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:42 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/964317983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T01:36:43.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:42 smithi046 ceph-mon[36108]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:43.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:42 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/674015697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T01:36:43.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:42 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/964317983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T01:36:43.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:42 smithi203 ceph-mon[36345]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:43.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:42 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/674015697' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T01:36:43.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:42 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/964317983' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T01:36:43.559 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645344 got 309237645350 for osd.10 2024-03-18T01:36:43.560 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:43.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.469+0000 7f1dbae00700 1 -- 172.21.15.7:0/3512260522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1db4069f80 msgr2=0x7f1db406a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:43.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.469+0000 7f1dbae00700 1 --2- 172.21.15.7:0/3512260522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1db4069f80 0x7f1db406a3e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1da00099c0 tx=0x7f1da0009cd0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.469+0000 7f1dbae00700 1 -- 172.21.15.7:0/3512260522 shutdown_connections 2024-03-18T01:36:43.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.469+0000 7f1dbae00700 1 --2- 172.21.15.7:0/3512260522 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1db406a920 0x7f1db410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.469+0000 7f1dbae00700 1 --2- 172.21.15.7:0/3512260522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db4108320 0x7f1db4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.469+0000 7f1dbae00700 1 --2- 172.21.15.7:0/3512260522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1db4069f80 0x7f1db406a3e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.469+0000 7f1dbae00700 1 -- 172.21.15.7:0/3512260522 >> 172.21.15.7:0/3512260522 conn(0x7f1db4078100 msgr2=0x7f1db4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:43.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.469+0000 7f1dbae00700 1 -- 172.21.15.7:0/3512260522 shutdown_connections 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.470+0000 7f1dbae00700 1 -- 172.21.15.7:0/3512260522 wait complete. 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.471+0000 7f1dbae00700 1 Processor -- start 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.471+0000 7f1dbae00700 1 -- start start 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.471+0000 7f1dbae00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1db4069f80 0x7f1db41012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.471+0000 7f1dbae00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db406a920 0x7f1db41017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.471+0000 7f1dbae00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1db4108320 0x7f1db4101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.471+0000 7f1dbae00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1db41125d0 con 0x7f1db4108320 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.471+0000 7f1dbae00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1db4112450 con 0x7f1db406a920 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.471+0000 7f1dbae00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1db4112750 con 0x7f1db4069f80 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db406a920 0x7f1db41017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1db4069f80 0x7f1db41012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:43.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db406a920 0x7f1db41017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:37922/0 (socket says 172.21.15.7:37922) 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3400700 1 -- 172.21.15.7:0/674015697 learned_addr learned my addr 172.21.15.7:0/674015697 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db8e00700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1db4108320 0x7f1db4101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3400700 1 -- 172.21.15.7:0/674015697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1db4069f80 msgr2=0x7f1db41012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3400700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1db4069f80 0x7f1db41012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3400700 1 -- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1db4108320 msgr2=0x7f1db4101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3400700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1db4108320 0x7f1db4101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.472+0000 7f1db3400700 1 -- 172.21.15.7:0/674015697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db41ab470 con 0x7f1db406a920 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.473+0000 7f1db3400700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db406a920 0x7f1db41017e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1da0006010 tx=0x7f1da0003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.473+0000 7f1db0c00700 1 -- 172.21.15.7:0/674015697 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da00050c0 con 0x7f1db406a920 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.473+0000 7f1db0c00700 1 -- 172.21.15.7:0/674015697 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1da000f460 con 0x7f1db406a920 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.473+0000 7f1db0c00700 1 -- 172.21.15.7:0/674015697 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0017650 con 0x7f1db406a920 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.473+0000 7f1dbae00700 1 -- 172.21.15.7:0/674015697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db41ab6a0 con 0x7f1db406a920 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.473+0000 7f1dbae00700 1 -- 172.21.15.7:0/674015697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db41abb00 con 0x7f1db406a920 2024-03-18T01:36:43.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.476+0000 7f1daa000700 1 -- 172.21.15.7:0/674015697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db41039d0 con 0x7f1db406a920 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.477+0000 7f1db0c00700 1 -- 172.21.15.7:0/674015697 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f1da000f610 con 0x7f1db406a920 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.480+0000 7f1db0c00700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d980a29a0 0x7f1d980a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.480+0000 7f1db0c00700 1 -- 172.21.15.7:0/674015697 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1da00cfce0 con 0x7f1db406a920 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.480+0000 7f1db3e00700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d980a29a0 0x7f1d980a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.481+0000 7f1db3e00700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d980a29a0 0x7f1d980a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009be0 tx=0x7f1da4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.483+0000 7f1db0c00700 1 -- 172.21.15.7:0/674015697 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1da0013070 con 0x7f1db406a920 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.757+0000 7f1daa000700 1 -- 172.21.15.7:0/674015697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1db41027d0 con 0x7f1db406a920 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.758+0000 7f1db0c00700 1 -- 172.21.15.7:0/674015697 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1da0020370 con 0x7f1db406a920 2024-03-18T01:36:43.612 INFO:teuthology.orchestra.run.smithi007.stdout:219043332153 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 -- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d980a29a0 msgr2=0x7f1d980a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d980a29a0 0x7f1d980a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009be0 tx=0x7f1da4009260 comp rx=0 tx=0).stop 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 -- 172.21.15.7:0/674015697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db406a920 msgr2=0x7f1db41017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db406a920 0x7f1db41017e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1da0006010 tx=0x7f1da0003a90 comp rx=0 tx=0).stop 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 -- 172.21.15.7:0/674015697 shutdown_connections 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1d980a29a0 0x7f1d980a4e60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1db4108320 0x7f1db4101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1db4069f80 0x7f1db41012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 --2- 172.21.15.7:0/674015697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db406a920 0x7f1db41017e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 -- 172.21.15.7:0/674015697 >> 172.21.15.7:0/674015697 conn(0x7f1db4078100 msgr2=0x7f1db40ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 -- 172.21.15.7:0/674015697 shutdown_connections 2024-03-18T01:36:43.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.761+0000 7f1daa000700 1 -- 172.21.15.7:0/674015697 wait complete. 2024-03-18T01:36:44.190 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:43 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/753181831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T01:36:44.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:43 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/753181831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T01:36:44.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:43 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/753181831' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T01:36:44.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.374+0000 7f3a95a00700 1 -- 172.21.15.7:0/1610266292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a90108320 msgr2=0x7f3a90108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.374+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1610266292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a90108320 0x7f3a90108700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009990 tx=0x7f3a80009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.375+0000 7f3a95a00700 1 -- 172.21.15.7:0/1610266292 shutdown_connections 2024-03-18T01:36:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.375+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1610266292 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a9006a920 0x7f3a9010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.375+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1610266292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a90108320 0x7f3a90108700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.375+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1610266292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a90069f80 0x7f3a9006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.375+0000 7f3a95a00700 1 -- 172.21.15.7:0/1610266292 >> 172.21.15.7:0/1610266292 conn(0x7f3a90078100 msgr2=0x7f3a90078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.375+0000 7f3a95a00700 1 -- 172.21.15.7:0/1610266292 shutdown_connections 2024-03-18T01:36:44.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.375+0000 7f3a95a00700 1 -- 172.21.15.7:0/1610266292 wait complete. 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.376+0000 7f3a95a00700 1 Processor -- start 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.377+0000 7f3a95a00700 1 -- start start 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.377+0000 7f3a95a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a90069f80 0x7f3a901012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.377+0000 7f3a95a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a9006a920 0x7f3a901017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.377+0000 7f3a95a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a90108320 0x7f3a90101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.377+0000 7f3a95a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3a901125b0 con 0x7f3a90069f80 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.377+0000 7f3a95a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90112430 con 0x7f3a90108320 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.377+0000 7f3a95a00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90112730 con 0x7f3a9006a920 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a9006a920 0x7f3a901017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8ea00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a90069f80 0x7f3a901012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a9006a920 0x7f3a901017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:49192/0 (socket says 172.21.15.7:49192) 2024-03-18T01:36:44.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 -- 172.21.15.7:0/1210305180 learned_addr learned my addr 172.21.15.7:0/1210305180 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8f400700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a90108320 0x7f3a90101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 -- 172.21.15.7:0/1210305180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a90108320 msgr2=0x7f3a90101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a90108320 0x7f3a90101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 -- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a90069f80 msgr2=0x7f3a901012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a90069f80 0x7f3a901012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 -- 172.21.15.7:0/1210305180 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a901a6f50 con 0x7f3a9006a920 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.378+0000 7f3a8e000700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a9006a920 0x7f3a901017e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00eb30 tx=0x7f3a7c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.379+0000 7f3a87400700 1 -- 172.21.15.7:0/1210305180 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c00cbd0 con 0x7f3a9006a920 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.379+0000 7f3a87400700 1 -- 172.21.15.7:0/1210305180 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a7c004500 con 0x7f3a9006a920 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.379+0000 7f3a87400700 1 -- 172.21.15.7:0/1210305180 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c010430 con 0x7f3a9006a920 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.379+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a901a7240 con 0x7f3a9006a920 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.379+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a901a7730 con 0x7f3a9006a920 2024-03-18T01:36:44.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.381+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a70005330 con 0x7f3a9006a920 2024-03-18T01:36:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.382+0000 7f3a87400700 1 -- 172.21.15.7:0/1210305180 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f3a7c004020 con 0x7f3a9006a920 2024-03-18T01:36:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.384+0000 7f3a87400700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3a740a29a0 0x7f3a740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.384+0000 7f3a87400700 1 -- 172.21.15.7:0/1210305180 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3a7c0cf160 con 0x7f3a9006a920 2024-03-18T01:36:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.385+0000 7f3a8ea00700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3a740a29a0 0x7f3a740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.386+0000 7f3a8ea00700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3a740a29a0 0x7f3a740a4e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009750 tx=0x7f3a8000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:44.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.387+0000 7f3a87400700 1 -- 172.21.15.7:0/1210305180 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3a7c09bce0 con 0x7f3a9006a920 2024-03-18T01:36:44.633 INFO:teuthology.orchestra.run.smithi007.stdout:34359738465 2024-03-18T01:36:44.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 -- 172.21.15.7:0/3015951096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf64102f30 msgr2=0x7fbf64106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 --2- 172.21.15.7:0/3015951096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf64102f30 0x7fbf64106f70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbf500099c0 tx=0x7fbf50009cd0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 -- 172.21.15.7:0/3015951096 shutdown_connections 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 --2- 172.21.15.7:0/3015951096 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf64107650 0x7fbf6410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 --2- 172.21.15.7:0/3015951096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64102580 0x7fbf64102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 --2- 172.21.15.7:0/3015951096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf64102f30 0x7fbf64106f70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 -- 172.21.15.7:0/3015951096 >> 172.21.15.7:0/3015951096 conn(0x7fbf64079430 msgr2=0x7fbf64079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 -- 172.21.15.7:0/3015951096 shutdown_connections 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.274+0000 7fbf69000700 1 -- 172.21.15.7:0/3015951096 wait complete. 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.275+0000 7fbf69000700 1 Processor -- start 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.276+0000 7fbf69000700 1 -- start start 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.276+0000 7fbf69000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64102580 0x7fbf641a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.276+0000 7fbf69000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf64102f30 0x7fbf641a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:44.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.276+0000 7fbf69000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf64107650 0x7fbf6419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.276+0000 7fbf69000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64112610 con 0x7fbf64107650 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.276+0000 7fbf69000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64112490 con 0x7fbf64102580 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.276+0000 7fbf69000700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbf64112790 con 0x7fbf64102f30 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.276+0000 7fbf61e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64102580 0x7fbf641a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64102580 0x7fbf641a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:37966/0 (socket says 172.21.15.7:37966) 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61e00700 1 -- 172.21.15.7:0/753181831 learned_addr learned my addr 172.21.15.7:0/753181831 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61400700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf64102f30 0x7fbf641a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf62800700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf64107650 0x7fbf6419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61e00700 1 -- 172.21.15.7:0/753181831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf64102f30 msgr2=0x7fbf641a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61e00700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf64102f30 0x7fbf641a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61e00700 1 -- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf64107650 msgr2=0x7fbf6419ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61e00700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf64107650 0x7fbf6419ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61e00700 1 -- 172.21.15.7:0/753181831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf6419b3f0 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf62800700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf64107650 0x7fbf6419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.277+0000 7fbf61e00700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64102580 0x7fbf641a0a40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400eb30 tx=0x7fbf5400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.278+0000 7fbf5aa00700 1 -- 172.21.15.7:0/753181831 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf5400cc90 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.278+0000 7fbf5aa00700 1 -- 172.21.15.7:0/753181831 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf54004500 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.278+0000 7fbf5aa00700 1 -- 172.21.15.7:0/753181831 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf54010430 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.278+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf6419b6e0 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.278+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf641a79c0 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.279+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf64103a50 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.282+0000 7fbf5aa00700 1 -- 172.21.15.7:0/753181831 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fbf54003f70 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.283+0000 7fbf5aa00700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf480a29a0 0x7fbf480a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.283+0000 7fbf5aa00700 1 -- 172.21.15.7:0/753181831 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbf540cf260 con 0x7fbf64102580 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.283+0000 7fbf61400700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf480a29a0 0x7fbf480a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.284+0000 7fbf61400700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf480a29a0 0x7fbf480a4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf50000c00 tx=0x7fbf50005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:44.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.287+0000 7fbf5aa00700 1 -- 172.21.15.7:0/753181831 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbf5409bde0 con 0x7fbf64102580 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.524+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fbf641a7b90 con 0x7fbf64102580 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.524+0000 7fbf5aa00700 1 -- 172.21.15.7:0/753181831 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbf5409b6e0 con 0x7fbf64102580 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.528+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf480a29a0 msgr2=0x7fbf480a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.528+0000 7fbf69000700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf480a29a0 0x7fbf480a4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf50000c00 tx=0x7fbf50005fb0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.528+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64102580 msgr2=0x7fbf641a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.528+0000 7fbf69000700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64102580 0x7fbf641a0a40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fbf5400eb30 tx=0x7fbf5400ee40 comp rx=0 tx=0).stop 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.529+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 shutdown_connections 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.529+0000 7fbf69000700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbf480a29a0 0x7fbf480a4e60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.529+0000 7fbf69000700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbf64107650 0x7fbf6419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.529+0000 7fbf69000700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64102580 0x7fbf641a0a40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.529+0000 7fbf69000700 1 --2- 172.21.15.7:0/753181831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf64102f30 0x7fbf641a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.529+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 >> 172.21.15.7:0/753181831 conn(0x7fbf64079430 msgr2=0x7fbf6410d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.529+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 shutdown_connections 2024-03-18T01:36:44.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:43.529+0000 7fbf69000700 1 -- 172.21.15.7:0/753181831 wait complete. 2024-03-18T01:36:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.637+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3a700058f0 con 0x7f3a9006a920 2024-03-18T01:36:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.637+0000 7f3a87400700 1 -- 172.21.15.7:0/1210305180 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3a7c09b5e0 con 0x7f3a9006a920 2024-03-18T01:36:44.639 INFO:teuthology.orchestra.run.smithi007.stdout:141733920842 2024-03-18T01:36:44.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3a740a29a0 msgr2=0x7f3a740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3a740a29a0 0x7f3a740a4e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009750 tx=0x7f3a8000b260 comp rx=0 tx=0).stop 2024-03-18T01:36:44.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a9006a920 msgr2=0x7f3a901017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:44.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a9006a920 0x7f3a901017e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00eb30 tx=0x7f3a7c00ee40 comp rx=0 tx=0).stop 2024-03-18T01:36:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 shutdown_connections 2024-03-18T01:36:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3a740a29a0 0x7f3a740a4e60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a90108320 0x7f3a90101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3a90069f80 0x7f3a901012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 --2- 172.21.15.7:0/1210305180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a9006a920 0x7f3a901017e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 >> 172.21.15.7:0/1210305180 conn(0x7f3a90078100 msgr2=0x7f3a900ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.642+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 shutdown_connections 2024-03-18T01:36:44.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.643+0000 7f3a95a00700 1 -- 172.21.15.7:0/1210305180 wait complete. 2024-03-18T01:36:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:44 smithi007 ceph-mon[31996]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:44 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1210305180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T01:36:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:36:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:36:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:44 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:44 smithi046 ceph-mon[36108]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:44 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1210305180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T01:36:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:36:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:36:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:44 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.712+0000 7f51fac00700 1 -- 172.21.15.7:0/3237236959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f406a920 msgr2=0x7f51f410f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.712+0000 7f51fac00700 1 --2- 172.21.15.7:0/3237236959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f406a920 0x7f51f410f160 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00b0a0 tx=0x7f51ec00b3b0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.712+0000 7f51fac00700 1 -- 172.21.15.7:0/3237236959 shutdown_connections 2024-03-18T01:36:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.712+0000 7f51fac00700 1 --2- 172.21.15.7:0/3237236959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f406a920 0x7f51f410f160 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.712+0000 7f51fac00700 1 --2- 172.21.15.7:0/3237236959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f4108320 0x7f51f4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.712+0000 7f51fac00700 1 --2- 172.21.15.7:0/3237236959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f4069f80 0x7f51f406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.712+0000 7f51fac00700 1 -- 172.21.15.7:0/3237236959 >> 172.21.15.7:0/3237236959 conn(0x7f51f4078100 msgr2=0x7f51f4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.714+0000 7f51fac00700 1 -- 172.21.15.7:0/3237236959 shutdown_connections 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.714+0000 7f51fac00700 1 -- 172.21.15.7:0/3237236959 wait complete. 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.715+0000 7f51fac00700 1 Processor -- start 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.716+0000 7f51fac00700 1 -- start start 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.716+0000 7f51fac00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4069f80 0x7f51f41a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.716+0000 7f51fac00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f406a920 0x7f51f41a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.716+0000 7f51fac00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f4108320 0x7f51f419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.716+0000 7f51fac00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4114880 con 0x7f51f406a920 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.716+0000 7f51fac00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4114700 con 0x7f51f4069f80 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.716+0000 7f51fac00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4114a00 con 0x7f51f4108320 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.717+0000 7f51f8c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f4108320 0x7f51f419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:45.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.717+0000 7f51f8c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f4108320 0x7f51f419aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:49198/0 (socket says 172.21.15.7:49198) 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.717+0000 7f51f8c00700 1 -- 172.21.15.7:0/2443036064 learned_addr learned my addr 172.21.15.7:0/2443036064 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.717+0000 7f51f3e00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4069f80 0x7f51f41a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.717+0000 7f51f3400700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f406a920 0x7f51f41a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.717+0000 7f51f3e00700 1 -- 172.21.15.7:0/2443036064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f4108320 msgr2=0x7f51f419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.718+0000 7f51f3e00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f4108320 0x7f51f419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.718+0000 7f51f3e00700 1 -- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f406a920 msgr2=0x7f51f41a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.718+0000 7f51f3e00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f406a920 0x7f51f41a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.718+0000 7f51f3e00700 1 -- 172.21.15.7:0/2443036064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f419b200 con 0x7f51f4069f80 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.718+0000 7f51f8c00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f4108320 0x7f51f419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.718+0000 7f51f3400700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f406a920 0x7f51f41a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.718+0000 7f51f3e00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4069f80 0x7f51f41a08e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f51e400c970 tx=0x7f51e400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:45.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.719+0000 7f51f0c00700 1 -- 172.21.15.7:0/2443036064 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e400ce60 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.719+0000 7f51f0c00700 1 -- 172.21.15.7:0/2443036064 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e400f450 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.719+0000 7f51f0c00700 1 -- 172.21.15.7:0/2443036064 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4018660 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.719+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f419b4f0 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.719+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f41a77b0 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.721+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f4068f90 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.723+0000 7f51f0c00700 1 -- 172.21.15.7:0/2443036064 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f51e400f600 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.725+0000 7f51f0c00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f51d80a29a0 0x7f51d80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.726+0000 7f51f0c00700 1 -- 172.21.15.7:0/2443036064 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f51e40ceda0 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.726+0000 7f51f3400700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f51d80a29a0 0x7f51d80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.726+0000 7f51f0c00700 1 -- 172.21.15.7:0/2443036064 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51e4014030 con 0x7f51f4069f80 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.727+0000 7f51f3400700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f51d80a29a0 0x7f51d80a4e60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f51e0005d90 tx=0x7f51e0005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:45.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.985+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f51f419bfd0 con 0x7f51f4069f80 2024-03-18T01:36:45.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.986+0000 7f51f0c00700 1 -- 172.21.15.7:0/2443036064 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f51e409b9a0 con 0x7f51f4069f80 2024-03-18T01:36:45.292 INFO:teuthology.orchestra.run.smithi007.stdout:287762808877 2024-03-18T01:36:45.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f51d80a29a0 msgr2=0x7f51d80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f51d80a29a0 0x7f51d80a4e60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f51e0005d90 tx=0x7f51e0005d00 comp rx=0 tx=0).stop 2024-03-18T01:36:45.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4069f80 msgr2=0x7f51f41a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4069f80 0x7f51f41a08e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f51e400c970 tx=0x7f51e400cc80 comp rx=0 tx=0).stop 2024-03-18T01:36:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 shutdown_connections 2024-03-18T01:36:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f51d80a29a0 0x7f51d80a4e60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f4108320 0x7f51f419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51f4069f80 0x7f51f41a08e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 --2- 172.21.15.7:0/2443036064 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51f406a920 0x7f51f41a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 >> 172.21.15.7:0/2443036064 conn(0x7f51f4078100 msgr2=0x7f51f40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 shutdown_connections 2024-03-18T01:36:45.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:44.989+0000 7f51fac00700 1 -- 172.21.15.7:0/2443036064 wait complete. 2024-03-18T01:36:45.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:44 smithi203 ceph-mon[36345]: pgmap v374: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:45.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:44 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1210305180' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T01:36:45.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:36:45.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:36:45.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:44 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:36:45.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.558+0000 7f4fe0000700 1 -- 172.21.15.7:0/1066245377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8069f80 msgr2=0x7f4fd806a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.558+0000 7f4fe0000700 1 --2- 172.21.15.7:0/1066245377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8069f80 0x7f4fd806a3e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009990 tx=0x7f4fc4009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.558+0000 7f4fe0000700 1 -- 172.21.15.7:0/1066245377 shutdown_connections 2024-03-18T01:36:45.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.558+0000 7f4fe0000700 1 --2- 172.21.15.7:0/1066245377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fd806a920 0x7f4fd810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.558+0000 7f4fe0000700 1 --2- 172.21.15.7:0/1066245377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd8108320 0x7f4fd8108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.558+0000 7f4fe0000700 1 --2- 172.21.15.7:0/1066245377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8069f80 0x7f4fd806a3e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.558+0000 7f4fe0000700 1 -- 172.21.15.7:0/1066245377 >> 172.21.15.7:0/1066245377 conn(0x7f4fd8078100 msgr2=0x7f4fd8078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.559+0000 7f4fe0000700 1 -- 172.21.15.7:0/1066245377 shutdown_connections 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.559+0000 7f4fe0000700 1 -- 172.21.15.7:0/1066245377 wait complete. 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.560+0000 7f4fe0000700 1 Processor -- start 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.560+0000 7f4fe0000700 1 -- start start 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.560+0000 7f4fe0000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fd8069f80 0x7f4fd81a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.560+0000 7f4fe0000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd806a920 0x7f4fd81a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fe0000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8108320 0x7f4fd819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fe0000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81125d0 con 0x7f4fd8069f80 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fe0000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8112450 con 0x7f4fd8108320 2024-03-18T01:36:45.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fe0000700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8112750 con 0x7f4fd806a920 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fdcc00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd806a920 0x7f4fd81a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fdcc00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd806a920 0x7f4fd81a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:49166/0 (socket says 172.21.15.7:49166) 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fdcc00700 1 -- 172.21.15.7:0/964317983 learned_addr learned my addr 172.21.15.7:0/964317983 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fde000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8108320 0x7f4fd819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fdcc00700 1 -- 172.21.15.7:0/964317983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8108320 msgr2=0x7f4fd819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fdcc00700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8108320 0x7f4fd819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fdcc00700 1 -- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fd8069f80 msgr2=0x7f4fd81a0940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.561+0000 7f4fdd600700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fd8069f80 0x7f4fd81a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fdcc00700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fd8069f80 0x7f4fd81a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fdcc00700 1 -- 172.21.15.7:0/964317983 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd819b260 con 0x7f4fd806a920 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fde000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8108320 0x7f4fd819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fdd600700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fd8069f80 0x7f4fd81a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fdcc00700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd806a920 0x7f4fd81a0e80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4000c00 tx=0x7f4fc400be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:45.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fd6000700 1 -- 172.21.15.7:0/964317983 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4027070 con 0x7f4fd806a920 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fd6000700 1 -- 172.21.15.7:0/964317983 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fc4022470 con 0x7f4fd806a920 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fd6000700 1 -- 172.21.15.7:0/964317983 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc4017650 con 0x7f4fd806a920 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.562+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd819b4f0 con 0x7f4fd806a920 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.563+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd81a7810 con 0x7f4fd806a920 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.564+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd81039d0 con 0x7f4fd806a920 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.566+0000 7f4fd6000700 1 -- 172.21.15.7:0/964317983 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f4fc402a070 con 0x7f4fd806a920 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.567+0000 7f4fd6000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.567+0000 7f4fd6000700 1 -- 172.21.15.7:0/964317983 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4fc40cfe40 con 0x7f4fd806a920 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.567+0000 7f4fdd600700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.568+0000 7f4fdd600700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009bc0 tx=0x7f4fcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:45.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.574+0000 7f4fd6000700 1 -- 172.21.15.7:0/964317983 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4fc4013070 con 0x7f4fd806a920 2024-03-18T01:36:45.818 INFO:teuthology.orchestra.run.smithi007.stdout:197568495679 2024-03-18T01:36:45.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.832+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4fd81a79e0 con 0x7f4fd806a920 2024-03-18T01:36:45.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.832+0000 7f4fd6000700 1 -- 172.21.15.7:0/964317983 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4fc409ca50 con 0x7f4fd806a920 2024-03-18T01:36:45.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4fc00a29a0 msgr2=0x7f4fc00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc009bc0 tx=0x7f4fcc009260 comp rx=0 tx=0).stop 2024-03-18T01:36:45.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd806a920 msgr2=0x7f4fd81a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:45.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd806a920 0x7f4fd81a0e80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4000c00 tx=0x7f4fc400be70 comp rx=0 tx=0).stop 2024-03-18T01:36:45.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 shutdown_connections 2024-03-18T01:36:45.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4fc00a29a0 0x7f4fc00a4e60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fd8108320 0x7f4fd819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4fd8069f80 0x7f4fd81a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 --2- 172.21.15.7:0/964317983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fd806a920 0x7f4fd81a0e80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:45.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 >> 172.21.15.7:0/964317983 conn(0x7f4fd8078100 msgr2=0x7f4fd810d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:45.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 shutdown_connections 2024-03-18T01:36:45.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:42.836+0000 7f4fe0000700 1 -- 172.21.15.7:0/964317983 wait complete. 2024-03-18T01:36:46.152 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:45 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2443036064' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T01:36:46.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:45 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2443036064' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T01:36:46.354 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411413 for osd.2 2024-03-18T01:36:46.355 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:46.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:45 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2443036064' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T01:36:46.890 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808877 for osd.9 2024-03-18T01:36:46.890 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:47.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:46 smithi007 ceph-mon[31996]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:47.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 -- 172.21.15.7:0/3402320070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 msgr2=0x7fbab410e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:47.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 --2- 172.21.15.7:0/3402320070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 0x7fbab410e650 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009990 tx=0x7fba9c009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 -- 172.21.15.7:0/3402320070 shutdown_connections 2024-03-18T01:36:47.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 --2- 172.21.15.7:0/3402320070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbab410eb90 0x7fbab4110fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 --2- 172.21.15.7:0/3402320070 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab40746a0 0x7fbab4074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 --2- 172.21.15.7:0/3402320070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 0x7fbab410e650 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 -- 172.21.15.7:0/3402320070 >> 172.21.15.7:0/3402320070 conn(0x7fbab40fdda0 msgr2=0x7fbab41001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 -- 172.21.15.7:0/3402320070 shutdown_connections 2024-03-18T01:36:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.223+0000 7fbab9600700 1 -- 172.21.15.7:0/3402320070 wait complete. 2024-03-18T01:36:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.224+0000 7fbab9600700 1 Processor -- start 2024-03-18T01:36:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbab9600700 1 -- start start 2024-03-18T01:36:47.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbab9600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab40746a0 0x7fbab41a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbab9600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 0x7fbab41a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbab9600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbab410eb90 0x7fbab419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbab9600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbab41148a0 con 0x7fbab40746a0 2024-03-18T01:36:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbab9600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbab4114720 con 0x7fbab4074fc0 2024-03-18T01:36:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbab9600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbab4114a20 con 0x7fbab410eb90 2024-03-18T01:36:47.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbaaa000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 0x7fbab41a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbab2a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab40746a0 0x7fbab41a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbaaa000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 0x7fbab41a0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:60076/0 (socket says 172.21.15.7:60076) 2024-03-18T01:36:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.225+0000 7fbaaa000700 1 -- 172.21.15.7:0/2987835788 learned_addr learned my addr 172.21.15.7:0/2987835788 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.226+0000 7fbab3400700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbab410eb90 0x7fbab419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.226+0000 7fbaaa000700 1 -- 172.21.15.7:0/2987835788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbab410eb90 msgr2=0x7fbab419ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.226+0000 7fbaaa000700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbab410eb90 0x7fbab419ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.226+0000 7fbaaa000700 1 -- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab40746a0 msgr2=0x7fbab41a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:47.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.226+0000 7fbaaa000700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab40746a0 0x7fbab41a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.226+0000 7fbaaa000700 1 -- 172.21.15.7:0/2987835788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab419b400 con 0x7fbab4074fc0 2024-03-18T01:36:47.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.226+0000 7fbaaa000700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 0x7fbab41a0f90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fba9c005910 tx=0x7fba9c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:47.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.226+0000 7fbaabe00700 1 -- 172.21.15.7:0/2987835788 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c027070 con 0x7fbab4074fc0 2024-03-18T01:36:47.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.227+0000 7fbaabe00700 1 -- 172.21.15.7:0/2987835788 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba9c004650 con 0x7fbab4074fc0 2024-03-18T01:36:47.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.227+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab419b690 con 0x7fbab4074fc0 2024-03-18T01:36:47.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.227+0000 7fbaabe00700 1 -- 172.21.15.7:0/2987835788 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c00f460 con 0x7fbab4074fc0 2024-03-18T01:36:47.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.227+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab41a79d0 con 0x7fbab4074fc0 2024-03-18T01:36:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.228+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbab419b9f0 con 0x7fbab4074fc0 2024-03-18T01:36:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.230+0000 7fbaabe00700 1 -- 172.21.15.7:0/2987835788 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fba9c02a070 con 0x7fbab4074fc0 2024-03-18T01:36:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.231+0000 7fbaabe00700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba980a29a0 0x7fba980a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.231+0000 7fbaabe00700 1 -- 172.21.15.7:0/2987835788 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fba9c0cfda0 con 0x7fbab4074fc0 2024-03-18T01:36:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.231+0000 7fbab2a00700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba980a29a0 0x7fba980a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:47.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.232+0000 7fbab2a00700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba980a29a0 0x7fba980a4e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009c60 tx=0x7fbaa4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:47.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.235+0000 7fbaabe00700 1 -- 172.21.15.7:0/2987835788 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba9c05e740 con 0x7fbab4074fc0 2024-03-18T01:36:47.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:46 smithi046 ceph-mon[36108]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:47.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:46 smithi203 ceph-mon[36345]: pgmap v375: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:47.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.482+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fbab419bfe0 con 0x7fbab4074fc0 2024-03-18T01:36:47.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.483+0000 7fbaabe00700 1 -- 172.21.15.7:0/2987835788 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fba9c096e20 con 0x7fbab4074fc0 2024-03-18T01:36:47.484 INFO:teuthology.orchestra.run.smithi007.stdout:253403070517 2024-03-18T01:36:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba980a29a0 msgr2=0x7fba980a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba980a29a0 0x7fba980a4e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009c60 tx=0x7fbaa4009260 comp rx=0 tx=0).stop 2024-03-18T01:36:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 msgr2=0x7fbab41a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 0x7fbab41a0f90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fba9c005910 tx=0x7fba9c00bab0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 shutdown_connections 2024-03-18T01:36:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba980a29a0 0x7fba980a4e60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbab410eb90 0x7fbab419ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbab40746a0 0x7fbab41a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 --2- 172.21.15.7:0/2987835788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab4074fc0 0x7fbab41a0f90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 >> 172.21.15.7:0/2987835788 conn(0x7fbab40fdda0 msgr2=0x7fbab40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 shutdown_connections 2024-03-18T01:36:47.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:47.486+0000 7fbab9600700 1 -- 172.21.15.7:0/2987835788 wait complete. 2024-03-18T01:36:47.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332149 got 219043332153 for osd.7 2024-03-18T01:36:47.972 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:48.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:47 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2987835788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T01:36:48.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:47 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2987835788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T01:36:48.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:47 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2987835788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T01:36:48.655 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738461 got 34359738465 for osd.0 2024-03-18T01:36:48.655 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:48.761 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:49.167 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495672 got 197568495679 for osd.6 2024-03-18T01:36:49.168 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:49.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:48 smithi007 ceph-mon[31996]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:49.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:48 smithi046 ceph-mon[36108]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:49.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:48 smithi203 ceph-mon[36345]: pgmap v376: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:49.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920835 got 141733920842 for osd.4 2024-03-18T01:36:49.802 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:50.781 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:50.824 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070511 got 253403070517 for osd.8 2024-03-18T01:36:50.825 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:51.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:50 smithi007 ceph-mon[31996]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:50 smithi046 ceph-mon[36108]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:51.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:50 smithi203 ceph-mon[36345]: pgmap v377: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:53.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:52 smithi007 ceph-mon[31996]: pgmap v378: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:53.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:52 smithi046 ceph-mon[36108]: pgmap v378: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:53.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:52 smithi203 ceph-mon[36345]: pgmap v378: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 -- 172.21.15.7:0/1308531131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998107630 msgr2=0x7fa99810f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 --2- 172.21.15.7:0/1308531131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998107630 0x7fa99810f200 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa99000b0d0 tx=0x7fa99000b3e0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 -- 172.21.15.7:0/1308531131 shutdown_connections 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 --2- 172.21.15.7:0/1308531131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998107630 0x7fa99810f200 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 --2- 172.21.15.7:0/1308531131 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa998102550 0x7fa998102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 --2- 172.21.15.7:0/1308531131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa998102f00 0x7fa998106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 -- 172.21.15.7:0/1308531131 >> 172.21.15.7:0/1308531131 conn(0x7fa9980fdda0 msgr2=0x7fa9981001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 -- 172.21.15.7:0/1308531131 shutdown_connections 2024-03-18T01:36:53.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.529+0000 7fa99cc00700 1 -- 172.21.15.7:0/1308531131 wait complete. 2024-03-18T01:36:53.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.531+0000 7fa99cc00700 1 Processor -- start 2024-03-18T01:36:53.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.531+0000 7fa99cc00700 1 -- start start 2024-03-18T01:36:53.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa99cc00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998102550 0x7fa9981a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:53.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa99cc00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa998102f00 0x7fa9981a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:53.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa99cc00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa998107630 0x7fa99819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:53.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa99cc00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa998112670 con 0x7fa998107630 2024-03-18T01:36:53.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa99cc00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981124f0 con 0x7fa998102f00 2024-03-18T01:36:53.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa99cc00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981127f0 con 0x7fa998102550 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998102550 0x7fa9981a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998102550 0x7fa9981a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52132/0 (socket says 172.21.15.7:52132) 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995e00700 1 -- 172.21.15.7:0/136278503 learned_addr learned my addr 172.21.15.7:0/136278503 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995400700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa998102f00 0x7fa9981a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa996800700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa998107630 0x7fa99819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995e00700 1 -- 172.21.15.7:0/136278503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa998102f00 msgr2=0x7fa9981a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995e00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa998102f00 0x7fa9981a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995e00700 1 -- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa998107630 msgr2=0x7fa99819aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995e00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa998107630 0x7fa99819aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.532+0000 7fa995e00700 1 -- 172.21.15.7:0/136278503 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99819b1c0 con 0x7fa998102550 2024-03-18T01:36:53.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.533+0000 7fa995e00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998102550 0x7fa9981a08a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa98800e9a0 tx=0x7fa98800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:53.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.533+0000 7fa98ea00700 1 -- 172.21.15.7:0/136278503 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98800cbd0 con 0x7fa998102550 2024-03-18T01:36:53.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.533+0000 7fa98ea00700 1 -- 172.21.15.7:0/136278503 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa98800cd70 con 0x7fa998102550 2024-03-18T01:36:53.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.533+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99819b4b0 con 0x7fa998102550 2024-03-18T01:36:53.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.533+0000 7fa98ea00700 1 -- 172.21.15.7:0/136278503 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9880187c0 con 0x7fa998102550 2024-03-18T01:36:53.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.533+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9981a7770 con 0x7fa998102550 2024-03-18T01:36:53.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.534+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa998068f90 con 0x7fa998102550 2024-03-18T01:36:53.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.536+0000 7fa98ea00700 1 -- 172.21.15.7:0/136278503 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7fa988010430 con 0x7fa998102550 2024-03-18T01:36:53.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.538+0000 7fa98ea00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa97c0a29a0 0x7fa97c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:53.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.538+0000 7fa98ea00700 1 -- 172.21.15.7:0/136278503 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa9880cf230 con 0x7fa998102550 2024-03-18T01:36:53.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.538+0000 7fa995400700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa97c0a29a0 0x7fa97c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:53.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.539+0000 7fa995400700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa97c0a29a0 0x7fa97c0a4e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa984009680 tx=0x7fa984009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:53.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.541+0000 7fa98ea00700 1 -- 172.21.15.7:0/136278503 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa98809bdb0 con 0x7fa998102550 2024-03-18T01:36:53.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.816+0000 7f5b07e00700 1 -- 172.21.15.7:0/2383850808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0810ea50 msgr2=0x7f5b08110e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:53.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.816+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2383850808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0810ea50 0x7f5b08110e60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009990 tx=0x7f5b00009ca0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.816+0000 7f5b07e00700 1 -- 172.21.15.7:0/2383850808 shutdown_connections 2024-03-18T01:36:53.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.816+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2383850808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b0810ea50 0x7f5b08110e60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.816+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2383850808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b080746a0 0x7f5b08074a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.816+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2383850808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b08074fc0 0x7f5b0810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.816+0000 7f5b07e00700 1 -- 172.21.15.7:0/2383850808 >> 172.21.15.7:0/2383850808 conn(0x7f5b080fdca0 msgr2=0x7f5b081000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:53.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.816+0000 7f5b07e00700 1 -- 172.21.15.7:0/2383850808 shutdown_connections 2024-03-18T01:36:53.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.817+0000 7f5b07e00700 1 -- 172.21.15.7:0/2383850808 wait complete. 2024-03-18T01:36:53.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b07e00700 1 Processor -- start 2024-03-18T01:36:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b07e00700 1 -- start start 2024-03-18T01:36:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b07e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b080746a0 0x7f5b081a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b07e00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b08074fc0 0x7f5b081a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b07e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b0810ea50 0x7f5b0819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b07e00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5b08114710 con 0x7f5b0810ea50 2024-03-18T01:36:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b07e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b08114590 con 0x7f5b080746a0 2024-03-18T01:36:53.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b07e00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5b08114890 con 0x7f5b08074fc0 2024-03-18T01:36:53.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b06a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b080746a0 0x7f5b081a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:53.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.818+0000 7f5b06000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b08074fc0 0x7f5b081a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:53.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b06a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b080746a0 0x7f5b081a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:60108/0 (socket says 172.21.15.7:60108) 2024-03-18T01:36:53.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b06a00700 1 -- 172.21.15.7:0/2261848013 learned_addr learned my addr 172.21.15.7:0/2261848013 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:53.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b07400700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b0810ea50 0x7f5b0819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:53.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b06a00700 1 -- 172.21.15.7:0/2261848013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b08074fc0 msgr2=0x7f5b081a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:53.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b06a00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b08074fc0 0x7f5b081a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b06a00700 1 -- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b0810ea50 msgr2=0x7f5b0819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:53.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b06a00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b0810ea50 0x7f5b0819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b06a00700 1 -- 172.21.15.7:0/2261848013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0819b0f0 con 0x7f5b080746a0 2024-03-18T01:36:53.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.819+0000 7f5b06a00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b080746a0 0x7f5b081a07d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5af800ef10 tx=0x7f5af800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:53.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.820+0000 7f5aff400700 1 -- 172.21.15.7:0/2261848013 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af800ca70 con 0x7f5b080746a0 2024-03-18T01:36:53.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.820+0000 7f5aff400700 1 -- 172.21.15.7:0/2261848013 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5af8019070 con 0x7f5b080746a0 2024-03-18T01:36:53.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.820+0000 7f5aff400700 1 -- 172.21.15.7:0/2261848013 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af80146e0 con 0x7f5b080746a0 2024-03-18T01:36:53.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.820+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0819b380 con 0x7f5b080746a0 2024-03-18T01:36:53.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.820+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b081a76a0 con 0x7f5b080746a0 2024-03-18T01:36:53.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.822+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b08068f90 con 0x7f5b080746a0 2024-03-18T01:36:53.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.824+0000 7f5aff400700 1 -- 172.21.15.7:0/2261848013 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 137340+0+0 (secure 0 0 0) 0x7f5af8012050 con 0x7f5b080746a0 2024-03-18T01:36:53.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.825+0000 7f5aff400700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aec0a29a0 0x7f5aec0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:53.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.825+0000 7f5aff400700 1 -- 172.21.15.7:0/2261848013 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5af80ceeb0 con 0x7f5b080746a0 2024-03-18T01:36:53.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.825+0000 7f5b06000700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aec0a29a0 0x7f5aec0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:53.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.826+0000 7f5b06000700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aec0a29a0 0x7f5aec0a4e60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5b081018a0 tx=0x7f5af4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:53.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.829+0000 7f5aff400700 1 -- 172.21.15.7:0/2261848013 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5af8010040 con 0x7f5b080746a0 2024-03-18T01:36:53.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.837+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa9981a79d0 con 0x7fa998102550 2024-03-18T01:36:53.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.838+0000 7fa98ea00700 1 -- 172.21.15.7:0/136278503 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa9880bb070 con 0x7fa998102550 2024-03-18T01:36:53.840 INFO:teuthology.orchestra.run.smithi007.stdout:107374182481 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa97c0a29a0 msgr2=0x7fa97c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa97c0a29a0 0x7fa97c0a4e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa984009680 tx=0x7fa984009260 comp rx=0 tx=0).stop 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998102550 msgr2=0x7fa9981a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998102550 0x7fa9981a08a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa98800e9a0 tx=0x7fa98800ed60 comp rx=0 tx=0).stop 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 shutdown_connections 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa97c0a29a0 0x7fa97c0a4e60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa998107630 0x7fa99819aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa998102550 0x7fa9981a08a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 --2- 172.21.15.7:0/136278503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa998102f00 0x7fa9981a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 >> 172.21.15.7:0/136278503 conn(0x7fa9980fdda0 msgr2=0x7fa99810eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 shutdown_connections 2024-03-18T01:36:53.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:53.841+0000 7fa99cc00700 1 -- 172.21.15.7:0/136278503 wait complete. 2024-03-18T01:36:54.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:53 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/136278503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T01:36:54.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.089+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f5b0802db70 con 0x7f5b080746a0 2024-03-18T01:36:54.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.089+0000 7f5aff400700 1 -- 172.21.15.7:0/2261848013 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5af809ba30 con 0x7f5b080746a0 2024-03-18T01:36:54.091 INFO:teuthology.orchestra.run.smithi007.stdout:163208757320 2024-03-18T01:36:54.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.092+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aec0a29a0 msgr2=0x7f5aec0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.092+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aec0a29a0 0x7f5aec0a4e60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5b081018a0 tx=0x7f5af4009260 comp rx=0 tx=0).stop 2024-03-18T01:36:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.092+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b080746a0 msgr2=0x7f5b081a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.092+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b080746a0 0x7f5b081a07d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5af800ef10 tx=0x7f5af800c3a0 comp rx=0 tx=0).stop 2024-03-18T01:36:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.093+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 shutdown_connections 2024-03-18T01:36:54.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.093+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5aec0a29a0 0x7f5aec0a4e60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.093+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b0810ea50 0x7f5b0819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.093+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b080746a0 0x7f5b081a07d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.093+0000 7f5b07e00700 1 --2- 172.21.15.7:0/2261848013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b08074fc0 0x7f5b081a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.093+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 >> 172.21.15.7:0/2261848013 conn(0x7f5b080fdca0 msgr2=0x7f5b080fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.093+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 shutdown_connections 2024-03-18T01:36:54.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:54.093+0000 7f5b07e00700 1 -- 172.21.15.7:0/2261848013 wait complete. 2024-03-18T01:36:54.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:53 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/136278503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T01:36:54.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:53 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/136278503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T01:36:54.923 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757313 got 163208757320 for osd.5 2024-03-18T01:36:54.924 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:55.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:54 smithi046 ceph-mon[36108]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:54 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2261848013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T01:36:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:54 smithi007 ceph-mon[31996]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:54 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2261848013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T01:36:55.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:54 smithi203 ceph-mon[36345]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:36:55.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:54 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2261848013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T01:36:55.481 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182475 got 107374182481 for osd.3 2024-03-18T01:36:55.481 DEBUG:teuthology.parallel:result is None 2024-03-18T01:36:55.482 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T01:36:55.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T01:36:55.958 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:36:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:56 smithi046 ceph-mon[36108]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:57.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:56 smithi007 ceph-mon[31996]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:56 smithi203 ceph-mon[36345]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 -- 172.21.15.7:0/2861757476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 msgr2=0x7fd874106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 --2- 172.21.15.7:0/2861757476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 0x7fd874106f10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd8600099c0 tx=0x7fd860009cd0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 -- 172.21.15.7:0/2861757476 shutdown_connections 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 --2- 172.21.15.7:0/2861757476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd874107640 0x7fd87410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 --2- 172.21.15.7:0/2861757476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd87406abd0 0x7fd87406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 --2- 172.21.15.7:0/2861757476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 0x7fd874106f10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 -- 172.21.15.7:0/2861757476 >> 172.21.15.7:0/2861757476 conn(0x7fd874078140 msgr2=0x7fd874078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 -- 172.21.15.7:0/2861757476 shutdown_connections 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.257+0000 7fd87be00700 1 -- 172.21.15.7:0/2861757476 wait complete. 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.258+0000 7fd87be00700 1 Processor -- start 2024-03-18T01:36:58.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.258+0000 7fd87be00700 1 -- start start 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd87be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd87406abd0 0x7fd8741a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd87be00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 0x7fd8741a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd87be00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd874107640 0x7fd87419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd87be00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd874112600 con 0x7fd874107640 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd87be00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd874112480 con 0x7fd87406abd0 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd87be00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd874112780 con 0x7fd87406b580 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd878a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 0x7fd8741a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd879e00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd874107640 0x7fd87419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd878a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 0x7fd8741a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:55218/0 (socket says 172.21.15.7:55218) 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd878a00700 1 -- 172.21.15.7:0/1400551239 learned_addr learned my addr 172.21.15.7:0/1400551239 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:36:58.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.259+0000 7fd879400700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd87406abd0 0x7fd8741a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd878a00700 1 -- 172.21.15.7:0/1400551239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd87406abd0 msgr2=0x7fd8741a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd878a00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd87406abd0 0x7fd8741a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd878a00700 1 -- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd874107640 msgr2=0x7fd87419ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd878a00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd874107640 0x7fd87419ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd878a00700 1 -- 172.21.15.7:0/1400551239 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87419b2b0 con 0x7fd87406b580 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd879e00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd874107640 0x7fd87419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd878a00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 0x7fd8741a0e80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098e0 tx=0x7fd860003e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd872000700 1 -- 172.21.15.7:0/1400551239 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd860009e60 con 0x7fd87406b580 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd872000700 1 -- 172.21.15.7:0/1400551239 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd860003680 con 0x7fd87406b580 2024-03-18T01:36:58.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87419b540 con 0x7fd87406b580 2024-03-18T01:36:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.260+0000 7fd872000700 1 -- 172.21.15.7:0/1400551239 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8600173f0 con 0x7fd87406b580 2024-03-18T01:36:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.261+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8741a7810 con 0x7fd87406b580 2024-03-18T01:36:58.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.262+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd87406a220 con 0x7fd87406b580 2024-03-18T01:36:58.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.264+0000 7fd872000700 1 -- 172.21.15.7:0/1400551239 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fd86000f660 con 0x7fd87406b580 2024-03-18T01:36:58.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.266+0000 7fd872000700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd85c0a2890 0x7fd85c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:36:58.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.267+0000 7fd872000700 1 -- 172.21.15.7:0/1400551239 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd8600cfb70 con 0x7fd87406b580 2024-03-18T01:36:58.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.267+0000 7fd879400700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd85c0a2890 0x7fd85c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:36:58.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.268+0000 7fd879400700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd85c0a2890 0x7fd85c0a4d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd868009c60 tx=0x7fd868009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:36:58.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.269+0000 7fd872000700 1 -- 172.21.15.7:0/1400551239 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd860013070 con 0x7fd87406b580 2024-03-18T01:36:58.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.490+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd87410dc50 con 0x7fd85c0a2890 2024-03-18T01:36:58.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.534+0000 7fd872000700 1 -- 172.21.15.7:0/1400551239 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340417 (secure 0 0 0) 0x7fd87410dc50 con 0x7fd85c0a2890 2024-03-18T01:36:58.541 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.540+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd85c0a2890 msgr2=0x7fd85c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd85c0a2890 0x7fd85c0a4d50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd868009c60 tx=0x7fd868009260 comp rx=0 tx=0).stop 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 msgr2=0x7fd8741a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 0x7fd8741a0e80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd8600098e0 tx=0x7fd860003e00 comp rx=0 tx=0).stop 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 shutdown_connections 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd85c0a2890 0x7fd85c0a4d50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd874107640 0x7fd87419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd87406abd0 0x7fd8741a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 --2- 172.21.15.7:0/1400551239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd87406b580 0x7fd8741a0e80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:36:58.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 >> 172.21.15.7:0/1400551239 conn(0x7fd874078140 msgr2=0x7fd87410d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:36:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 shutdown_connections 2024-03-18T01:36:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:36:58.541+0000 7fd87be00700 1 -- 172.21.15.7:0/1400551239 wait complete. 2024-03-18T01:36:58.546 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-18T01:36:59.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:58 smithi046 ceph-mon[36108]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:58 smithi046 ceph-mon[36108]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T01:36:59.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:58 smithi007 ceph-mon[31996]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:59.288 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:58 smithi007 ceph-mon[31996]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T01:36:59.358 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-03-18T01:36:57.218390+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659487,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1003,"num_read_kb":620,"num_write":424,"num_write_kb":3078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":347,"ondisk_log_size":347,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363460,"kb_used_data":14228,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578364,"statfs":{"total":1151940427776,"available":1151568244736,"internally_reserved":0,"allocated":14569472,"data_stored":7964267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005785"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804994+0000","last_change":"2024-03-18T01:34:56.661330+0000","last_active":"2024-03-18T01:35:58.804994+0000","last_peered":"2024-03-18T01:35:58.804994+0000","last_clean":"2024-03-18T01:35:58.804994+0000","last_became_active":"2024-03-18T01:34:56.660949+0000","last_became_peered":"2024-03-18T01:34:56.660949+0000","last_unstale":"2024-03-18T01:35:58.804994+0000","last_undegraded":"2024-03-18T01:35:58.804994+0000","last_fullsized":"2024-03-18T01:35:58.804994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:40.962487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811347+0000","last_change":"2024-03-18T01:34:56.652417+0000","last_active":"2024-03-18T01:35:58.811347+0000","last_peered":"2024-03-18T01:35:58.811347+0000","last_clean":"2024-03-18T01:35:58.811347+0000","last_became_active":"2024-03-18T01:34:56.651851+0000","last_became_peered":"2024-03-18T01:34:56.651851+0000","last_unstale":"2024-03-18T01:35:58.811347+0000","last_undegraded":"2024-03-18T01:35:58.811347+0000","last_fullsized":"2024-03-18T01:35:58.811347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:14:24.172177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812975+0000","last_change":"2024-03-18T01:34:56.647657+0000","last_active":"2024-03-18T01:35:58.812975+0000","last_peered":"2024-03-18T01:35:58.812975+0000","last_clean":"2024-03-18T01:35:58.812975+0000","last_became_active":"2024-03-18T01:34:56.647241+0000","last_became_peered":"2024-03-18T01:34:56.647241+0000","last_unstale":"2024-03-18T01:35:58.812975+0000","last_undegraded":"2024-03-18T01:35:58.812975+0000","last_fullsized":"2024-03-18T01:35:58.812975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:01.706351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.815941+0000","last_change":"2024-03-18T01:35:58.815017+0000","last_active":"2024-03-18T01:35:58.815941+0000","last_peered":"2024-03-18T01:35:58.815941+0000","last_clean":"2024-03-18T01:35:58.815941+0000","last_became_active":"2024-03-18T01:35:58.814140+0000","last_became_peered":"2024-03-18T01:35:58.814140+0000","last_unstale":"2024-03-18T01:35:58.815941+0000","last_undegraded":"2024-03-18T01:35:58.815941+0000","last_fullsized":"2024-03-18T01:35:58.815941+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:13:21.960119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802130+0000","last_change":"2024-03-18T01:34:56.643992+0000","last_active":"2024-03-18T01:35:58.802130+0000","last_peered":"2024-03-18T01:35:58.802130+0000","last_clean":"2024-03-18T01:35:58.802130+0000","last_became_active":"2024-03-18T01:34:56.643652+0000","last_became_peered":"2024-03-18T01:34:56.643652+0000","last_unstale":"2024-03-18T01:35:58.802130+0000","last_undegraded":"2024-03-18T01:35:58.802130+0000","last_fullsized":"2024-03-18T01:35:58.802130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:12.005357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804093+0000","last_change":"2024-03-18T01:34:56.649955+0000","last_active":"2024-03-18T01:35:58.804093+0000","last_peered":"2024-03-18T01:35:58.804093+0000","last_clean":"2024-03-18T01:35:58.804093+0000","last_became_active":"2024-03-18T01:34:56.649609+0000","last_became_peered":"2024-03-18T01:34:56.649609+0000","last_unstale":"2024-03-18T01:35:58.804093+0000","last_undegraded":"2024-03-18T01:35:58.804093+0000","last_fullsized":"2024-03-18T01:35:58.804093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:34:28.113733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806244+0000","last_change":"2024-03-18T01:34:56.647561+0000","last_active":"2024-03-18T01:35:58.806244+0000","last_peered":"2024-03-18T01:35:58.806244+0000","last_clean":"2024-03-18T01:35:58.806244+0000","last_became_active":"2024-03-18T01:34:56.647040+0000","last_became_peered":"2024-03-18T01:34:56.647040+0000","last_unstale":"2024-03-18T01:35:58.806244+0000","last_undegraded":"2024-03-18T01:35:58.806244+0000","last_fullsized":"2024-03-18T01:35:58.806244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:42.433751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816764+0000","last_change":"2024-03-18T01:34:56.652073+0000","last_active":"2024-03-18T01:35:58.816764+0000","last_peered":"2024-03-18T01:35:58.816764+0000","last_clean":"2024-03-18T01:35:58.816764+0000","last_became_active":"2024-03-18T01:34:56.651748+0000","last_became_peered":"2024-03-18T01:34:56.651748+0000","last_unstale":"2024-03-18T01:35:58.816764+0000","last_undegraded":"2024-03-18T01:35:58.816764+0000","last_fullsized":"2024-03-18T01:35:58.816764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:33.148823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807631+0000","last_change":"2024-03-18T01:34:56.645782+0000","last_active":"2024-03-18T01:35:58.807631+0000","last_peered":"2024-03-18T01:35:58.807631+0000","last_clean":"2024-03-18T01:35:58.807631+0000","last_became_active":"2024-03-18T01:34:56.645439+0000","last_became_peered":"2024-03-18T01:34:56.645439+0000","last_unstale":"2024-03-18T01:35:58.807631+0000","last_undegraded":"2024-03-18T01:35:58.807631+0000","last_fullsized":"2024-03-18T01:35:58.807631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:40.242348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804895+0000","last_change":"2024-03-18T01:34:56.639697+0000","last_active":"2024-03-18T01:35:58.804895+0000","last_peered":"2024-03-18T01:35:58.804895+0000","last_clean":"2024-03-18T01:35:58.804895+0000","last_became_active":"2024-03-18T01:34:56.639339+0000","last_became_peered":"2024-03-18T01:34:56.639339+0000","last_unstale":"2024-03-18T01:35:58.804895+0000","last_undegraded":"2024-03-18T01:35:58.804895+0000","last_fullsized":"2024-03-18T01:35:58.804895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:25:22.741437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806829+0000","last_change":"2024-03-18T01:34:56.655341+0000","last_active":"2024-03-18T01:35:58.806829+0000","last_peered":"2024-03-18T01:35:58.806829+0000","last_clean":"2024-03-18T01:35:58.806829+0000","last_became_active":"2024-03-18T01:34:56.655015+0000","last_became_peered":"2024-03-18T01:34:56.655015+0000","last_unstale":"2024-03-18T01:35:58.806829+0000","last_undegraded":"2024-03-18T01:35:58.806829+0000","last_fullsized":"2024-03-18T01:35:58.806829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:50.766000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802137+0000","last_change":"2024-03-18T01:34:56.648399+0000","last_active":"2024-03-18T01:35:58.802137+0000","last_peered":"2024-03-18T01:35:58.802137+0000","last_clean":"2024-03-18T01:35:58.802137+0000","last_became_active":"2024-03-18T01:34:56.648042+0000","last_became_peered":"2024-03-18T01:34:56.648042+0000","last_unstale":"2024-03-18T01:35:58.802137+0000","last_undegraded":"2024-03-18T01:35:58.802137+0000","last_fullsized":"2024-03-18T01:35:58.802137+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:04.972230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819084+0000","last_change":"2024-03-18T01:34:56.653465+0000","last_active":"2024-03-18T01:35:58.819084+0000","last_peered":"2024-03-18T01:35:58.819084+0000","last_clean":"2024-03-18T01:35:58.819084+0000","last_became_active":"2024-03-18T01:34:56.651183+0000","last_became_peered":"2024-03-18T01:34:56.651183+0000","last_unstale":"2024-03-18T01:35:58.819084+0000","last_undegraded":"2024-03-18T01:35:58.819084+0000","last_fullsized":"2024-03-18T01:35:58.819084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:57.983330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812736+0000","last_change":"2024-03-18T01:34:56.658845+0000","last_active":"2024-03-18T01:35:58.812736+0000","last_peered":"2024-03-18T01:35:58.812736+0000","last_clean":"2024-03-18T01:35:58.812736+0000","last_became_active":"2024-03-18T01:34:56.658435+0000","last_became_peered":"2024-03-18T01:34:56.658435+0000","last_unstale":"2024-03-18T01:35:58.812736+0000","last_undegraded":"2024-03-18T01:35:58.812736+0000","last_fullsized":"2024-03-18T01:35:58.812736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:02.176325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809489+0000","last_change":"2024-03-18T01:34:56.660752+0000","last_active":"2024-03-18T01:35:58.809489+0000","last_peered":"2024-03-18T01:35:58.809489+0000","last_clean":"2024-03-18T01:35:58.809489+0000","last_became_active":"2024-03-18T01:34:56.660378+0000","last_became_peered":"2024-03-18T01:34:56.660378+0000","last_unstale":"2024-03-18T01:35:58.809489+0000","last_undegraded":"2024-03-18T01:35:58.809489+0000","last_fullsized":"2024-03-18T01:35:58.809489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:13.143481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806591+0000","last_change":"2024-03-18T01:34:56.656589+0000","last_active":"2024-03-18T01:35:58.806591+0000","last_peered":"2024-03-18T01:35:58.806591+0000","last_clean":"2024-03-18T01:35:58.806591+0000","last_became_active":"2024-03-18T01:34:56.656184+0000","last_became_peered":"2024-03-18T01:34:56.656184+0000","last_unstale":"2024-03-18T01:35:58.806591+0000","last_undegraded":"2024-03-18T01:35:58.806591+0000","last_fullsized":"2024-03-18T01:35:58.806591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:56.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804766+0000","last_change":"2024-03-18T01:34:56.661327+0000","last_active":"2024-03-18T01:35:58.804766+0000","last_peered":"2024-03-18T01:35:58.804766+0000","last_clean":"2024-03-18T01:35:58.804766+0000","last_became_active":"2024-03-18T01:34:56.660767+0000","last_became_peered":"2024-03-18T01:34:56.660767+0000","last_unstale":"2024-03-18T01:35:58.804766+0000","last_undegraded":"2024-03-18T01:35:58.804766+0000","last_fullsized":"2024-03-18T01:35:58.804766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:07:59.066028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811514+0000","last_change":"2024-03-18T01:34:56.650149+0000","last_active":"2024-03-18T01:35:58.811514+0000","last_peered":"2024-03-18T01:35:58.811514+0000","last_clean":"2024-03-18T01:35:58.811514+0000","last_became_active":"2024-03-18T01:34:56.649780+0000","last_became_peered":"2024-03-18T01:34:56.649780+0000","last_unstale":"2024-03-18T01:35:58.811514+0000","last_undegraded":"2024-03-18T01:35:58.811514+0000","last_fullsized":"2024-03-18T01:35:58.811514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:41.605563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809699+0000","last_change":"2024-03-18T01:34:56.653919+0000","last_active":"2024-03-18T01:35:58.809699+0000","last_peered":"2024-03-18T01:35:58.809699+0000","last_clean":"2024-03-18T01:35:58.809699+0000","last_became_active":"2024-03-18T01:34:56.653512+0000","last_became_peered":"2024-03-18T01:34:56.653512+0000","last_unstale":"2024-03-18T01:35:58.809699+0000","last_undegraded":"2024-03-18T01:35:58.809699+0000","last_fullsized":"2024-03-18T01:35:58.809699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:27.419518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810112+0000","last_change":"2024-03-18T01:35:01.149999+0000","last_active":"2024-03-18T01:35:58.810112+0000","last_peered":"2024-03-18T01:35:58.810112+0000","last_clean":"2024-03-18T01:35:58.810112+0000","last_became_active":"2024-03-18T01:35:01.149603+0000","last_became_peered":"2024-03-18T01:35:01.149603+0000","last_unstale":"2024-03-18T01:35:58.810112+0000","last_undegraded":"2024-03-18T01:35:58.810112+0000","last_fullsized":"2024-03-18T01:35:58.810112+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:37.619517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:54.772445+0000","last_change":"2024-03-18T01:34:55.650404+0000","last_active":"2024-03-18T01:36:54.772445+0000","last_peered":"2024-03-18T01:36:54.772445+0000","last_clean":"2024-03-18T01:36:54.772445+0000","last_became_active":"2024-03-18T01:34:55.650068+0000","last_became_peered":"2024-03-18T01:34:55.650068+0000","last_unstale":"2024-03-18T01:36:54.772445+0000","last_undegraded":"2024-03-18T01:36:54.772445+0000","last_fullsized":"2024-03-18T01:36:54.772445+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:11.396529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821635+0000","last_change":"2024-03-18T01:34:56.654265+0000","last_active":"2024-03-18T01:35:58.821635+0000","last_peered":"2024-03-18T01:35:58.821635+0000","last_clean":"2024-03-18T01:35:58.821635+0000","last_became_active":"2024-03-18T01:34:56.653886+0000","last_became_peered":"2024-03-18T01:34:56.653886+0000","last_unstale":"2024-03-18T01:35:58.821635+0000","last_undegraded":"2024-03-18T01:35:58.821635+0000","last_fullsized":"2024-03-18T01:35:58.821635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:01:44.069966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801447+0000","last_change":"2024-03-18T01:35:01.155838+0000","last_active":"2024-03-18T01:35:58.801447+0000","last_peered":"2024-03-18T01:35:58.801447+0000","last_clean":"2024-03-18T01:35:58.801447+0000","last_became_active":"2024-03-18T01:35:01.155391+0000","last_became_peered":"2024-03-18T01:35:01.155391+0000","last_unstale":"2024-03-18T01:35:58.801447+0000","last_undegraded":"2024-03-18T01:35:58.801447+0000","last_fullsized":"2024-03-18T01:35:58.801447+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:31.885767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821927+0000","last_change":"2024-03-18T01:34:55.658830+0000","last_active":"2024-03-18T01:35:58.821927+0000","last_peered":"2024-03-18T01:35:58.821927+0000","last_clean":"2024-03-18T01:35:58.821927+0000","last_became_active":"2024-03-18T01:34:55.658395+0000","last_became_peered":"2024-03-18T01:34:55.658395+0000","last_unstale":"2024-03-18T01:35:58.821927+0000","last_undegraded":"2024-03-18T01:35:58.821927+0000","last_fullsized":"2024-03-18T01:35:58.821927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:07.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810867+0000","last_change":"2024-03-18T01:34:56.656874+0000","last_active":"2024-03-18T01:35:58.810867+0000","last_peered":"2024-03-18T01:35:58.810867+0000","last_clean":"2024-03-18T01:35:58.810867+0000","last_became_active":"2024-03-18T01:34:56.656525+0000","last_became_peered":"2024-03-18T01:34:56.656525+0000","last_unstale":"2024-03-18T01:35:58.810867+0000","last_undegraded":"2024-03-18T01:35:58.810867+0000","last_fullsized":"2024-03-18T01:35:58.810867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:35:19.363586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.822392+0000","last_change":"2024-03-18T01:35:58.822392+0000","last_active":"2024-03-18T01:35:58.822392+0000","last_peered":"2024-03-18T01:35:58.822392+0000","last_clean":"2024-03-18T01:35:58.822392+0000","last_became_active":"2024-03-18T01:35:58.821792+0000","last_became_peered":"2024-03-18T01:35:58.821792+0000","last_unstale":"2024-03-18T01:35:58.822392+0000","last_undegraded":"2024-03-18T01:35:58.822392+0000","last_fullsized":"2024-03-18T01:35:58.822392+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:43:26.345066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802688+0000","last_change":"2024-03-18T01:34:55.648218+0000","last_active":"2024-03-18T01:35:58.802688+0000","last_peered":"2024-03-18T01:35:58.802688+0000","last_clean":"2024-03-18T01:35:58.802688+0000","last_became_active":"2024-03-18T01:34:55.647779+0000","last_became_peered":"2024-03-18T01:34:55.647779+0000","last_unstale":"2024-03-18T01:35:58.802688+0000","last_undegraded":"2024-03-18T01:35:58.802688+0000","last_fullsized":"2024-03-18T01:35:58.802688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:22:08.136973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807503+0000","last_change":"2024-03-18T01:34:56.649900+0000","last_active":"2024-03-18T01:35:58.807503+0000","last_peered":"2024-03-18T01:35:58.807503+0000","last_clean":"2024-03-18T01:35:58.807503+0000","last_became_active":"2024-03-18T01:34:56.649564+0000","last_became_peered":"2024-03-18T01:34:56.649564+0000","last_unstale":"2024-03-18T01:35:58.807503+0000","last_undegraded":"2024-03-18T01:35:58.807503+0000","last_fullsized":"2024-03-18T01:35:58.807503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:29.861861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801554+0000","last_change":"2024-03-18T01:35:01.147062+0000","last_active":"2024-03-18T01:35:58.801554+0000","last_peered":"2024-03-18T01:35:58.801554+0000","last_clean":"2024-03-18T01:35:58.801554+0000","last_became_active":"2024-03-18T01:35:01.146666+0000","last_became_peered":"2024-03-18T01:35:01.146666+0000","last_unstale":"2024-03-18T01:35:58.801554+0000","last_undegraded":"2024-03-18T01:35:58.801554+0000","last_fullsized":"2024-03-18T01:35:58.801554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:15:21.433263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.817437+0000","last_change":"2024-03-18T01:35:58.816601+0000","last_active":"2024-03-18T01:35:58.817437+0000","last_peered":"2024-03-18T01:35:58.817437+0000","last_clean":"2024-03-18T01:35:58.817437+0000","last_became_active":"2024-03-18T01:35:58.808678+0000","last_became_peered":"2024-03-18T01:35:58.808678+0000","last_unstale":"2024-03-18T01:35:58.817437+0000","last_undegraded":"2024-03-18T01:35:58.817437+0000","last_fullsized":"2024-03-18T01:35:58.817437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:37:42.998856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808641+0000","last_change":"2024-03-18T01:34:56.650449+0000","last_active":"2024-03-18T01:35:58.808641+0000","last_peered":"2024-03-18T01:35:58.808641+0000","last_clean":"2024-03-18T01:35:58.808641+0000","last_became_active":"2024-03-18T01:34:56.650103+0000","last_became_peered":"2024-03-18T01:34:56.650103+0000","last_unstale":"2024-03-18T01:35:58.808641+0000","last_undegraded":"2024-03-18T01:35:58.808641+0000","last_fullsized":"2024-03-18T01:35:58.808641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:07:55.003592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809552+0000","last_change":"2024-03-18T01:35:01.132227+0000","last_active":"2024-03-18T01:35:58.809552+0000","last_peered":"2024-03-18T01:35:58.809552+0000","last_clean":"2024-03-18T01:35:58.809552+0000","last_became_active":"2024-03-18T01:35:01.131767+0000","last_became_peered":"2024-03-18T01:35:01.131767+0000","last_unstale":"2024-03-18T01:35:58.809552+0000","last_undegraded":"2024-03-18T01:35:58.809552+0000","last_fullsized":"2024-03-18T01:35:58.809552+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:54.335482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:52.731051+0000","last_change":"2024-03-18T01:34:55.649277+0000","last_active":"2024-03-18T01:36:52.731051+0000","last_peered":"2024-03-18T01:36:52.731051+0000","last_clean":"2024-03-18T01:36:52.731051+0000","last_became_active":"2024-03-18T01:34:55.648917+0000","last_became_peered":"2024-03-18T01:34:55.648917+0000","last_unstale":"2024-03-18T01:36:52.731051+0000","last_undegraded":"2024-03-18T01:36:52.731051+0000","last_fullsized":"2024-03-18T01:36:52.731051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:00.117647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802011+0000","last_change":"2024-03-18T01:34:56.648394+0000","last_active":"2024-03-18T01:35:58.802011+0000","last_peered":"2024-03-18T01:35:58.802011+0000","last_clean":"2024-03-18T01:35:58.802011+0000","last_became_active":"2024-03-18T01:34:56.648043+0000","last_became_peered":"2024-03-18T01:34:56.648043+0000","last_unstale":"2024-03-18T01:35:58.802011+0000","last_undegraded":"2024-03-18T01:35:58.802011+0000","last_fullsized":"2024-03-18T01:35:58.802011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:54:52.531851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809458+0000","last_change":"2024-03-18T01:35:01.151346+0000","last_active":"2024-03-18T01:35:58.809458+0000","last_peered":"2024-03-18T01:35:58.809458+0000","last_clean":"2024-03-18T01:35:58.809458+0000","last_became_active":"2024-03-18T01:35:01.150916+0000","last_became_peered":"2024-03-18T01:35:01.150916+0000","last_unstale":"2024-03-18T01:35:58.809458+0000","last_undegraded":"2024-03-18T01:35:58.809458+0000","last_fullsized":"2024-03-18T01:35:58.809458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:44:36.000983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.803936+0000","last_change":"2024-03-18T01:34:55.648498+0000","last_active":"2024-03-18T01:35:58.803936+0000","last_peered":"2024-03-18T01:35:58.803936+0000","last_clean":"2024-03-18T01:35:58.803936+0000","last_became_active":"2024-03-18T01:34:55.648152+0000","last_became_peered":"2024-03-18T01:34:55.648152+0000","last_unstale":"2024-03-18T01:35:58.803936+0000","last_undegraded":"2024-03-18T01:35:58.803936+0000","last_fullsized":"2024-03-18T01:35:58.803936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:33.138620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808232+0000","last_change":"2024-03-18T01:34:56.651420+0000","last_active":"2024-03-18T01:35:58.808232+0000","last_peered":"2024-03-18T01:35:58.808232+0000","last_clean":"2024-03-18T01:35:58.808232+0000","last_became_active":"2024-03-18T01:34:56.650909+0000","last_became_peered":"2024-03-18T01:34:56.650909+0000","last_unstale":"2024-03-18T01:35:58.808232+0000","last_undegraded":"2024-03-18T01:35:58.808232+0000","last_fullsized":"2024-03-18T01:35:58.808232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:43:40.548970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804655+0000","last_change":"2024-03-18T01:35:01.157759+0000","last_active":"2024-03-18T01:35:58.804655+0000","last_peered":"2024-03-18T01:35:58.804655+0000","last_clean":"2024-03-18T01:35:58.804655+0000","last_became_active":"2024-03-18T01:35:01.157288+0000","last_became_peered":"2024-03-18T01:35:01.157288+0000","last_unstale":"2024-03-18T01:35:58.804655+0000","last_undegraded":"2024-03-18T01:35:58.804655+0000","last_fullsized":"2024-03-18T01:35:58.804655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:05:41.272727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.818719+0000","last_change":"2024-03-18T01:34:55.650388+0000","last_active":"2024-03-18T01:35:58.818719+0000","last_peered":"2024-03-18T01:35:58.818719+0000","last_clean":"2024-03-18T01:35:58.818719+0000","last_became_active":"2024-03-18T01:34:55.650055+0000","last_became_peered":"2024-03-18T01:34:55.650055+0000","last_unstale":"2024-03-18T01:35:58.818719+0000","last_undegraded":"2024-03-18T01:35:58.818719+0000","last_fullsized":"2024-03-18T01:35:58.818719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:29:40.758561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816634+0000","last_change":"2024-03-18T01:34:56.658634+0000","last_active":"2024-03-18T01:35:58.816634+0000","last_peered":"2024-03-18T01:35:58.816634+0000","last_clean":"2024-03-18T01:35:58.816634+0000","last_became_active":"2024-03-18T01:34:56.657196+0000","last_became_peered":"2024-03-18T01:34:56.657196+0000","last_unstale":"2024-03-18T01:35:58.816634+0000","last_undegraded":"2024-03-18T01:35:58.816634+0000","last_fullsized":"2024-03-18T01:35:58.816634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:12:36.265949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811177+0000","last_change":"2024-03-18T01:35:01.150431+0000","last_active":"2024-03-18T01:35:58.811177+0000","last_peered":"2024-03-18T01:35:58.811177+0000","last_clean":"2024-03-18T01:35:58.811177+0000","last_became_active":"2024-03-18T01:35:01.149981+0000","last_became_peered":"2024-03-18T01:35:01.149981+0000","last_unstale":"2024-03-18T01:35:58.811177+0000","last_undegraded":"2024-03-18T01:35:58.811177+0000","last_fullsized":"2024-03-18T01:35:58.811177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:34:55.371860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806989+0000","last_change":"2024-03-18T01:34:55.649346+0000","last_active":"2024-03-18T01:35:58.806989+0000","last_peered":"2024-03-18T01:35:58.806989+0000","last_clean":"2024-03-18T01:35:58.806989+0000","last_became_active":"2024-03-18T01:34:55.648958+0000","last_became_peered":"2024-03-18T01:34:55.648958+0000","last_unstale":"2024-03-18T01:35:58.806989+0000","last_undegraded":"2024-03-18T01:35:58.806989+0000","last_fullsized":"2024-03-18T01:35:58.806989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:53.788081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809672+0000","last_change":"2024-03-18T01:34:56.653525+0000","last_active":"2024-03-18T01:35:58.809672+0000","last_peered":"2024-03-18T01:35:58.809672+0000","last_clean":"2024-03-18T01:35:58.809672+0000","last_became_active":"2024-03-18T01:34:56.653218+0000","last_became_peered":"2024-03-18T01:34:56.653218+0000","last_unstale":"2024-03-18T01:35:58.809672+0000","last_undegraded":"2024-03-18T01:35:58.809672+0000","last_fullsized":"2024-03-18T01:35:58.809672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:32:37.479782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816301+0000","last_change":"2024-03-18T01:35:01.135505+0000","last_active":"2024-03-18T01:35:58.816301+0000","last_peered":"2024-03-18T01:35:58.816301+0000","last_clean":"2024-03-18T01:35:58.816301+0000","last_became_active":"2024-03-18T01:35:01.135055+0000","last_became_peered":"2024-03-18T01:35:01.135055+0000","last_unstale":"2024-03-18T01:35:58.816301+0000","last_undegraded":"2024-03-18T01:35:58.816301+0000","last_fullsized":"2024-03-18T01:35:58.816301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:52:35.069313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":197,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:52.730173+0000","last_change":"2024-03-18T01:34:55.659389+0000","last_active":"2024-03-18T01:36:52.730173+0000","last_peered":"2024-03-18T01:36:52.730173+0000","last_clean":"2024-03-18T01:36:52.730173+0000","last_became_active":"2024-03-18T01:34:55.659090+0000","last_became_peered":"2024-03-18T01:34:55.659090+0000","last_unstale":"2024-03-18T01:36:52.730173+0000","last_undegraded":"2024-03-18T01:36:52.730173+0000","last_fullsized":"2024-03-18T01:36:52.730173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:55:51.731788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":166,"num_read_kb":166,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807631+0000","last_change":"2024-03-18T01:34:56.650411+0000","last_active":"2024-03-18T01:35:58.807631+0000","last_peered":"2024-03-18T01:35:58.807631+0000","last_clean":"2024-03-18T01:35:58.807631+0000","last_became_active":"2024-03-18T01:34:56.650036+0000","last_became_peered":"2024-03-18T01:34:56.650036+0000","last_unstale":"2024-03-18T01:35:58.807631+0000","last_undegraded":"2024-03-18T01:35:58.807631+0000","last_fullsized":"2024-03-18T01:35:58.807631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:59.912207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811599+0000","last_change":"2024-03-18T01:35:58.811137+0000","last_active":"2024-03-18T01:35:58.811599+0000","last_peered":"2024-03-18T01:35:58.811599+0000","last_clean":"2024-03-18T01:35:58.811599+0000","last_became_active":"2024-03-18T01:35:58.810398+0000","last_became_peered":"2024-03-18T01:35:58.810398+0000","last_unstale":"2024-03-18T01:35:58.811599+0000","last_undegraded":"2024-03-18T01:35:58.811599+0000","last_fullsized":"2024-03-18T01:35:58.811599+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:08.271404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807051+0000","last_change":"2024-03-18T01:34:55.647879+0000","last_active":"2024-03-18T01:35:58.807051+0000","last_peered":"2024-03-18T01:35:58.807051+0000","last_clean":"2024-03-18T01:35:58.807051+0000","last_became_active":"2024-03-18T01:34:55.647549+0000","last_became_peered":"2024-03-18T01:34:55.647549+0000","last_unstale":"2024-03-18T01:35:58.807051+0000","last_undegraded":"2024-03-18T01:35:58.807051+0000","last_fullsized":"2024-03-18T01:35:58.807051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:12.261729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.813429+0000","last_change":"2024-03-18T01:35:58.813429+0000","last_active":"2024-03-18T01:35:58.813429+0000","last_peered":"2024-03-18T01:35:58.813429+0000","last_clean":"2024-03-18T01:35:58.813429+0000","last_became_active":"2024-03-18T01:35:58.812659+0000","last_became_peered":"2024-03-18T01:35:58.812659+0000","last_unstale":"2024-03-18T01:35:58.813429+0000","last_undegraded":"2024-03-18T01:35:58.813429+0000","last_fullsized":"2024-03-18T01:35:58.813429+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:22:01.273649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802202+0000","last_change":"2024-03-18T01:35:01.153448+0000","last_active":"2024-03-18T01:35:58.802202+0000","last_peered":"2024-03-18T01:35:58.802202+0000","last_clean":"2024-03-18T01:35:58.802202+0000","last_became_active":"2024-03-18T01:35:01.152593+0000","last_became_peered":"2024-03-18T01:35:01.152593+0000","last_unstale":"2024-03-18T01:35:58.802202+0000","last_undegraded":"2024-03-18T01:35:58.802202+0000","last_fullsized":"2024-03-18T01:35:58.802202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:46.363662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808537+0000","last_change":"2024-03-18T01:34:55.655079+0000","last_active":"2024-03-18T01:35:58.808537+0000","last_peered":"2024-03-18T01:35:58.808537+0000","last_clean":"2024-03-18T01:35:58.808537+0000","last_became_active":"2024-03-18T01:34:55.654702+0000","last_became_peered":"2024-03-18T01:34:55.654702+0000","last_unstale":"2024-03-18T01:35:58.808537+0000","last_undegraded":"2024-03-18T01:35:58.808537+0000","last_fullsized":"2024-03-18T01:35:58.808537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:50:46.524614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807621+0000","last_change":"2024-03-18T01:34:56.644441+0000","last_active":"2024-03-18T01:35:58.807621+0000","last_peered":"2024-03-18T01:35:58.807621+0000","last_clean":"2024-03-18T01:35:58.807621+0000","last_became_active":"2024-03-18T01:34:56.643924+0000","last_became_peered":"2024-03-18T01:34:56.643924+0000","last_unstale":"2024-03-18T01:35:58.807621+0000","last_undegraded":"2024-03-18T01:35:58.807621+0000","last_fullsized":"2024-03-18T01:35:58.807621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:44.260300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809026+0000","last_change":"2024-03-18T01:35:01.150022+0000","last_active":"2024-03-18T01:35:58.809026+0000","last_peered":"2024-03-18T01:35:58.809026+0000","last_clean":"2024-03-18T01:35:58.809026+0000","last_became_active":"2024-03-18T01:35:01.149510+0000","last_became_peered":"2024-03-18T01:35:01.149510+0000","last_unstale":"2024-03-18T01:35:58.809026+0000","last_undegraded":"2024-03-18T01:35:58.809026+0000","last_fullsized":"2024-03-18T01:35:58.809026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:40:36.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'32","reported_seq":230,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:52.730208+0000","last_change":"2024-03-18T01:34:55.642397+0000","last_active":"2024-03-18T01:36:52.730208+0000","last_peered":"2024-03-18T01:36:52.730208+0000","last_clean":"2024-03-18T01:36:52.730208+0000","last_became_active":"2024-03-18T01:34:55.642046+0000","last_became_peered":"2024-03-18T01:34:55.642046+0000","last_unstale":"2024-03-18T01:36:52.730208+0000","last_undegraded":"2024-03-18T01:36:52.730208+0000","last_fullsized":"2024-03-18T01:36:52.730208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:48:31.631414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62072,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":171,"num_read_kb":146,"num_write":34,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802738+0000","last_change":"2024-03-18T01:34:56.644388+0000","last_active":"2024-03-18T01:35:58.802738+0000","last_peered":"2024-03-18T01:35:58.802738+0000","last_clean":"2024-03-18T01:35:58.802738+0000","last_became_active":"2024-03-18T01:34:56.644014+0000","last_became_peered":"2024-03-18T01:34:56.644014+0000","last_unstale":"2024-03-18T01:35:58.802738+0000","last_undegraded":"2024-03-18T01:35:58.802738+0000","last_fullsized":"2024-03-18T01:35:58.802738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:06:18.193600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.823299+0000","last_change":"2024-03-18T01:35:58.822872+0000","last_active":"2024-03-18T01:35:58.823299+0000","last_peered":"2024-03-18T01:35:58.823299+0000","last_clean":"2024-03-18T01:35:58.823299+0000","last_became_active":"2024-03-18T01:35:58.822325+0000","last_became_peered":"2024-03-18T01:35:58.822325+0000","last_unstale":"2024-03-18T01:35:58.823299+0000","last_undegraded":"2024-03-18T01:35:58.823299+0000","last_fullsized":"2024-03-18T01:35:58.823299+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:54:21.392764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.834419+0000","last_change":"2024-03-18T01:35:58.834419+0000","last_active":"2024-03-18T01:35:58.834419+0000","last_peered":"2024-03-18T01:35:58.834419+0000","last_clean":"2024-03-18T01:35:58.834419+0000","last_became_active":"2024-03-18T01:35:58.822930+0000","last_became_peered":"2024-03-18T01:35:58.822930+0000","last_unstale":"2024-03-18T01:35:58.834419+0000","last_undegraded":"2024-03-18T01:35:58.834419+0000","last_fullsized":"2024-03-18T01:35:58.834419+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:15.777787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.818885+0000","last_change":"2024-03-18T01:34:56.657786+0000","last_active":"2024-03-18T01:35:58.818885+0000","last_peered":"2024-03-18T01:35:58.818885+0000","last_clean":"2024-03-18T01:35:58.818885+0000","last_became_active":"2024-03-18T01:34:56.657432+0000","last_became_peered":"2024-03-18T01:34:56.657432+0000","last_unstale":"2024-03-18T01:35:58.818885+0000","last_undegraded":"2024-03-18T01:35:58.818885+0000","last_fullsized":"2024-03-18T01:35:58.818885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:13:39.498562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804914+0000","last_change":"2024-03-18T01:35:01.150495+0000","last_active":"2024-03-18T01:35:58.804914+0000","last_peered":"2024-03-18T01:35:58.804914+0000","last_clean":"2024-03-18T01:35:58.804914+0000","last_became_active":"2024-03-18T01:35:01.150033+0000","last_became_peered":"2024-03-18T01:35:01.150033+0000","last_unstale":"2024-03-18T01:35:58.804914+0000","last_undegraded":"2024-03-18T01:35:58.804914+0000","last_fullsized":"2024-03-18T01:35:58.804914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:38.433827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807787+0000","last_change":"2024-03-18T01:34:55.658820+0000","last_active":"2024-03-18T01:35:58.807787+0000","last_peered":"2024-03-18T01:35:58.807787+0000","last_clean":"2024-03-18T01:35:58.807787+0000","last_became_active":"2024-03-18T01:34:55.658446+0000","last_became_peered":"2024-03-18T01:34:55.658446+0000","last_unstale":"2024-03-18T01:35:58.807787+0000","last_undegraded":"2024-03-18T01:35:58.807787+0000","last_fullsized":"2024-03-18T01:35:58.807787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:19:21.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806325+0000","last_change":"2024-03-18T01:34:56.652890+0000","last_active":"2024-03-18T01:35:58.806325+0000","last_peered":"2024-03-18T01:35:58.806325+0000","last_clean":"2024-03-18T01:35:58.806325+0000","last_became_active":"2024-03-18T01:34:56.652444+0000","last_became_peered":"2024-03-18T01:34:56.652444+0000","last_unstale":"2024-03-18T01:35:58.806325+0000","last_undegraded":"2024-03-18T01:35:58.806325+0000","last_fullsized":"2024-03-18T01:35:58.806325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:58:47.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812325+0000","last_change":"2024-03-18T01:35:01.154700+0000","last_active":"2024-03-18T01:35:58.812325+0000","last_peered":"2024-03-18T01:35:58.812325+0000","last_clean":"2024-03-18T01:35:58.812325+0000","last_became_active":"2024-03-18T01:35:01.148453+0000","last_became_peered":"2024-03-18T01:35:01.148453+0000","last_unstale":"2024-03-18T01:35:58.812325+0000","last_undegraded":"2024-03-18T01:35:58.812325+0000","last_fullsized":"2024-03-18T01:35:58.812325+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:08:40.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:53.805945+0000","last_change":"2024-03-18T01:34:55.643861+0000","last_active":"2024-03-18T01:36:53.805945+0000","last_peered":"2024-03-18T01:36:53.805945+0000","last_clean":"2024-03-18T01:36:53.805945+0000","last_became_active":"2024-03-18T01:34:55.643352+0000","last_became_peered":"2024-03-18T01:34:55.643352+0000","last_unstale":"2024-03-18T01:36:53.805945+0000","last_undegraded":"2024-03-18T01:36:53.805945+0000","last_fullsized":"2024-03-18T01:36:53.805945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:56:53.685571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.814042+0000","last_change":"2024-03-18T01:35:58.814042+0000","last_active":"2024-03-18T01:35:58.814042+0000","last_peered":"2024-03-18T01:35:58.814042+0000","last_clean":"2024-03-18T01:35:58.814042+0000","last_became_active":"2024-03-18T01:35:58.813622+0000","last_became_peered":"2024-03-18T01:35:58.813622+0000","last_unstale":"2024-03-18T01:35:58.814042+0000","last_undegraded":"2024-03-18T01:35:58.814042+0000","last_fullsized":"2024-03-18T01:35:58.814042+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:52.815608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.824798+0000","last_change":"2024-03-18T01:35:58.824798+0000","last_active":"2024-03-18T01:35:58.824798+0000","last_peered":"2024-03-18T01:35:58.824798+0000","last_clean":"2024-03-18T01:35:58.824798+0000","last_became_active":"2024-03-18T01:35:58.823278+0000","last_became_peered":"2024-03-18T01:35:58.823278+0000","last_unstale":"2024-03-18T01:35:58.824798+0000","last_undegraded":"2024-03-18T01:35:58.824798+0000","last_fullsized":"2024-03-18T01:35:58.824798+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:18:53.871541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801920+0000","last_change":"2024-03-18T01:34:55.658245+0000","last_active":"2024-03-18T01:35:58.801920+0000","last_peered":"2024-03-18T01:35:58.801920+0000","last_clean":"2024-03-18T01:35:58.801920+0000","last_became_active":"2024-03-18T01:34:55.657916+0000","last_became_peered":"2024-03-18T01:34:55.657916+0000","last_unstale":"2024-03-18T01:35:58.801920+0000","last_undegraded":"2024-03-18T01:35:58.801920+0000","last_fullsized":"2024-03-18T01:35:58.801920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:18:49.207721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807666+0000","last_change":"2024-03-18T01:34:56.648646+0000","last_active":"2024-03-18T01:35:58.807666+0000","last_peered":"2024-03-18T01:35:58.807666+0000","last_clean":"2024-03-18T01:35:58.807666+0000","last_became_active":"2024-03-18T01:34:56.648257+0000","last_became_peered":"2024-03-18T01:34:56.648257+0000","last_unstale":"2024-03-18T01:35:58.807666+0000","last_undegraded":"2024-03-18T01:35:58.807666+0000","last_fullsized":"2024-03-18T01:35:58.807666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:53.788445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807562+0000","last_change":"2024-03-18T01:35:01.136912+0000","last_active":"2024-03-18T01:35:58.807562+0000","last_peered":"2024-03-18T01:35:58.807562+0000","last_clean":"2024-03-18T01:35:58.807562+0000","last_became_active":"2024-03-18T01:35:01.136611+0000","last_became_peered":"2024-03-18T01:35:01.136611+0000","last_unstale":"2024-03-18T01:35:58.807562+0000","last_undegraded":"2024-03-18T01:35:58.807562+0000","last_fullsized":"2024-03-18T01:35:58.807562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:31.089605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802505+0000","last_change":"2024-03-18T01:34:55.648345+0000","last_active":"2024-03-18T01:35:58.802505+0000","last_peered":"2024-03-18T01:35:58.802505+0000","last_clean":"2024-03-18T01:35:58.802505+0000","last_became_active":"2024-03-18T01:34:55.647113+0000","last_became_peered":"2024-03-18T01:34:55.647113+0000","last_unstale":"2024-03-18T01:35:58.802505+0000","last_undegraded":"2024-03-18T01:35:58.802505+0000","last_fullsized":"2024-03-18T01:35:58.802505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:20.904129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804788+0000","last_change":"2024-03-18T01:34:56.649147+0000","last_active":"2024-03-18T01:35:58.804788+0000","last_peered":"2024-03-18T01:35:58.804788+0000","last_clean":"2024-03-18T01:35:58.804788+0000","last_became_active":"2024-03-18T01:34:56.648815+0000","last_became_peered":"2024-03-18T01:34:56.648815+0000","last_unstale":"2024-03-18T01:35:58.804788+0000","last_undegraded":"2024-03-18T01:35:58.804788+0000","last_fullsized":"2024-03-18T01:35:58.804788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:53.560679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802008+0000","last_change":"2024-03-18T01:35:01.156778+0000","last_active":"2024-03-18T01:35:58.802008+0000","last_peered":"2024-03-18T01:35:58.802008+0000","last_clean":"2024-03-18T01:35:58.802008+0000","last_became_active":"2024-03-18T01:35:01.156367+0000","last_became_peered":"2024-03-18T01:35:01.156367+0000","last_unstale":"2024-03-18T01:35:58.802008+0000","last_undegraded":"2024-03-18T01:35:58.802008+0000","last_fullsized":"2024-03-18T01:35:58.802008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:46.223915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810814+0000","last_change":"2024-03-18T01:34:55.658722+0000","last_active":"2024-03-18T01:35:58.810814+0000","last_peered":"2024-03-18T01:35:58.810814+0000","last_clean":"2024-03-18T01:35:58.810814+0000","last_became_active":"2024-03-18T01:34:55.658321+0000","last_became_peered":"2024-03-18T01:34:55.658321+0000","last_unstale":"2024-03-18T01:35:58.810814+0000","last_undegraded":"2024-03-18T01:35:58.810814+0000","last_fullsized":"2024-03-18T01:35:58.810814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:05.753226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804686+0000","last_change":"2024-03-18T01:34:56.649415+0000","last_active":"2024-03-18T01:35:58.804686+0000","last_peered":"2024-03-18T01:35:58.804686+0000","last_clean":"2024-03-18T01:35:58.804686+0000","last_became_active":"2024-03-18T01:34:56.649019+0000","last_became_peered":"2024-03-18T01:34:56.649019+0000","last_unstale":"2024-03-18T01:35:58.804686+0000","last_undegraded":"2024-03-18T01:35:58.804686+0000","last_fullsized":"2024-03-18T01:35:58.804686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:28.473683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819084+0000","last_change":"2024-03-18T01:35:01.148937+0000","last_active":"2024-03-18T01:35:58.819084+0000","last_peered":"2024-03-18T01:35:58.819084+0000","last_clean":"2024-03-18T01:35:58.819084+0000","last_became_active":"2024-03-18T01:35:01.148415+0000","last_became_peered":"2024-03-18T01:35:01.148415+0000","last_unstale":"2024-03-18T01:35:58.819084+0000","last_undegraded":"2024-03-18T01:35:58.819084+0000","last_fullsized":"2024-03-18T01:35:58.819084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:45.926970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:54.772891+0000","last_change":"2024-03-18T01:34:55.653443+0000","last_active":"2024-03-18T01:36:54.772891+0000","last_peered":"2024-03-18T01:36:54.772891+0000","last_clean":"2024-03-18T01:36:54.772891+0000","last_became_active":"2024-03-18T01:34:55.653072+0000","last_became_peered":"2024-03-18T01:34:55.653072+0000","last_unstale":"2024-03-18T01:36:54.772891+0000","last_undegraded":"2024-03-18T01:36:54.772891+0000","last_fullsized":"2024-03-18T01:36:54.772891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:26.376636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811178+0000","last_change":"2024-03-18T01:35:58.811178+0000","last_active":"2024-03-18T01:35:58.811178+0000","last_peered":"2024-03-18T01:35:58.811178+0000","last_clean":"2024-03-18T01:35:58.811178+0000","last_became_active":"2024-03-18T01:35:58.810764+0000","last_became_peered":"2024-03-18T01:35:58.810764+0000","last_unstale":"2024-03-18T01:35:58.811178+0000","last_undegraded":"2024-03-18T01:35:58.811178+0000","last_fullsized":"2024-03-18T01:35:58.811178+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:49.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819415+0000","last_change":"2024-03-18T01:35:01.152767+0000","last_active":"2024-03-18T01:35:58.819415+0000","last_peered":"2024-03-18T01:35:58.819415+0000","last_clean":"2024-03-18T01:35:58.819415+0000","last_became_active":"2024-03-18T01:35:01.150619+0000","last_became_peered":"2024-03-18T01:35:01.150619+0000","last_unstale":"2024-03-18T01:35:58.819415+0000","last_undegraded":"2024-03-18T01:35:58.819415+0000","last_fullsized":"2024-03-18T01:35:58.819415+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:57:28.258251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807082+0000","last_change":"2024-03-18T01:34:56.650541+0000","last_active":"2024-03-18T01:35:58.807082+0000","last_peered":"2024-03-18T01:35:58.807082+0000","last_clean":"2024-03-18T01:35:58.807082+0000","last_became_active":"2024-03-18T01:34:56.650072+0000","last_became_peered":"2024-03-18T01:34:56.650072+0000","last_unstale":"2024-03-18T01:35:58.807082+0000","last_undegraded":"2024-03-18T01:35:58.807082+0000","last_fullsized":"2024-03-18T01:35:58.807082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:51.224883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":117,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:52.804051+0000","last_change":"2024-03-18T01:34:55.664297+0000","last_active":"2024-03-18T01:36:52.804051+0000","last_peered":"2024-03-18T01:36:52.804051+0000","last_clean":"2024-03-18T01:36:52.804051+0000","last_became_active":"2024-03-18T01:34:55.663938+0000","last_became_peered":"2024-03-18T01:34:55.663938+0000","last_unstale":"2024-03-18T01:36:52.804051+0000","last_undegraded":"2024-03-18T01:36:52.804051+0000","last_fullsized":"2024-03-18T01:36:52.804051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:51.771987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806662+0000","last_change":"2024-03-18T01:35:01.141736+0000","last_active":"2024-03-18T01:35:58.806662+0000","last_peered":"2024-03-18T01:35:58.806662+0000","last_clean":"2024-03-18T01:35:58.806662+0000","last_became_active":"2024-03-18T01:35:01.141381+0000","last_became_peered":"2024-03-18T01:35:01.141381+0000","last_unstale":"2024-03-18T01:35:58.806662+0000","last_undegraded":"2024-03-18T01:35:58.806662+0000","last_fullsized":"2024-03-18T01:35:58.806662+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:25:04.190250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.815962+0000","last_change":"2024-03-18T01:34:56.666439+0000","last_active":"2024-03-18T01:35:58.815962+0000","last_peered":"2024-03-18T01:35:58.815962+0000","last_clean":"2024-03-18T01:35:58.815962+0000","last_became_active":"2024-03-18T01:34:56.664556+0000","last_became_peered":"2024-03-18T01:34:56.664556+0000","last_unstale":"2024-03-18T01:35:58.815962+0000","last_undegraded":"2024-03-18T01:35:58.815962+0000","last_fullsized":"2024-03-18T01:35:58.815962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:08:48.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802117+0000","last_change":"2024-03-18T01:34:55.656484+0000","last_active":"2024-03-18T01:35:58.802117+0000","last_peered":"2024-03-18T01:35:58.802117+0000","last_clean":"2024-03-18T01:35:58.802117+0000","last_became_active":"2024-03-18T01:34:55.656125+0000","last_became_peered":"2024-03-18T01:34:55.656125+0000","last_unstale":"2024-03-18T01:35:58.802117+0000","last_undegraded":"2024-03-18T01:35:58.802117+0000","last_fullsized":"2024-03-18T01:35:58.802117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:58:25.944403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806027+0000","last_change":"2024-03-18T01:33:35.824651+0000","last_active":"2024-03-18T01:35:58.806027+0000","last_peered":"2024-03-18T01:35:58.806027+0000","last_clean":"2024-03-18T01:35:58.806027+0000","last_became_active":"2024-03-18T01:33:35.824106+0000","last_became_peered":"2024-03-18T01:33:35.824106+0000","last_unstale":"2024-03-18T01:35:58.806027+0000","last_undegraded":"2024-03-18T01:35:58.806027+0000","last_fullsized":"2024-03-18T01:35:58.806027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_clean_scrub_stamp":"2024-03-18T01:29:41.724134+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:06.422316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808469+0000","last_change":"2024-03-18T01:35:01.157735+0000","last_active":"2024-03-18T01:35:58.808469+0000","last_peered":"2024-03-18T01:35:58.808469+0000","last_clean":"2024-03-18T01:35:58.808469+0000","last_became_active":"2024-03-18T01:35:01.156954+0000","last_became_peered":"2024-03-18T01:35:01.156954+0000","last_unstale":"2024-03-18T01:35:58.808469+0000","last_undegraded":"2024-03-18T01:35:58.808469+0000","last_fullsized":"2024-03-18T01:35:58.808469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:00.106809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.818562+0000","last_change":"2024-03-18T01:34:56.646808+0000","last_active":"2024-03-18T01:35:58.818562+0000","last_peered":"2024-03-18T01:35:58.818562+0000","last_clean":"2024-03-18T01:35:58.818562+0000","last_became_active":"2024-03-18T01:34:56.646437+0000","last_became_peered":"2024-03-18T01:34:56.646437+0000","last_unstale":"2024-03-18T01:35:58.818562+0000","last_undegraded":"2024-03-18T01:35:58.818562+0000","last_fullsized":"2024-03-18T01:35:58.818562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:46:09.703889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806136+0000","last_change":"2024-03-18T01:34:55.645365+0000","last_active":"2024-03-18T01:35:58.806136+0000","last_peered":"2024-03-18T01:35:58.806136+0000","last_clean":"2024-03-18T01:35:58.806136+0000","last_became_active":"2024-03-18T01:34:55.645018+0000","last_became_peered":"2024-03-18T01:34:55.645018+0000","last_unstale":"2024-03-18T01:35:58.806136+0000","last_undegraded":"2024-03-18T01:35:58.806136+0000","last_fullsized":"2024-03-18T01:35:58.806136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:08:31.869328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804906+0000","last_change":"2024-03-18T01:35:01.147294+0000","last_active":"2024-03-18T01:35:58.804906+0000","last_peered":"2024-03-18T01:35:58.804906+0000","last_clean":"2024-03-18T01:35:58.804906+0000","last_became_active":"2024-03-18T01:35:01.146858+0000","last_became_peered":"2024-03-18T01:35:01.146858+0000","last_unstale":"2024-03-18T01:35:58.804906+0000","last_undegraded":"2024-03-18T01:35:58.804906+0000","last_fullsized":"2024-03-18T01:35:58.804906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:45:24.010746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802536+0000","last_change":"2024-03-18T01:34:55.658110+0000","last_active":"2024-03-18T01:35:58.802536+0000","last_peered":"2024-03-18T01:35:58.802536+0000","last_clean":"2024-03-18T01:35:58.802536+0000","last_became_active":"2024-03-18T01:34:55.657802+0000","last_became_peered":"2024-03-18T01:34:55.657802+0000","last_unstale":"2024-03-18T01:35:58.802536+0000","last_undegraded":"2024-03-18T01:35:58.802536+0000","last_fullsized":"2024-03-18T01:35:58.802536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:34.264115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819895+0000","last_change":"2024-03-18T01:34:56.656689+0000","last_active":"2024-03-18T01:35:58.819895+0000","last_peered":"2024-03-18T01:35:58.819895+0000","last_clean":"2024-03-18T01:35:58.819895+0000","last_became_active":"2024-03-18T01:34:56.655653+0000","last_became_peered":"2024-03-18T01:34:56.655653+0000","last_unstale":"2024-03-18T01:35:58.819895+0000","last_undegraded":"2024-03-18T01:35:58.819895+0000","last_fullsized":"2024-03-18T01:35:58.819895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:58.370904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816471+0000","last_change":"2024-03-18T01:35:01.137419+0000","last_active":"2024-03-18T01:35:58.816471+0000","last_peered":"2024-03-18T01:35:58.816471+0000","last_clean":"2024-03-18T01:35:58.816471+0000","last_became_active":"2024-03-18T01:35:01.137033+0000","last_became_peered":"2024-03-18T01:35:01.137033+0000","last_unstale":"2024-03-18T01:35:58.816471+0000","last_undegraded":"2024-03-18T01:35:58.816471+0000","last_fullsized":"2024-03-18T01:35:58.816471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:13:48.382329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'13","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811266+0000","last_change":"2024-03-18T01:34:55.663134+0000","last_active":"2024-03-18T01:35:58.811266+0000","last_peered":"2024-03-18T01:35:58.811266+0000","last_clean":"2024-03-18T01:35:58.811266+0000","last_became_active":"2024-03-18T01:34:55.662083+0000","last_became_peered":"2024-03-18T01:34:55.662083+0000","last_unstale":"2024-03-18T01:35:58.811266+0000","last_undegraded":"2024-03-18T01:35:58.811266+0000","last_fullsized":"2024-03-18T01:35:58.811266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:01:09.186599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821690+0000","last_change":"2024-03-18T01:34:56.664761+0000","last_active":"2024-03-18T01:35:58.821690+0000","last_peered":"2024-03-18T01:35:58.821690+0000","last_clean":"2024-03-18T01:35:58.821690+0000","last_became_active":"2024-03-18T01:34:56.663657+0000","last_became_peered":"2024-03-18T01:34:56.663657+0000","last_unstale":"2024-03-18T01:35:58.821690+0000","last_undegraded":"2024-03-18T01:35:58.821690+0000","last_fullsized":"2024-03-18T01:35:58.821690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:26:53.075000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809944+0000","last_change":"2024-03-18T01:35:01.133002+0000","last_active":"2024-03-18T01:35:58.809944+0000","last_peered":"2024-03-18T01:35:58.809944+0000","last_clean":"2024-03-18T01:35:58.809944+0000","last_became_active":"2024-03-18T01:35:01.132542+0000","last_became_peered":"2024-03-18T01:35:01.132542+0000","last_unstale":"2024-03-18T01:35:58.809944+0000","last_undegraded":"2024-03-18T01:35:58.809944+0000","last_fullsized":"2024-03-18T01:35:58.809944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:23.823409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801907+0000","last_change":"2024-03-18T01:34:55.648924+0000","last_active":"2024-03-18T01:35:58.801907+0000","last_peered":"2024-03-18T01:35:58.801907+0000","last_clean":"2024-03-18T01:35:58.801907+0000","last_became_active":"2024-03-18T01:34:55.648548+0000","last_became_peered":"2024-03-18T01:34:55.648548+0000","last_unstale":"2024-03-18T01:35:58.801907+0000","last_undegraded":"2024-03-18T01:35:58.801907+0000","last_fullsized":"2024-03-18T01:35:58.801907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:26:38.916856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811550+0000","last_change":"2024-03-18T01:34:56.663142+0000","last_active":"2024-03-18T01:35:58.811550+0000","last_peered":"2024-03-18T01:35:58.811550+0000","last_clean":"2024-03-18T01:35:58.811550+0000","last_became_active":"2024-03-18T01:34:56.660256+0000","last_became_peered":"2024-03-18T01:34:56.660256+0000","last_unstale":"2024-03-18T01:35:58.811550+0000","last_undegraded":"2024-03-18T01:35:58.811550+0000","last_fullsized":"2024-03-18T01:35:58.811550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:59.503277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801675+0000","last_change":"2024-03-18T01:35:01.136186+0000","last_active":"2024-03-18T01:35:58.801675+0000","last_peered":"2024-03-18T01:35:58.801675+0000","last_clean":"2024-03-18T01:35:58.801675+0000","last_became_active":"2024-03-18T01:35:01.135803+0000","last_became_peered":"2024-03-18T01:35:01.135803+0000","last_unstale":"2024-03-18T01:35:58.801675+0000","last_undegraded":"2024-03-18T01:35:58.801675+0000","last_fullsized":"2024-03-18T01:35:58.801675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:20.027448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806421+0000","last_change":"2024-03-18T01:34:55.663090+0000","last_active":"2024-03-18T01:35:58.806421+0000","last_peered":"2024-03-18T01:35:58.806421+0000","last_clean":"2024-03-18T01:35:58.806421+0000","last_became_active":"2024-03-18T01:34:55.662774+0000","last_became_peered":"2024-03-18T01:34:55.662774+0000","last_unstale":"2024-03-18T01:35:58.806421+0000","last_undegraded":"2024-03-18T01:35:58.806421+0000","last_fullsized":"2024-03-18T01:35:58.806421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:30:35.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801874+0000","last_change":"2024-03-18T01:34:56.639509+0000","last_active":"2024-03-18T01:35:58.801874+0000","last_peered":"2024-03-18T01:35:58.801874+0000","last_clean":"2024-03-18T01:35:58.801874+0000","last_became_active":"2024-03-18T01:34:56.639144+0000","last_became_peered":"2024-03-18T01:34:56.639144+0000","last_unstale":"2024-03-18T01:35:58.801874+0000","last_undegraded":"2024-03-18T01:35:58.801874+0000","last_fullsized":"2024-03-18T01:35:58.801874+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:12:36.220793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812718+0000","last_change":"2024-03-18T01:35:01.141762+0000","last_active":"2024-03-18T01:35:58.812718+0000","last_peered":"2024-03-18T01:35:58.812718+0000","last_clean":"2024-03-18T01:35:58.812718+0000","last_became_active":"2024-03-18T01:35:01.141219+0000","last_became_peered":"2024-03-18T01:35:01.141219+0000","last_unstale":"2024-03-18T01:35:58.812718+0000","last_undegraded":"2024-03-18T01:35:58.812718+0000","last_fullsized":"2024-03-18T01:35:58.812718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:55.839120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.815857+0000","last_change":"2024-03-18T01:34:55.662530+0000","last_active":"2024-03-18T01:35:58.815857+0000","last_peered":"2024-03-18T01:35:58.815857+0000","last_clean":"2024-03-18T01:35:58.815857+0000","last_became_active":"2024-03-18T01:34:55.661178+0000","last_became_peered":"2024-03-18T01:34:55.661178+0000","last_unstale":"2024-03-18T01:35:58.815857+0000","last_undegraded":"2024-03-18T01:35:58.815857+0000","last_fullsized":"2024-03-18T01:35:58.815857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:47:16.678553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804373+0000","last_change":"2024-03-18T01:34:56.660936+0000","last_active":"2024-03-18T01:35:58.804373+0000","last_peered":"2024-03-18T01:35:58.804373+0000","last_clean":"2024-03-18T01:35:58.804373+0000","last_became_active":"2024-03-18T01:34:56.660513+0000","last_became_peered":"2024-03-18T01:34:56.660513+0000","last_unstale":"2024-03-18T01:35:58.804373+0000","last_undegraded":"2024-03-18T01:35:58.804373+0000","last_fullsized":"2024-03-18T01:35:58.804373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:35:21.037366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802003+0000","last_change":"2024-03-18T01:35:01.136914+0000","last_active":"2024-03-18T01:35:58.802003+0000","last_peered":"2024-03-18T01:35:58.802003+0000","last_clean":"2024-03-18T01:35:58.802003+0000","last_became_active":"2024-03-18T01:35:01.136480+0000","last_became_peered":"2024-03-18T01:35:01.136480+0000","last_unstale":"2024-03-18T01:35:58.802003+0000","last_undegraded":"2024-03-18T01:35:58.802003+0000","last_fullsized":"2024-03-18T01:35:58.802003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:49.262255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809429+0000","last_change":"2024-03-18T01:34:55.654938+0000","last_active":"2024-03-18T01:35:58.809429+0000","last_peered":"2024-03-18T01:35:58.809429+0000","last_clean":"2024-03-18T01:35:58.809429+0000","last_became_active":"2024-03-18T01:34:55.654611+0000","last_became_peered":"2024-03-18T01:34:55.654611+0000","last_unstale":"2024-03-18T01:35:58.809429+0000","last_undegraded":"2024-03-18T01:35:58.809429+0000","last_fullsized":"2024-03-18T01:35:58.809429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:23.210205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821426+0000","last_change":"2024-03-18T01:34:56.650731+0000","last_active":"2024-03-18T01:35:58.821426+0000","last_peered":"2024-03-18T01:35:58.821426+0000","last_clean":"2024-03-18T01:35:58.821426+0000","last_became_active":"2024-03-18T01:34:56.650363+0000","last_became_peered":"2024-03-18T01:34:56.650363+0000","last_unstale":"2024-03-18T01:35:58.821426+0000","last_undegraded":"2024-03-18T01:35:58.821426+0000","last_fullsized":"2024-03-18T01:35:58.821426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:09:50.305019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801649+0000","last_change":"2024-03-18T01:35:01.135101+0000","last_active":"2024-03-18T01:35:58.801649+0000","last_peered":"2024-03-18T01:35:58.801649+0000","last_clean":"2024-03-18T01:35:58.801649+0000","last_became_active":"2024-03-18T01:35:01.134690+0000","last_became_peered":"2024-03-18T01:35:01.134690+0000","last_unstale":"2024-03-18T01:35:58.801649+0000","last_undegraded":"2024-03-18T01:35:58.801649+0000","last_fullsized":"2024-03-18T01:35:58.801649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:33.381629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802112+0000","last_change":"2024-03-18T01:34:55.648108+0000","last_active":"2024-03-18T01:35:58.802112+0000","last_peered":"2024-03-18T01:35:58.802112+0000","last_clean":"2024-03-18T01:35:58.802112+0000","last_became_active":"2024-03-18T01:34:55.647754+0000","last_became_peered":"2024-03-18T01:34:55.647754+0000","last_unstale":"2024-03-18T01:35:58.802112+0000","last_undegraded":"2024-03-18T01:35:58.802112+0000","last_fullsized":"2024-03-18T01:35:58.802112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:05:48.508015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806678+0000","last_change":"2024-03-18T01:34:56.651589+0000","last_active":"2024-03-18T01:35:58.806678+0000","last_peered":"2024-03-18T01:35:58.806678+0000","last_clean":"2024-03-18T01:35:58.806678+0000","last_became_active":"2024-03-18T01:34:56.650929+0000","last_became_peered":"2024-03-18T01:34:56.650929+0000","last_unstale":"2024-03-18T01:35:58.806678+0000","last_undegraded":"2024-03-18T01:35:58.806678+0000","last_fullsized":"2024-03-18T01:35:58.806678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:43.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806777+0000","last_change":"2024-03-18T01:35:01.148347+0000","last_active":"2024-03-18T01:35:58.806777+0000","last_peered":"2024-03-18T01:35:58.806777+0000","last_clean":"2024-03-18T01:35:58.806777+0000","last_became_active":"2024-03-18T01:35:01.147936+0000","last_became_peered":"2024-03-18T01:35:01.147936+0000","last_unstale":"2024-03-18T01:35:58.806777+0000","last_undegraded":"2024-03-18T01:35:58.806777+0000","last_fullsized":"2024-03-18T01:35:58.806777+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:12.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:54.733145+0000","last_change":"2024-03-18T01:34:55.653625+0000","last_active":"2024-03-18T01:36:54.733145+0000","last_peered":"2024-03-18T01:36:54.733145+0000","last_clean":"2024-03-18T01:36:54.733145+0000","last_became_active":"2024-03-18T01:34:55.653186+0000","last_became_peered":"2024-03-18T01:34:55.653186+0000","last_unstale":"2024-03-18T01:36:54.733145+0000","last_undegraded":"2024-03-18T01:36:54.733145+0000","last_fullsized":"2024-03-18T01:36:54.733145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:10:51.193296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806920+0000","last_change":"2024-03-18T01:34:56.652703+0000","last_active":"2024-03-18T01:35:58.806920+0000","last_peered":"2024-03-18T01:35:58.806920+0000","last_clean":"2024-03-18T01:35:58.806920+0000","last_became_active":"2024-03-18T01:34:56.651768+0000","last_became_peered":"2024-03-18T01:34:56.651768+0000","last_unstale":"2024-03-18T01:35:58.806920+0000","last_undegraded":"2024-03-18T01:35:58.806920+0000","last_fullsized":"2024-03-18T01:35:58.806920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:59.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809291+0000","last_change":"2024-03-18T01:35:01.144900+0000","last_active":"2024-03-18T01:35:58.809291+0000","last_peered":"2024-03-18T01:35:58.809291+0000","last_clean":"2024-03-18T01:35:58.809291+0000","last_became_active":"2024-03-18T01:35:01.144431+0000","last_became_peered":"2024-03-18T01:35:01.144431+0000","last_unstale":"2024-03-18T01:35:58.809291+0000","last_undegraded":"2024-03-18T01:35:58.809291+0000","last_fullsized":"2024-03-18T01:35:58.809291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:36.199904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808737+0000","last_change":"2024-03-18T01:34:55.661259+0000","last_active":"2024-03-18T01:35:58.808737+0000","last_peered":"2024-03-18T01:35:58.808737+0000","last_clean":"2024-03-18T01:35:58.808737+0000","last_became_active":"2024-03-18T01:34:55.660964+0000","last_became_peered":"2024-03-18T01:34:55.660964+0000","last_unstale":"2024-03-18T01:35:58.808737+0000","last_undegraded":"2024-03-18T01:35:58.808737+0000","last_fullsized":"2024-03-18T01:35:58.808737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:14.723905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807058+0000","last_change":"2024-03-18T01:34:56.663075+0000","last_active":"2024-03-18T01:35:58.807058+0000","last_peered":"2024-03-18T01:35:58.807058+0000","last_clean":"2024-03-18T01:35:58.807058+0000","last_became_active":"2024-03-18T01:34:56.662458+0000","last_became_peered":"2024-03-18T01:34:56.662458+0000","last_unstale":"2024-03-18T01:35:58.807058+0000","last_undegraded":"2024-03-18T01:35:58.807058+0000","last_fullsized":"2024-03-18T01:35:58.807058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:31:31.837493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806819+0000","last_change":"2024-03-18T01:35:01.133317+0000","last_active":"2024-03-18T01:35:58.806819+0000","last_peered":"2024-03-18T01:35:58.806819+0000","last_clean":"2024-03-18T01:35:58.806819+0000","last_became_active":"2024-03-18T01:35:01.131798+0000","last_became_peered":"2024-03-18T01:35:01.131798+0000","last_unstale":"2024-03-18T01:35:58.806819+0000","last_undegraded":"2024-03-18T01:35:58.806819+0000","last_fullsized":"2024-03-18T01:35:58.806819+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:55.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819454+0000","last_change":"2024-03-18T01:34:55.660887+0000","last_active":"2024-03-18T01:35:58.819454+0000","last_peered":"2024-03-18T01:35:58.819454+0000","last_clean":"2024-03-18T01:35:58.819454+0000","last_became_active":"2024-03-18T01:34:55.660524+0000","last_became_peered":"2024-03-18T01:34:55.660524+0000","last_unstale":"2024-03-18T01:35:58.819454+0000","last_undegraded":"2024-03-18T01:35:58.819454+0000","last_fullsized":"2024-03-18T01:35:58.819454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:24.816951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807252+0000","last_change":"2024-03-18T01:34:56.654107+0000","last_active":"2024-03-18T01:35:58.807252+0000","last_peered":"2024-03-18T01:35:58.807252+0000","last_clean":"2024-03-18T01:35:58.807252+0000","last_became_active":"2024-03-18T01:34:56.653742+0000","last_became_peered":"2024-03-18T01:34:56.653742+0000","last_unstale":"2024-03-18T01:35:58.807252+0000","last_undegraded":"2024-03-18T01:35:58.807252+0000","last_fullsized":"2024-03-18T01:35:58.807252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:58:46.093589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804481+0000","last_change":"2024-03-18T01:34:56.647818+0000","last_active":"2024-03-18T01:35:58.804481+0000","last_peered":"2024-03-18T01:35:58.804481+0000","last_clean":"2024-03-18T01:35:58.804481+0000","last_became_active":"2024-03-18T01:34:56.647174+0000","last_became_peered":"2024-03-18T01:34:56.647174+0000","last_unstale":"2024-03-18T01:35:58.804481+0000","last_undegraded":"2024-03-18T01:35:58.804481+0000","last_fullsized":"2024-03-18T01:35:58.804481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:55.656083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810324+0000","last_change":"2024-03-18T01:34:56.643221+0000","last_active":"2024-03-18T01:35:58.810324+0000","last_peered":"2024-03-18T01:35:58.810324+0000","last_clean":"2024-03-18T01:35:58.810324+0000","last_became_active":"2024-03-18T01:34:56.642871+0000","last_became_peered":"2024-03-18T01:34:56.642871+0000","last_unstale":"2024-03-18T01:35:58.810324+0000","last_undegraded":"2024-03-18T01:35:58.810324+0000","last_fullsized":"2024-03-18T01:35:58.810324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:23:25.106060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809304+0000","last_change":"2024-03-18T01:34:56.656227+0000","last_active":"2024-03-18T01:35:58.809304+0000","last_peered":"2024-03-18T01:35:58.809304+0000","last_clean":"2024-03-18T01:35:58.809304+0000","last_became_active":"2024-03-18T01:34:56.655859+0000","last_became_peered":"2024-03-18T01:34:56.655859+0000","last_unstale":"2024-03-18T01:35:58.809304+0000","last_undegraded":"2024-03-18T01:35:58.809304+0000","last_fullsized":"2024-03-18T01:35:58.809304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:51:54.424342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809637+0000","last_change":"2024-03-18T01:34:56.664655+0000","last_active":"2024-03-18T01:35:58.809637+0000","last_peered":"2024-03-18T01:35:58.809637+0000","last_clean":"2024-03-18T01:35:58.809637+0000","last_became_active":"2024-03-18T01:34:56.662526+0000","last_became_peered":"2024-03-18T01:34:56.662526+0000","last_unstale":"2024-03-18T01:35:58.809637+0000","last_undegraded":"2024-03-18T01:35:58.809637+0000","last_fullsized":"2024-03-18T01:35:58.809637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:45:06.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.820035+0000","last_change":"2024-03-18T01:34:56.657045+0000","last_active":"2024-03-18T01:35:58.820035+0000","last_peered":"2024-03-18T01:35:58.820035+0000","last_clean":"2024-03-18T01:35:58.820035+0000","last_became_active":"2024-03-18T01:34:56.656339+0000","last_became_peered":"2024-03-18T01:34:56.656339+0000","last_unstale":"2024-03-18T01:35:58.820035+0000","last_undegraded":"2024-03-18T01:35:58.820035+0000","last_fullsized":"2024-03-18T01:35:58.820035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:41.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804793+0000","last_change":"2024-03-18T01:34:56.647626+0000","last_active":"2024-03-18T01:35:58.804793+0000","last_peered":"2024-03-18T01:35:58.804793+0000","last_clean":"2024-03-18T01:35:58.804793+0000","last_became_active":"2024-03-18T01:34:56.647216+0000","last_became_peered":"2024-03-18T01:34:56.647216+0000","last_unstale":"2024-03-18T01:35:58.804793+0000","last_undegraded":"2024-03-18T01:35:58.804793+0000","last_fullsized":"2024-03-18T01:35:58.804793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:12:33.330925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809827+0000","last_change":"2024-03-18T01:34:56.645091+0000","last_active":"2024-03-18T01:35:58.809827+0000","last_peered":"2024-03-18T01:35:58.809827+0000","last_clean":"2024-03-18T01:35:58.809827+0000","last_became_active":"2024-03-18T01:34:56.644725+0000","last_became_peered":"2024-03-18T01:34:56.644725+0000","last_unstale":"2024-03-18T01:35:58.809827+0000","last_undegraded":"2024-03-18T01:35:58.809827+0000","last_fullsized":"2024-03-18T01:35:58.809827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821880+0000","last_change":"2024-03-18T01:34:56.651282+0000","last_active":"2024-03-18T01:35:58.821880+0000","last_peered":"2024-03-18T01:35:58.821880+0000","last_clean":"2024-03-18T01:35:58.821880+0000","last_became_active":"2024-03-18T01:34:56.650872+0000","last_became_peered":"2024-03-18T01:34:56.650872+0000","last_unstale":"2024-03-18T01:35:58.821880+0000","last_undegraded":"2024-03-18T01:35:58.821880+0000","last_fullsized":"2024-03-18T01:35:58.821880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:14:45.106848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807140+0000","last_change":"2024-03-18T01:34:56.651347+0000","last_active":"2024-03-18T01:35:58.807140+0000","last_peered":"2024-03-18T01:35:58.807140+0000","last_clean":"2024-03-18T01:35:58.807140+0000","last_became_active":"2024-03-18T01:34:56.650935+0000","last_became_peered":"2024-03-18T01:34:56.650935+0000","last_unstale":"2024-03-18T01:35:58.807140+0000","last_undegraded":"2024-03-18T01:35:58.807140+0000","last_fullsized":"2024-03-18T01:35:58.807140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:41.849883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816038+0000","last_change":"2024-03-18T01:34:56.667333+0000","last_active":"2024-03-18T01:35:58.816038+0000","last_peered":"2024-03-18T01:35:58.816038+0000","last_clean":"2024-03-18T01:35:58.816038+0000","last_became_active":"2024-03-18T01:34:56.666845+0000","last_became_peered":"2024-03-18T01:34:56.666845+0000","last_unstale":"2024-03-18T01:35:58.816038+0000","last_undegraded":"2024-03-18T01:35:58.816038+0000","last_fullsized":"2024-03-18T01:35:58.816038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:49:23.813455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806990+0000","last_change":"2024-03-18T01:34:56.662677+0000","last_active":"2024-03-18T01:35:58.806990+0000","last_peered":"2024-03-18T01:35:58.806990+0000","last_clean":"2024-03-18T01:35:58.806990+0000","last_became_active":"2024-03-18T01:34:56.662306+0000","last_became_peered":"2024-03-18T01:34:56.662306+0000","last_unstale":"2024-03-18T01:35:58.806990+0000","last_undegraded":"2024-03-18T01:35:58.806990+0000","last_fullsized":"2024-03-18T01:35:58.806990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:53:56.868984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.818853+0000","last_change":"2024-03-18T01:34:56.657721+0000","last_active":"2024-03-18T01:35:58.818853+0000","last_peered":"2024-03-18T01:35:58.818853+0000","last_clean":"2024-03-18T01:35:58.818853+0000","last_became_active":"2024-03-18T01:34:56.657072+0000","last_became_peered":"2024-03-18T01:34:56.657072+0000","last_unstale":"2024-03-18T01:35:58.818853+0000","last_undegraded":"2024-03-18T01:35:58.818853+0000","last_fullsized":"2024-03-18T01:35:58.818853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:21.120391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.805065+0000","last_change":"2024-03-18T01:34:56.647327+0000","last_active":"2024-03-18T01:35:58.805065+0000","last_peered":"2024-03-18T01:35:58.805065+0000","last_clean":"2024-03-18T01:35:58.805065+0000","last_became_active":"2024-03-18T01:34:56.646910+0000","last_became_peered":"2024-03-18T01:34:56.646910+0000","last_unstale":"2024-03-18T01:35:58.805065+0000","last_undegraded":"2024-03-18T01:35:58.805065+0000","last_fullsized":"2024-03-18T01:35:58.805065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:37:25.970272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68885,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":877,"num_read_kb":511,"num_write":241,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":502083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1724416,"data_stored":1150712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":501159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1098238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1097728,"data_stored":559969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1110016,"data_stored":560121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":501332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32804,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712348,"statfs":{"total":95995035648,"available":95961444352,"internally_reserved":0,"allocated":1658880,"data_stored":1093429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":63976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":64199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:36:59.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T01:36:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:58 smithi203 ceph-mon[36345]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:36:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:58 smithi203 ceph-mon[36345]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T01:36:59.828 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:00.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:36:59 smithi046 ceph-mon[36108]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:37:00.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:36:59 smithi007 ceph-mon[31996]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:37:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:36:59 smithi203 ceph-mon[36345]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:37:01.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:00 smithi007 ceph-mon[31996]: pgmap v382: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T01:37:01.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:00 smithi046 ceph-mon[36108]: pgmap v382: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T01:37:01.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:00 smithi203 ceph-mon[36345]: pgmap v382: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T01:37:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 -- 172.21.15.7:0/3583560777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75041044e0 msgr2=0x7f7504104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 --2- 172.21.15.7:0/3583560777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75041044e0 0x7f7504104960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009990 tx=0x7f74f0009ca0 comp rx=0 tx=0).stop 2024-03-18T01:37:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 -- 172.21.15.7:0/3583560777 shutdown_connections 2024-03-18T01:37:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 --2- 172.21.15.7:0/3583560777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7504104ea0 0x7f7504111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 --2- 172.21.15.7:0/3583560777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f750410a590 0x7f750410a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 --2- 172.21.15.7:0/3583560777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75041044e0 0x7f7504104960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 -- 172.21.15.7:0/3583560777 >> 172.21.15.7:0/3583560777 conn(0x7f75040fffd0 msgr2=0x7f75041023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 -- 172.21.15.7:0/3583560777 shutdown_connections 2024-03-18T01:37:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.969+0000 7f7509000700 1 -- 172.21.15.7:0/3583560777 wait complete. 2024-03-18T01:37:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.970+0000 7f7509000700 1 Processor -- start 2024-03-18T01:37:01.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.970+0000 7f7509000700 1 -- start start 2024-03-18T01:37:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7509000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75041044e0 0x7f75041a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7509000700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7504104ea0 0x7f75041a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7509000700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750410a590 0x7f750419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7509000700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f75041147d0 con 0x7f7504104ea0 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7509000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7504114650 con 0x7f75041044e0 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7509000700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7504114950 con 0x7f750410a590 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7501e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75041044e0 0x7f75041a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7501e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75041044e0 0x7f75041a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:60098/0 (socket says 172.21.15.7:60098) 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7501e00700 1 -- 172.21.15.7:0/475305494 learned_addr learned my addr 172.21.15.7:0/475305494 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7502800700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750410a590 0x7f750419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.971+0000 7f7501400700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7504104ea0 0x7f75041a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f7501e00700 1 -- 172.21.15.7:0/475305494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750410a590 msgr2=0x7f750419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f7501e00700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750410a590 0x7f750419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f7501e00700 1 -- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7504104ea0 msgr2=0x7f75041a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f7501e00700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7504104ea0 0x7f75041a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f7501e00700 1 -- 172.21.15.7:0/475305494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750419b130 con 0x7f75041044e0 2024-03-18T01:37:01.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f7501e00700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75041044e0 0x7f75041a0810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f74f400d980 tx=0x7f74f400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:01.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f74faa00700 1 -- 172.21.15.7:0/475305494 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4016070 con 0x7f75041044e0 2024-03-18T01:37:01.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750419b420 con 0x7f75041044e0 2024-03-18T01:37:01.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.972+0000 7f74faa00700 1 -- 172.21.15.7:0/475305494 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f4004d10 con 0x7f75041044e0 2024-03-18T01:37:01.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.973+0000 7f74faa00700 1 -- 172.21.15.7:0/475305494 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f40055f0 con 0x7f75041044e0 2024-03-18T01:37:01.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.973+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75041a76e0 con 0x7f75041044e0 2024-03-18T01:37:01.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.974+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7504068f90 con 0x7f75041044e0 2024-03-18T01:37:01.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.976+0000 7f74faa00700 1 -- 172.21.15.7:0/475305494 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f74f4004750 con 0x7f75041044e0 2024-03-18T01:37:01.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.978+0000 7f74faa00700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e80a2910 0x7f74e80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:01.978 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.978+0000 7f74faa00700 1 -- 172.21.15.7:0/475305494 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74f40cf250 con 0x7f75041044e0 2024-03-18T01:37:01.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.979+0000 7f7501400700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e80a2910 0x7f74e80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:01.979 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.979+0000 7f7501400700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e80a2910 0x7f74e80a4dd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f74f0006010 tx=0x7f74f001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:01.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:01.985+0000 7f74faa00700 1 -- 172.21.15.7:0/475305494 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f74f401f020 con 0x7f75041044e0 2024-03-18T01:37:02.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.213+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7504062710 con 0x7f74e80a2910 2024-03-18T01:37:02.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.224+0000 7f74faa00700 1 -- 172.21.15.7:0/475305494 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340690 (secure 0 0 0) 0x7f7504062710 con 0x7f74e80a2910 2024-03-18T01:37:02.232 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e80a2910 msgr2=0x7f74e80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e80a2910 0x7f74e80a4dd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f74f0006010 tx=0x7f74f001a040 comp rx=0 tx=0).stop 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75041044e0 msgr2=0x7f75041a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75041044e0 0x7f75041a0810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f74f400d980 tx=0x7f74f400dc90 comp rx=0 tx=0).stop 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 shutdown_connections 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74e80a2910 0x7f74e80a4dd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f750410a590 0x7f750419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f75041044e0 0x7f75041a0810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 --2- 172.21.15.7:0/475305494 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7504104ea0 0x7f75041a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 >> 172.21.15.7:0/475305494 conn(0x7f75040fffd0 msgr2=0x7f75041015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 shutdown_connections 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:02.232+0000 7f7509000700 1 -- 172.21.15.7:0/475305494 wait complete. 2024-03-18T01:37:02.235 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-18T01:37:02.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:02 smithi007 ceph-mon[31996]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:02.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:02 smithi203 ceph-mon[36345]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:02.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:02 smithi046 ceph-mon[36108]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:03.214 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-03-18T01:37:01.220121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659877,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1050,"num_read_kb":647,"num_write":431,"num_write_kb":3083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367576,"kb_used_data":14232,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574248,"statfs":{"total":1151940427776,"available":1151564029952,"internally_reserved":0,"allocated":14573568,"data_stored":7964657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":62,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005658"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202595+0000","last_change":"2024-03-18T01:34:56.661330+0000","last_active":"2024-03-18T01:36:57.202595+0000","last_peered":"2024-03-18T01:36:57.202595+0000","last_clean":"2024-03-18T01:36:57.202595+0000","last_became_active":"2024-03-18T01:34:56.660949+0000","last_became_peered":"2024-03-18T01:34:56.660949+0000","last_unstale":"2024-03-18T01:36:57.202595+0000","last_undegraded":"2024-03-18T01:36:57.202595+0000","last_fullsized":"2024-03-18T01:36:57.202595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:40.962487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191808+0000","last_change":"2024-03-18T01:34:56.652417+0000","last_active":"2024-03-18T01:36:57.191808+0000","last_peered":"2024-03-18T01:36:57.191808+0000","last_clean":"2024-03-18T01:36:57.191808+0000","last_became_active":"2024-03-18T01:34:56.651851+0000","last_became_peered":"2024-03-18T01:34:56.651851+0000","last_unstale":"2024-03-18T01:36:57.191808+0000","last_undegraded":"2024-03-18T01:36:57.191808+0000","last_fullsized":"2024-03-18T01:36:57.191808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:14:24.172177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214353+0000","last_change":"2024-03-18T01:34:56.647657+0000","last_active":"2024-03-18T01:36:57.214353+0000","last_peered":"2024-03-18T01:36:57.214353+0000","last_clean":"2024-03-18T01:36:57.214353+0000","last_became_active":"2024-03-18T01:34:56.647241+0000","last_became_peered":"2024-03-18T01:34:56.647241+0000","last_unstale":"2024-03-18T01:36:57.214353+0000","last_undegraded":"2024-03-18T01:36:57.214353+0000","last_fullsized":"2024-03-18T01:36:57.214353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:01.706351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.198329+0000","last_change":"2024-03-18T01:35:58.815017+0000","last_active":"2024-03-18T01:36:57.198329+0000","last_peered":"2024-03-18T01:36:57.198329+0000","last_clean":"2024-03-18T01:36:57.198329+0000","last_became_active":"2024-03-18T01:35:58.814140+0000","last_became_peered":"2024-03-18T01:35:58.814140+0000","last_unstale":"2024-03-18T01:36:57.198329+0000","last_undegraded":"2024-03-18T01:36:57.198329+0000","last_fullsized":"2024-03-18T01:36:57.198329+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:13:21.960119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.212284+0000","last_change":"2024-03-18T01:34:56.643992+0000","last_active":"2024-03-18T01:36:57.212284+0000","last_peered":"2024-03-18T01:36:57.212284+0000","last_clean":"2024-03-18T01:36:57.212284+0000","last_became_active":"2024-03-18T01:34:56.643652+0000","last_became_peered":"2024-03-18T01:34:56.643652+0000","last_unstale":"2024-03-18T01:36:57.212284+0000","last_undegraded":"2024-03-18T01:36:57.212284+0000","last_fullsized":"2024-03-18T01:36:57.212284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:12.005357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202858+0000","last_change":"2024-03-18T01:34:56.649955+0000","last_active":"2024-03-18T01:36:57.202858+0000","last_peered":"2024-03-18T01:36:57.202858+0000","last_clean":"2024-03-18T01:36:57.202858+0000","last_became_active":"2024-03-18T01:34:56.649609+0000","last_became_peered":"2024-03-18T01:34:56.649609+0000","last_unstale":"2024-03-18T01:36:57.202858+0000","last_undegraded":"2024-03-18T01:36:57.202858+0000","last_fullsized":"2024-03-18T01:36:57.202858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:34:28.113733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806244+0000","last_change":"2024-03-18T01:34:56.647561+0000","last_active":"2024-03-18T01:35:58.806244+0000","last_peered":"2024-03-18T01:35:58.806244+0000","last_clean":"2024-03-18T01:35:58.806244+0000","last_became_active":"2024-03-18T01:34:56.647040+0000","last_became_peered":"2024-03-18T01:34:56.647040+0000","last_unstale":"2024-03-18T01:35:58.806244+0000","last_undegraded":"2024-03-18T01:35:58.806244+0000","last_fullsized":"2024-03-18T01:35:58.806244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:42.433751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220223+0000","last_change":"2024-03-18T01:34:56.652073+0000","last_active":"2024-03-18T01:36:57.220223+0000","last_peered":"2024-03-18T01:36:57.220223+0000","last_clean":"2024-03-18T01:36:57.220223+0000","last_became_active":"2024-03-18T01:34:56.651748+0000","last_became_peered":"2024-03-18T01:34:56.651748+0000","last_unstale":"2024-03-18T01:36:57.220223+0000","last_undegraded":"2024-03-18T01:36:57.220223+0000","last_fullsized":"2024-03-18T01:36:57.220223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:33.148823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.200861+0000","last_change":"2024-03-18T01:34:56.645782+0000","last_active":"2024-03-18T01:36:57.200861+0000","last_peered":"2024-03-18T01:36:57.200861+0000","last_clean":"2024-03-18T01:36:57.200861+0000","last_became_active":"2024-03-18T01:34:56.645439+0000","last_became_peered":"2024-03-18T01:34:56.645439+0000","last_unstale":"2024-03-18T01:36:57.200861+0000","last_undegraded":"2024-03-18T01:36:57.200861+0000","last_fullsized":"2024-03-18T01:36:57.200861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:40.242348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.184217+0000","last_change":"2024-03-18T01:34:56.639697+0000","last_active":"2024-03-18T01:36:57.184217+0000","last_peered":"2024-03-18T01:36:57.184217+0000","last_clean":"2024-03-18T01:36:57.184217+0000","last_became_active":"2024-03-18T01:34:56.639339+0000","last_became_peered":"2024-03-18T01:34:56.639339+0000","last_unstale":"2024-03-18T01:36:57.184217+0000","last_undegraded":"2024-03-18T01:36:57.184217+0000","last_fullsized":"2024-03-18T01:36:57.184217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:25:22.741437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806829+0000","last_change":"2024-03-18T01:34:56.655341+0000","last_active":"2024-03-18T01:35:58.806829+0000","last_peered":"2024-03-18T01:35:58.806829+0000","last_clean":"2024-03-18T01:35:58.806829+0000","last_became_active":"2024-03-18T01:34:56.655015+0000","last_became_peered":"2024-03-18T01:34:56.655015+0000","last_unstale":"2024-03-18T01:35:58.806829+0000","last_undegraded":"2024-03-18T01:35:58.806829+0000","last_fullsized":"2024-03-18T01:35:58.806829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:50.766000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.185395+0000","last_change":"2024-03-18T01:34:56.648399+0000","last_active":"2024-03-18T01:36:57.185395+0000","last_peered":"2024-03-18T01:36:57.185395+0000","last_clean":"2024-03-18T01:36:57.185395+0000","last_became_active":"2024-03-18T01:34:56.648042+0000","last_became_peered":"2024-03-18T01:34:56.648042+0000","last_unstale":"2024-03-18T01:36:57.185395+0000","last_undegraded":"2024-03-18T01:36:57.185395+0000","last_fullsized":"2024-03-18T01:36:57.185395+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:04.972230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211557+0000","last_change":"2024-03-18T01:34:56.653465+0000","last_active":"2024-03-18T01:36:57.211557+0000","last_peered":"2024-03-18T01:36:57.211557+0000","last_clean":"2024-03-18T01:36:57.211557+0000","last_became_active":"2024-03-18T01:34:56.651183+0000","last_became_peered":"2024-03-18T01:34:56.651183+0000","last_unstale":"2024-03-18T01:36:57.211557+0000","last_undegraded":"2024-03-18T01:36:57.211557+0000","last_fullsized":"2024-03-18T01:36:57.211557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:57.983330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211473+0000","last_change":"2024-03-18T01:34:56.658845+0000","last_active":"2024-03-18T01:36:57.211473+0000","last_peered":"2024-03-18T01:36:57.211473+0000","last_clean":"2024-03-18T01:36:57.211473+0000","last_became_active":"2024-03-18T01:34:56.658435+0000","last_became_peered":"2024-03-18T01:34:56.658435+0000","last_unstale":"2024-03-18T01:36:57.211473+0000","last_undegraded":"2024-03-18T01:36:57.211473+0000","last_fullsized":"2024-03-18T01:36:57.211473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:02.176325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211315+0000","last_change":"2024-03-18T01:34:56.660752+0000","last_active":"2024-03-18T01:36:57.211315+0000","last_peered":"2024-03-18T01:36:57.211315+0000","last_clean":"2024-03-18T01:36:57.211315+0000","last_became_active":"2024-03-18T01:34:56.660378+0000","last_became_peered":"2024-03-18T01:34:56.660378+0000","last_unstale":"2024-03-18T01:36:57.211315+0000","last_undegraded":"2024-03-18T01:36:57.211315+0000","last_fullsized":"2024-03-18T01:36:57.211315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:13.143481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214133+0000","last_change":"2024-03-18T01:34:56.656589+0000","last_active":"2024-03-18T01:36:57.214133+0000","last_peered":"2024-03-18T01:36:57.214133+0000","last_clean":"2024-03-18T01:36:57.214133+0000","last_became_active":"2024-03-18T01:34:56.656184+0000","last_became_peered":"2024-03-18T01:34:56.656184+0000","last_unstale":"2024-03-18T01:36:57.214133+0000","last_undegraded":"2024-03-18T01:36:57.214133+0000","last_fullsized":"2024-03-18T01:36:57.214133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:56.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202733+0000","last_change":"2024-03-18T01:34:56.661327+0000","last_active":"2024-03-18T01:36:57.202733+0000","last_peered":"2024-03-18T01:36:57.202733+0000","last_clean":"2024-03-18T01:36:57.202733+0000","last_became_active":"2024-03-18T01:34:56.660767+0000","last_became_peered":"2024-03-18T01:34:56.660767+0000","last_unstale":"2024-03-18T01:36:57.202733+0000","last_undegraded":"2024-03-18T01:36:57.202733+0000","last_fullsized":"2024-03-18T01:36:57.202733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:07:59.066028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203219+0000","last_change":"2024-03-18T01:34:56.650149+0000","last_active":"2024-03-18T01:36:57.203219+0000","last_peered":"2024-03-18T01:36:57.203219+0000","last_clean":"2024-03-18T01:36:57.203219+0000","last_became_active":"2024-03-18T01:34:56.649780+0000","last_became_peered":"2024-03-18T01:34:56.649780+0000","last_unstale":"2024-03-18T01:36:57.203219+0000","last_undegraded":"2024-03-18T01:36:57.203219+0000","last_fullsized":"2024-03-18T01:36:57.203219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:41.605563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.188672+0000","last_change":"2024-03-18T01:34:56.653919+0000","last_active":"2024-03-18T01:36:57.188672+0000","last_peered":"2024-03-18T01:36:57.188672+0000","last_clean":"2024-03-18T01:36:57.188672+0000","last_became_active":"2024-03-18T01:34:56.653512+0000","last_became_peered":"2024-03-18T01:34:56.653512+0000","last_unstale":"2024-03-18T01:36:57.188672+0000","last_undegraded":"2024-03-18T01:36:57.188672+0000","last_fullsized":"2024-03-18T01:36:57.188672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:27.419518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810112+0000","last_change":"2024-03-18T01:35:01.149999+0000","last_active":"2024-03-18T01:35:58.810112+0000","last_peered":"2024-03-18T01:35:58.810112+0000","last_clean":"2024-03-18T01:35:58.810112+0000","last_became_active":"2024-03-18T01:35:01.149603+0000","last_became_peered":"2024-03-18T01:35:01.149603+0000","last_unstale":"2024-03-18T01:35:58.810112+0000","last_undegraded":"2024-03-18T01:35:58.810112+0000","last_fullsized":"2024-03-18T01:35:58.810112+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:37.619517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":133,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:00.781269+0000","last_change":"2024-03-18T01:34:55.650404+0000","last_active":"2024-03-18T01:37:00.781269+0000","last_peered":"2024-03-18T01:37:00.781269+0000","last_clean":"2024-03-18T01:37:00.781269+0000","last_became_active":"2024-03-18T01:34:55.650068+0000","last_became_peered":"2024-03-18T01:34:55.650068+0000","last_unstale":"2024-03-18T01:37:00.781269+0000","last_undegraded":"2024-03-18T01:37:00.781269+0000","last_fullsized":"2024-03-18T01:37:00.781269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:11.396529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211353+0000","last_change":"2024-03-18T01:34:56.654265+0000","last_active":"2024-03-18T01:36:57.211353+0000","last_peered":"2024-03-18T01:36:57.211353+0000","last_clean":"2024-03-18T01:36:57.211353+0000","last_became_active":"2024-03-18T01:34:56.653886+0000","last_became_peered":"2024-03-18T01:34:56.653886+0000","last_unstale":"2024-03-18T01:36:57.211353+0000","last_undegraded":"2024-03-18T01:36:57.211353+0000","last_fullsized":"2024-03-18T01:36:57.211353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:01:44.069966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801447+0000","last_change":"2024-03-18T01:35:01.155838+0000","last_active":"2024-03-18T01:35:58.801447+0000","last_peered":"2024-03-18T01:35:58.801447+0000","last_clean":"2024-03-18T01:35:58.801447+0000","last_became_active":"2024-03-18T01:35:01.155391+0000","last_became_peered":"2024-03-18T01:35:01.155391+0000","last_unstale":"2024-03-18T01:35:58.801447+0000","last_undegraded":"2024-03-18T01:35:58.801447+0000","last_fullsized":"2024-03-18T01:35:58.801447+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:31.885767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821989+0000","last_change":"2024-03-18T01:34:55.658830+0000","last_active":"2024-03-18T01:35:58.821989+0000","last_peered":"2024-03-18T01:35:58.821989+0000","last_clean":"2024-03-18T01:35:58.821989+0000","last_became_active":"2024-03-18T01:34:55.658395+0000","last_became_peered":"2024-03-18T01:34:55.658395+0000","last_unstale":"2024-03-18T01:35:58.821989+0000","last_undegraded":"2024-03-18T01:35:58.821989+0000","last_fullsized":"2024-03-18T01:35:58.821989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:07.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213648+0000","last_change":"2024-03-18T01:34:56.656874+0000","last_active":"2024-03-18T01:36:57.213648+0000","last_peered":"2024-03-18T01:36:57.213648+0000","last_clean":"2024-03-18T01:36:57.213648+0000","last_became_active":"2024-03-18T01:34:56.656525+0000","last_became_peered":"2024-03-18T01:34:56.656525+0000","last_unstale":"2024-03-18T01:36:57.213648+0000","last_undegraded":"2024-03-18T01:36:57.213648+0000","last_fullsized":"2024-03-18T01:36:57.213648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:35:19.363586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.822392+0000","last_change":"2024-03-18T01:35:58.822392+0000","last_active":"2024-03-18T01:35:58.822392+0000","last_peered":"2024-03-18T01:35:58.822392+0000","last_clean":"2024-03-18T01:35:58.822392+0000","last_became_active":"2024-03-18T01:35:58.821792+0000","last_became_peered":"2024-03-18T01:35:58.821792+0000","last_unstale":"2024-03-18T01:35:58.822392+0000","last_undegraded":"2024-03-18T01:35:58.822392+0000","last_fullsized":"2024-03-18T01:35:58.822392+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:43:26.345066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802688+0000","last_change":"2024-03-18T01:34:55.648218+0000","last_active":"2024-03-18T01:35:58.802688+0000","last_peered":"2024-03-18T01:35:58.802688+0000","last_clean":"2024-03-18T01:35:58.802688+0000","last_became_active":"2024-03-18T01:34:55.647779+0000","last_became_peered":"2024-03-18T01:34:55.647779+0000","last_unstale":"2024-03-18T01:35:58.802688+0000","last_undegraded":"2024-03-18T01:35:58.802688+0000","last_fullsized":"2024-03-18T01:35:58.802688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:22:08.136973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205267+0000","last_change":"2024-03-18T01:34:56.649900+0000","last_active":"2024-03-18T01:36:57.205267+0000","last_peered":"2024-03-18T01:36:57.205267+0000","last_clean":"2024-03-18T01:36:57.205267+0000","last_became_active":"2024-03-18T01:34:56.649564+0000","last_became_peered":"2024-03-18T01:34:56.649564+0000","last_unstale":"2024-03-18T01:36:57.205267+0000","last_undegraded":"2024-03-18T01:36:57.205267+0000","last_fullsized":"2024-03-18T01:36:57.205267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:29.861861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801554+0000","last_change":"2024-03-18T01:35:01.147062+0000","last_active":"2024-03-18T01:35:58.801554+0000","last_peered":"2024-03-18T01:35:58.801554+0000","last_clean":"2024-03-18T01:35:58.801554+0000","last_became_active":"2024-03-18T01:35:01.146666+0000","last_became_peered":"2024-03-18T01:35:01.146666+0000","last_unstale":"2024-03-18T01:35:58.801554+0000","last_undegraded":"2024-03-18T01:35:58.801554+0000","last_fullsized":"2024-03-18T01:35:58.801554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:15:21.433263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.817437+0000","last_change":"2024-03-18T01:35:58.816601+0000","last_active":"2024-03-18T01:35:58.817437+0000","last_peered":"2024-03-18T01:35:58.817437+0000","last_clean":"2024-03-18T01:35:58.817437+0000","last_became_active":"2024-03-18T01:35:58.808678+0000","last_became_peered":"2024-03-18T01:35:58.808678+0000","last_unstale":"2024-03-18T01:35:58.817437+0000","last_undegraded":"2024-03-18T01:35:58.817437+0000","last_fullsized":"2024-03-18T01:35:58.817437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:37:42.998856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192317+0000","last_change":"2024-03-18T01:34:56.650449+0000","last_active":"2024-03-18T01:36:57.192317+0000","last_peered":"2024-03-18T01:36:57.192317+0000","last_clean":"2024-03-18T01:36:57.192317+0000","last_became_active":"2024-03-18T01:34:56.650103+0000","last_became_peered":"2024-03-18T01:34:56.650103+0000","last_unstale":"2024-03-18T01:36:57.192317+0000","last_undegraded":"2024-03-18T01:36:57.192317+0000","last_fullsized":"2024-03-18T01:36:57.192317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:07:55.003592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809552+0000","last_change":"2024-03-18T01:35:01.132227+0000","last_active":"2024-03-18T01:35:58.809552+0000","last_peered":"2024-03-18T01:35:58.809552+0000","last_clean":"2024-03-18T01:35:58.809552+0000","last_became_active":"2024-03-18T01:35:01.131767+0000","last_became_peered":"2024-03-18T01:35:01.131767+0000","last_unstale":"2024-03-18T01:35:58.809552+0000","last_undegraded":"2024-03-18T01:35:58.809552+0000","last_fullsized":"2024-03-18T01:35:58.809552+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:54.335482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.736265+0000","last_change":"2024-03-18T01:34:55.649277+0000","last_active":"2024-03-18T01:36:57.736265+0000","last_peered":"2024-03-18T01:36:57.736265+0000","last_clean":"2024-03-18T01:36:57.736265+0000","last_became_active":"2024-03-18T01:34:55.648917+0000","last_became_peered":"2024-03-18T01:34:55.648917+0000","last_unstale":"2024-03-18T01:36:57.736265+0000","last_undegraded":"2024-03-18T01:36:57.736265+0000","last_fullsized":"2024-03-18T01:36:57.736265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:00.117647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192873+0000","last_change":"2024-03-18T01:34:56.648394+0000","last_active":"2024-03-18T01:36:57.192873+0000","last_peered":"2024-03-18T01:36:57.192873+0000","last_clean":"2024-03-18T01:36:57.192873+0000","last_became_active":"2024-03-18T01:34:56.648043+0000","last_became_peered":"2024-03-18T01:34:56.648043+0000","last_unstale":"2024-03-18T01:36:57.192873+0000","last_undegraded":"2024-03-18T01:36:57.192873+0000","last_fullsized":"2024-03-18T01:36:57.192873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:54:52.531851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809458+0000","last_change":"2024-03-18T01:35:01.151346+0000","last_active":"2024-03-18T01:35:58.809458+0000","last_peered":"2024-03-18T01:35:58.809458+0000","last_clean":"2024-03-18T01:35:58.809458+0000","last_became_active":"2024-03-18T01:35:01.150916+0000","last_became_peered":"2024-03-18T01:35:01.150916+0000","last_unstale":"2024-03-18T01:35:58.809458+0000","last_undegraded":"2024-03-18T01:35:58.809458+0000","last_fullsized":"2024-03-18T01:35:58.809458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:44:36.000983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.803936+0000","last_change":"2024-03-18T01:34:55.648498+0000","last_active":"2024-03-18T01:35:58.803936+0000","last_peered":"2024-03-18T01:35:58.803936+0000","last_clean":"2024-03-18T01:35:58.803936+0000","last_became_active":"2024-03-18T01:34:55.648152+0000","last_became_peered":"2024-03-18T01:34:55.648152+0000","last_unstale":"2024-03-18T01:35:58.803936+0000","last_undegraded":"2024-03-18T01:35:58.803936+0000","last_fullsized":"2024-03-18T01:35:58.803936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:33.138620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211275+0000","last_change":"2024-03-18T01:34:56.651420+0000","last_active":"2024-03-18T01:36:57.211275+0000","last_peered":"2024-03-18T01:36:57.211275+0000","last_clean":"2024-03-18T01:36:57.211275+0000","last_became_active":"2024-03-18T01:34:56.650909+0000","last_became_peered":"2024-03-18T01:34:56.650909+0000","last_unstale":"2024-03-18T01:36:57.211275+0000","last_undegraded":"2024-03-18T01:36:57.211275+0000","last_fullsized":"2024-03-18T01:36:57.211275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:43:40.548970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804655+0000","last_change":"2024-03-18T01:35:01.157759+0000","last_active":"2024-03-18T01:35:58.804655+0000","last_peered":"2024-03-18T01:35:58.804655+0000","last_clean":"2024-03-18T01:35:58.804655+0000","last_became_active":"2024-03-18T01:35:01.157288+0000","last_became_peered":"2024-03-18T01:35:01.157288+0000","last_unstale":"2024-03-18T01:35:58.804655+0000","last_undegraded":"2024-03-18T01:35:58.804655+0000","last_fullsized":"2024-03-18T01:35:58.804655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:05:41.272727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.818719+0000","last_change":"2024-03-18T01:34:55.650388+0000","last_active":"2024-03-18T01:35:58.818719+0000","last_peered":"2024-03-18T01:35:58.818719+0000","last_clean":"2024-03-18T01:35:58.818719+0000","last_became_active":"2024-03-18T01:34:55.650055+0000","last_became_peered":"2024-03-18T01:34:55.650055+0000","last_unstale":"2024-03-18T01:35:58.818719+0000","last_undegraded":"2024-03-18T01:35:58.818719+0000","last_fullsized":"2024-03-18T01:35:58.818719+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:29:40.758561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.212889+0000","last_change":"2024-03-18T01:34:56.658634+0000","last_active":"2024-03-18T01:36:57.212889+0000","last_peered":"2024-03-18T01:36:57.212889+0000","last_clean":"2024-03-18T01:36:57.212889+0000","last_became_active":"2024-03-18T01:34:56.657196+0000","last_became_peered":"2024-03-18T01:34:56.657196+0000","last_unstale":"2024-03-18T01:36:57.212889+0000","last_undegraded":"2024-03-18T01:36:57.212889+0000","last_fullsized":"2024-03-18T01:36:57.212889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:12:36.265949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811177+0000","last_change":"2024-03-18T01:35:01.150431+0000","last_active":"2024-03-18T01:35:58.811177+0000","last_peered":"2024-03-18T01:35:58.811177+0000","last_clean":"2024-03-18T01:35:58.811177+0000","last_became_active":"2024-03-18T01:35:01.149981+0000","last_became_peered":"2024-03-18T01:35:01.149981+0000","last_unstale":"2024-03-18T01:35:58.811177+0000","last_undegraded":"2024-03-18T01:35:58.811177+0000","last_fullsized":"2024-03-18T01:35:58.811177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:34:55.371860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806989+0000","last_change":"2024-03-18T01:34:55.649346+0000","last_active":"2024-03-18T01:35:58.806989+0000","last_peered":"2024-03-18T01:35:58.806989+0000","last_clean":"2024-03-18T01:35:58.806989+0000","last_became_active":"2024-03-18T01:34:55.648958+0000","last_became_peered":"2024-03-18T01:34:55.648958+0000","last_unstale":"2024-03-18T01:35:58.806989+0000","last_undegraded":"2024-03-18T01:35:58.806989+0000","last_fullsized":"2024-03-18T01:35:58.806989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:53.788081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.218835+0000","last_change":"2024-03-18T01:34:56.653525+0000","last_active":"2024-03-18T01:36:57.218835+0000","last_peered":"2024-03-18T01:36:57.218835+0000","last_clean":"2024-03-18T01:36:57.218835+0000","last_became_active":"2024-03-18T01:34:56.653218+0000","last_became_peered":"2024-03-18T01:34:56.653218+0000","last_unstale":"2024-03-18T01:36:57.218835+0000","last_undegraded":"2024-03-18T01:36:57.218835+0000","last_fullsized":"2024-03-18T01:36:57.218835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:32:37.479782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816301+0000","last_change":"2024-03-18T01:35:01.135505+0000","last_active":"2024-03-18T01:35:58.816301+0000","last_peered":"2024-03-18T01:35:58.816301+0000","last_clean":"2024-03-18T01:35:58.816301+0000","last_became_active":"2024-03-18T01:35:01.135055+0000","last_became_peered":"2024-03-18T01:35:01.135055+0000","last_unstale":"2024-03-18T01:35:58.816301+0000","last_undegraded":"2024-03-18T01:35:58.816301+0000","last_fullsized":"2024-03-18T01:35:58.816301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:52:35.069313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":207,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.735429+0000","last_change":"2024-03-18T01:34:55.659389+0000","last_active":"2024-03-18T01:36:57.735429+0000","last_peered":"2024-03-18T01:36:57.735429+0000","last_clean":"2024-03-18T01:36:57.735429+0000","last_became_active":"2024-03-18T01:34:55.659090+0000","last_became_peered":"2024-03-18T01:34:55.659090+0000","last_unstale":"2024-03-18T01:36:57.735429+0000","last_undegraded":"2024-03-18T01:36:57.735429+0000","last_fullsized":"2024-03-18T01:36:57.735429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:55:51.731788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":176,"num_read_kb":176,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.206532+0000","last_change":"2024-03-18T01:34:56.650411+0000","last_active":"2024-03-18T01:36:57.206532+0000","last_peered":"2024-03-18T01:36:57.206532+0000","last_clean":"2024-03-18T01:36:57.206532+0000","last_became_active":"2024-03-18T01:34:56.650036+0000","last_became_peered":"2024-03-18T01:34:56.650036+0000","last_unstale":"2024-03-18T01:36:57.206532+0000","last_undegraded":"2024-03-18T01:36:57.206532+0000","last_fullsized":"2024-03-18T01:36:57.206532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:59.912207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811599+0000","last_change":"2024-03-18T01:35:58.811137+0000","last_active":"2024-03-18T01:35:58.811599+0000","last_peered":"2024-03-18T01:35:58.811599+0000","last_clean":"2024-03-18T01:35:58.811599+0000","last_became_active":"2024-03-18T01:35:58.810398+0000","last_became_peered":"2024-03-18T01:35:58.810398+0000","last_unstale":"2024-03-18T01:35:58.811599+0000","last_undegraded":"2024-03-18T01:35:58.811599+0000","last_fullsized":"2024-03-18T01:35:58.811599+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:08.271404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807051+0000","last_change":"2024-03-18T01:34:55.647879+0000","last_active":"2024-03-18T01:35:58.807051+0000","last_peered":"2024-03-18T01:35:58.807051+0000","last_clean":"2024-03-18T01:35:58.807051+0000","last_became_active":"2024-03-18T01:34:55.647549+0000","last_became_peered":"2024-03-18T01:34:55.647549+0000","last_unstale":"2024-03-18T01:35:58.807051+0000","last_undegraded":"2024-03-18T01:35:58.807051+0000","last_fullsized":"2024-03-18T01:35:58.807051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:12.261729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.207456+0000","last_change":"2024-03-18T01:35:58.813429+0000","last_active":"2024-03-18T01:36:57.207456+0000","last_peered":"2024-03-18T01:36:57.207456+0000","last_clean":"2024-03-18T01:36:57.207456+0000","last_became_active":"2024-03-18T01:35:58.812659+0000","last_became_peered":"2024-03-18T01:35:58.812659+0000","last_unstale":"2024-03-18T01:36:57.207456+0000","last_undegraded":"2024-03-18T01:36:57.207456+0000","last_fullsized":"2024-03-18T01:36:57.207456+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:22:01.273649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802202+0000","last_change":"2024-03-18T01:35:01.153448+0000","last_active":"2024-03-18T01:35:58.802202+0000","last_peered":"2024-03-18T01:35:58.802202+0000","last_clean":"2024-03-18T01:35:58.802202+0000","last_became_active":"2024-03-18T01:35:01.152593+0000","last_became_peered":"2024-03-18T01:35:01.152593+0000","last_unstale":"2024-03-18T01:35:58.802202+0000","last_undegraded":"2024-03-18T01:35:58.802202+0000","last_fullsized":"2024-03-18T01:35:58.802202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:46.363662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808537+0000","last_change":"2024-03-18T01:34:55.655079+0000","last_active":"2024-03-18T01:35:58.808537+0000","last_peered":"2024-03-18T01:35:58.808537+0000","last_clean":"2024-03-18T01:35:58.808537+0000","last_became_active":"2024-03-18T01:34:55.654702+0000","last_became_peered":"2024-03-18T01:34:55.654702+0000","last_unstale":"2024-03-18T01:35:58.808537+0000","last_undegraded":"2024-03-18T01:35:58.808537+0000","last_fullsized":"2024-03-18T01:35:58.808537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:50:46.524614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.204870+0000","last_change":"2024-03-18T01:34:56.644441+0000","last_active":"2024-03-18T01:36:57.204870+0000","last_peered":"2024-03-18T01:36:57.204870+0000","last_clean":"2024-03-18T01:36:57.204870+0000","last_became_active":"2024-03-18T01:34:56.643924+0000","last_became_peered":"2024-03-18T01:34:56.643924+0000","last_unstale":"2024-03-18T01:36:57.204870+0000","last_undegraded":"2024-03-18T01:36:57.204870+0000","last_fullsized":"2024-03-18T01:36:57.204870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:44.260300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809026+0000","last_change":"2024-03-18T01:35:01.150022+0000","last_active":"2024-03-18T01:35:58.809026+0000","last_peered":"2024-03-18T01:35:58.809026+0000","last_clean":"2024-03-18T01:35:58.809026+0000","last_became_active":"2024-03-18T01:35:01.149510+0000","last_became_peered":"2024-03-18T01:35:01.149510+0000","last_unstale":"2024-03-18T01:35:58.809026+0000","last_undegraded":"2024-03-18T01:35:58.809026+0000","last_fullsized":"2024-03-18T01:35:58.809026+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:40:36.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":244,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.735561+0000","last_change":"2024-03-18T01:34:55.642397+0000","last_active":"2024-03-18T01:36:57.735561+0000","last_peered":"2024-03-18T01:36:57.735561+0000","last_clean":"2024-03-18T01:36:57.735561+0000","last_became_active":"2024-03-18T01:34:55.642046+0000","last_became_peered":"2024-03-18T01:34:55.642046+0000","last_unstale":"2024-03-18T01:36:57.735561+0000","last_undegraded":"2024-03-18T01:36:57.735561+0000","last_fullsized":"2024-03-18T01:36:57.735561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:48:31.631414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62321,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":182,"num_read_kb":152,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.179405+0000","last_change":"2024-03-18T01:34:56.644388+0000","last_active":"2024-03-18T01:36:57.179405+0000","last_peered":"2024-03-18T01:36:57.179405+0000","last_clean":"2024-03-18T01:36:57.179405+0000","last_became_active":"2024-03-18T01:34:56.644014+0000","last_became_peered":"2024-03-18T01:34:56.644014+0000","last_unstale":"2024-03-18T01:36:57.179405+0000","last_undegraded":"2024-03-18T01:36:57.179405+0000","last_fullsized":"2024-03-18T01:36:57.179405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:06:18.193600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.823299+0000","last_change":"2024-03-18T01:35:58.822872+0000","last_active":"2024-03-18T01:35:58.823299+0000","last_peered":"2024-03-18T01:35:58.823299+0000","last_clean":"2024-03-18T01:35:58.823299+0000","last_became_active":"2024-03-18T01:35:58.822325+0000","last_became_peered":"2024-03-18T01:35:58.822325+0000","last_unstale":"2024-03-18T01:35:58.823299+0000","last_undegraded":"2024-03-18T01:35:58.823299+0000","last_fullsized":"2024-03-18T01:35:58.823299+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:54:21.392764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.834419+0000","last_change":"2024-03-18T01:35:58.834419+0000","last_active":"2024-03-18T01:35:58.834419+0000","last_peered":"2024-03-18T01:35:58.834419+0000","last_clean":"2024-03-18T01:35:58.834419+0000","last_became_active":"2024-03-18T01:35:58.822930+0000","last_became_peered":"2024-03-18T01:35:58.822930+0000","last_unstale":"2024-03-18T01:35:58.834419+0000","last_undegraded":"2024-03-18T01:35:58.834419+0000","last_fullsized":"2024-03-18T01:35:58.834419+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:15.777787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.221908+0000","last_change":"2024-03-18T01:34:56.657786+0000","last_active":"2024-03-18T01:36:57.221908+0000","last_peered":"2024-03-18T01:36:57.221908+0000","last_clean":"2024-03-18T01:36:57.221908+0000","last_became_active":"2024-03-18T01:34:56.657432+0000","last_became_peered":"2024-03-18T01:34:56.657432+0000","last_unstale":"2024-03-18T01:36:57.221908+0000","last_undegraded":"2024-03-18T01:36:57.221908+0000","last_fullsized":"2024-03-18T01:36:57.221908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:13:39.498562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804914+0000","last_change":"2024-03-18T01:35:01.150495+0000","last_active":"2024-03-18T01:35:58.804914+0000","last_peered":"2024-03-18T01:35:58.804914+0000","last_clean":"2024-03-18T01:35:58.804914+0000","last_became_active":"2024-03-18T01:35:01.150033+0000","last_became_peered":"2024-03-18T01:35:01.150033+0000","last_unstale":"2024-03-18T01:35:58.804914+0000","last_undegraded":"2024-03-18T01:35:58.804914+0000","last_fullsized":"2024-03-18T01:35:58.804914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:38.433827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807787+0000","last_change":"2024-03-18T01:34:55.658820+0000","last_active":"2024-03-18T01:35:58.807787+0000","last_peered":"2024-03-18T01:35:58.807787+0000","last_clean":"2024-03-18T01:35:58.807787+0000","last_became_active":"2024-03-18T01:34:55.658446+0000","last_became_peered":"2024-03-18T01:34:55.658446+0000","last_unstale":"2024-03-18T01:35:58.807787+0000","last_undegraded":"2024-03-18T01:35:58.807787+0000","last_fullsized":"2024-03-18T01:35:58.807787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:19:21.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806325+0000","last_change":"2024-03-18T01:34:56.652890+0000","last_active":"2024-03-18T01:35:58.806325+0000","last_peered":"2024-03-18T01:35:58.806325+0000","last_clean":"2024-03-18T01:35:58.806325+0000","last_became_active":"2024-03-18T01:34:56.652444+0000","last_became_peered":"2024-03-18T01:34:56.652444+0000","last_unstale":"2024-03-18T01:35:58.806325+0000","last_undegraded":"2024-03-18T01:35:58.806325+0000","last_fullsized":"2024-03-18T01:35:58.806325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:58:47.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812325+0000","last_change":"2024-03-18T01:35:01.154700+0000","last_active":"2024-03-18T01:35:58.812325+0000","last_peered":"2024-03-18T01:35:58.812325+0000","last_clean":"2024-03-18T01:35:58.812325+0000","last_became_active":"2024-03-18T01:35:01.148453+0000","last_became_peered":"2024-03-18T01:35:01.148453+0000","last_unstale":"2024-03-18T01:35:58.812325+0000","last_undegraded":"2024-03-18T01:35:58.812325+0000","last_fullsized":"2024-03-18T01:35:58.812325+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:08:40.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:58.819311+0000","last_change":"2024-03-18T01:34:55.643861+0000","last_active":"2024-03-18T01:36:58.819311+0000","last_peered":"2024-03-18T01:36:58.819311+0000","last_clean":"2024-03-18T01:36:58.819311+0000","last_became_active":"2024-03-18T01:34:55.643352+0000","last_became_peered":"2024-03-18T01:34:55.643352+0000","last_unstale":"2024-03-18T01:36:58.819311+0000","last_undegraded":"2024-03-18T01:36:58.819311+0000","last_fullsized":"2024-03-18T01:36:58.819311+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:56:53.685571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.814042+0000","last_change":"2024-03-18T01:35:58.814042+0000","last_active":"2024-03-18T01:35:58.814042+0000","last_peered":"2024-03-18T01:35:58.814042+0000","last_clean":"2024-03-18T01:35:58.814042+0000","last_became_active":"2024-03-18T01:35:58.813622+0000","last_became_peered":"2024-03-18T01:35:58.813622+0000","last_unstale":"2024-03-18T01:35:58.814042+0000","last_undegraded":"2024-03-18T01:35:58.814042+0000","last_fullsized":"2024-03-18T01:35:58.814042+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:52.815608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.824798+0000","last_change":"2024-03-18T01:35:58.824798+0000","last_active":"2024-03-18T01:35:58.824798+0000","last_peered":"2024-03-18T01:35:58.824798+0000","last_clean":"2024-03-18T01:35:58.824798+0000","last_became_active":"2024-03-18T01:35:58.823278+0000","last_became_peered":"2024-03-18T01:35:58.823278+0000","last_unstale":"2024-03-18T01:35:58.824798+0000","last_undegraded":"2024-03-18T01:35:58.824798+0000","last_fullsized":"2024-03-18T01:35:58.824798+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:18:53.871541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801920+0000","last_change":"2024-03-18T01:34:55.658245+0000","last_active":"2024-03-18T01:35:58.801920+0000","last_peered":"2024-03-18T01:35:58.801920+0000","last_clean":"2024-03-18T01:35:58.801920+0000","last_became_active":"2024-03-18T01:34:55.657916+0000","last_became_peered":"2024-03-18T01:34:55.657916+0000","last_unstale":"2024-03-18T01:35:58.801920+0000","last_undegraded":"2024-03-18T01:35:58.801920+0000","last_fullsized":"2024-03-18T01:35:58.801920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:18:49.207721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192011+0000","last_change":"2024-03-18T01:34:56.648646+0000","last_active":"2024-03-18T01:36:57.192011+0000","last_peered":"2024-03-18T01:36:57.192011+0000","last_clean":"2024-03-18T01:36:57.192011+0000","last_became_active":"2024-03-18T01:34:56.648257+0000","last_became_peered":"2024-03-18T01:34:56.648257+0000","last_unstale":"2024-03-18T01:36:57.192011+0000","last_undegraded":"2024-03-18T01:36:57.192011+0000","last_fullsized":"2024-03-18T01:36:57.192011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:53.788445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807562+0000","last_change":"2024-03-18T01:35:01.136912+0000","last_active":"2024-03-18T01:35:58.807562+0000","last_peered":"2024-03-18T01:35:58.807562+0000","last_clean":"2024-03-18T01:35:58.807562+0000","last_became_active":"2024-03-18T01:35:01.136611+0000","last_became_peered":"2024-03-18T01:35:01.136611+0000","last_unstale":"2024-03-18T01:35:58.807562+0000","last_undegraded":"2024-03-18T01:35:58.807562+0000","last_fullsized":"2024-03-18T01:35:58.807562+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:31.089605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802505+0000","last_change":"2024-03-18T01:34:55.648345+0000","last_active":"2024-03-18T01:35:58.802505+0000","last_peered":"2024-03-18T01:35:58.802505+0000","last_clean":"2024-03-18T01:35:58.802505+0000","last_became_active":"2024-03-18T01:34:55.647113+0000","last_became_peered":"2024-03-18T01:34:55.647113+0000","last_unstale":"2024-03-18T01:35:58.802505+0000","last_undegraded":"2024-03-18T01:35:58.802505+0000","last_fullsized":"2024-03-18T01:35:58.802505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:20.904129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190059+0000","last_change":"2024-03-18T01:34:56.649147+0000","last_active":"2024-03-18T01:36:57.190059+0000","last_peered":"2024-03-18T01:36:57.190059+0000","last_clean":"2024-03-18T01:36:57.190059+0000","last_became_active":"2024-03-18T01:34:56.648815+0000","last_became_peered":"2024-03-18T01:34:56.648815+0000","last_unstale":"2024-03-18T01:36:57.190059+0000","last_undegraded":"2024-03-18T01:36:57.190059+0000","last_fullsized":"2024-03-18T01:36:57.190059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:53.560679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802008+0000","last_change":"2024-03-18T01:35:01.156778+0000","last_active":"2024-03-18T01:35:58.802008+0000","last_peered":"2024-03-18T01:35:58.802008+0000","last_clean":"2024-03-18T01:35:58.802008+0000","last_became_active":"2024-03-18T01:35:01.156367+0000","last_became_peered":"2024-03-18T01:35:01.156367+0000","last_unstale":"2024-03-18T01:35:58.802008+0000","last_undegraded":"2024-03-18T01:35:58.802008+0000","last_fullsized":"2024-03-18T01:35:58.802008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:46.223915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810972+0000","last_change":"2024-03-18T01:34:55.658722+0000","last_active":"2024-03-18T01:35:58.810972+0000","last_peered":"2024-03-18T01:35:58.810972+0000","last_clean":"2024-03-18T01:35:58.810972+0000","last_became_active":"2024-03-18T01:34:55.658321+0000","last_became_peered":"2024-03-18T01:34:55.658321+0000","last_unstale":"2024-03-18T01:35:58.810972+0000","last_undegraded":"2024-03-18T01:35:58.810972+0000","last_fullsized":"2024-03-18T01:35:58.810972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:05.753226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220948+0000","last_change":"2024-03-18T01:34:56.649415+0000","last_active":"2024-03-18T01:36:57.220948+0000","last_peered":"2024-03-18T01:36:57.220948+0000","last_clean":"2024-03-18T01:36:57.220948+0000","last_became_active":"2024-03-18T01:34:56.649019+0000","last_became_peered":"2024-03-18T01:34:56.649019+0000","last_unstale":"2024-03-18T01:36:57.220948+0000","last_undegraded":"2024-03-18T01:36:57.220948+0000","last_fullsized":"2024-03-18T01:36:57.220948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:28.473683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819084+0000","last_change":"2024-03-18T01:35:01.148937+0000","last_active":"2024-03-18T01:35:58.819084+0000","last_peered":"2024-03-18T01:35:58.819084+0000","last_clean":"2024-03-18T01:35:58.819084+0000","last_became_active":"2024-03-18T01:35:01.148415+0000","last_became_peered":"2024-03-18T01:35:01.148415+0000","last_unstale":"2024-03-18T01:35:58.819084+0000","last_undegraded":"2024-03-18T01:35:58.819084+0000","last_fullsized":"2024-03-18T01:35:58.819084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:45.926970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:54.772891+0000","last_change":"2024-03-18T01:34:55.653443+0000","last_active":"2024-03-18T01:36:54.772891+0000","last_peered":"2024-03-18T01:36:54.772891+0000","last_clean":"2024-03-18T01:36:54.772891+0000","last_became_active":"2024-03-18T01:34:55.653072+0000","last_became_peered":"2024-03-18T01:34:55.653072+0000","last_unstale":"2024-03-18T01:36:54.772891+0000","last_undegraded":"2024-03-18T01:36:54.772891+0000","last_fullsized":"2024-03-18T01:36:54.772891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:26.376636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202438+0000","last_change":"2024-03-18T01:35:58.811178+0000","last_active":"2024-03-18T01:36:57.202438+0000","last_peered":"2024-03-18T01:36:57.202438+0000","last_clean":"2024-03-18T01:36:57.202438+0000","last_became_active":"2024-03-18T01:35:58.810764+0000","last_became_peered":"2024-03-18T01:35:58.810764+0000","last_unstale":"2024-03-18T01:36:57.202438+0000","last_undegraded":"2024-03-18T01:36:57.202438+0000","last_fullsized":"2024-03-18T01:36:57.202438+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:49.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819415+0000","last_change":"2024-03-18T01:35:01.152767+0000","last_active":"2024-03-18T01:35:58.819415+0000","last_peered":"2024-03-18T01:35:58.819415+0000","last_clean":"2024-03-18T01:35:58.819415+0000","last_became_active":"2024-03-18T01:35:01.150619+0000","last_became_peered":"2024-03-18T01:35:01.150619+0000","last_unstale":"2024-03-18T01:35:58.819415+0000","last_undegraded":"2024-03-18T01:35:58.819415+0000","last_fullsized":"2024-03-18T01:35:58.819415+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:57:28.258251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191740+0000","last_change":"2024-03-18T01:34:56.650541+0000","last_active":"2024-03-18T01:36:57.191740+0000","last_peered":"2024-03-18T01:36:57.191740+0000","last_clean":"2024-03-18T01:36:57.191740+0000","last_became_active":"2024-03-18T01:34:56.650072+0000","last_became_peered":"2024-03-18T01:34:56.650072+0000","last_unstale":"2024-03-18T01:36:57.191740+0000","last_undegraded":"2024-03-18T01:36:57.191740+0000","last_fullsized":"2024-03-18T01:36:57.191740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:51.224883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.813728+0000","last_change":"2024-03-18T01:34:55.664297+0000","last_active":"2024-03-18T01:36:57.813728+0000","last_peered":"2024-03-18T01:36:57.813728+0000","last_clean":"2024-03-18T01:36:57.813728+0000","last_became_active":"2024-03-18T01:34:55.663938+0000","last_became_peered":"2024-03-18T01:34:55.663938+0000","last_unstale":"2024-03-18T01:36:57.813728+0000","last_undegraded":"2024-03-18T01:36:57.813728+0000","last_fullsized":"2024-03-18T01:36:57.813728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:51.771987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806662+0000","last_change":"2024-03-18T01:35:01.141736+0000","last_active":"2024-03-18T01:35:58.806662+0000","last_peered":"2024-03-18T01:35:58.806662+0000","last_clean":"2024-03-18T01:35:58.806662+0000","last_became_active":"2024-03-18T01:35:01.141381+0000","last_became_peered":"2024-03-18T01:35:01.141381+0000","last_unstale":"2024-03-18T01:35:58.806662+0000","last_undegraded":"2024-03-18T01:35:58.806662+0000","last_fullsized":"2024-03-18T01:35:58.806662+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:25:04.190250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.212379+0000","last_change":"2024-03-18T01:34:56.666439+0000","last_active":"2024-03-18T01:36:57.212379+0000","last_peered":"2024-03-18T01:36:57.212379+0000","last_clean":"2024-03-18T01:36:57.212379+0000","last_became_active":"2024-03-18T01:34:56.664556+0000","last_became_peered":"2024-03-18T01:34:56.664556+0000","last_unstale":"2024-03-18T01:36:57.212379+0000","last_undegraded":"2024-03-18T01:36:57.212379+0000","last_fullsized":"2024-03-18T01:36:57.212379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:08:48.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802167+0000","last_change":"2024-03-18T01:34:55.656484+0000","last_active":"2024-03-18T01:35:58.802167+0000","last_peered":"2024-03-18T01:35:58.802167+0000","last_clean":"2024-03-18T01:35:58.802167+0000","last_became_active":"2024-03-18T01:34:55.656125+0000","last_became_peered":"2024-03-18T01:34:55.656125+0000","last_unstale":"2024-03-18T01:35:58.802167+0000","last_undegraded":"2024-03-18T01:35:58.802167+0000","last_fullsized":"2024-03-18T01:35:58.802167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:58:25.944403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806027+0000","last_change":"2024-03-18T01:33:35.824651+0000","last_active":"2024-03-18T01:35:58.806027+0000","last_peered":"2024-03-18T01:35:58.806027+0000","last_clean":"2024-03-18T01:35:58.806027+0000","last_became_active":"2024-03-18T01:33:35.824106+0000","last_became_peered":"2024-03-18T01:33:35.824106+0000","last_unstale":"2024-03-18T01:35:58.806027+0000","last_undegraded":"2024-03-18T01:35:58.806027+0000","last_fullsized":"2024-03-18T01:35:58.806027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_clean_scrub_stamp":"2024-03-18T01:29:41.724134+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:06.422316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808469+0000","last_change":"2024-03-18T01:35:01.157735+0000","last_active":"2024-03-18T01:35:58.808469+0000","last_peered":"2024-03-18T01:35:58.808469+0000","last_clean":"2024-03-18T01:35:58.808469+0000","last_became_active":"2024-03-18T01:35:01.156954+0000","last_became_peered":"2024-03-18T01:35:01.156954+0000","last_unstale":"2024-03-18T01:35:58.808469+0000","last_undegraded":"2024-03-18T01:35:58.808469+0000","last_fullsized":"2024-03-18T01:35:58.808469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:00.106809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.189363+0000","last_change":"2024-03-18T01:34:56.646808+0000","last_active":"2024-03-18T01:36:57.189363+0000","last_peered":"2024-03-18T01:36:57.189363+0000","last_clean":"2024-03-18T01:36:57.189363+0000","last_became_active":"2024-03-18T01:34:56.646437+0000","last_became_peered":"2024-03-18T01:34:56.646437+0000","last_unstale":"2024-03-18T01:36:57.189363+0000","last_undegraded":"2024-03-18T01:36:57.189363+0000","last_fullsized":"2024-03-18T01:36:57.189363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:46:09.703889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806136+0000","last_change":"2024-03-18T01:34:55.645365+0000","last_active":"2024-03-18T01:35:58.806136+0000","last_peered":"2024-03-18T01:35:58.806136+0000","last_clean":"2024-03-18T01:35:58.806136+0000","last_became_active":"2024-03-18T01:34:55.645018+0000","last_became_peered":"2024-03-18T01:34:55.645018+0000","last_unstale":"2024-03-18T01:35:58.806136+0000","last_undegraded":"2024-03-18T01:35:58.806136+0000","last_fullsized":"2024-03-18T01:35:58.806136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:08:31.869328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804906+0000","last_change":"2024-03-18T01:35:01.147294+0000","last_active":"2024-03-18T01:35:58.804906+0000","last_peered":"2024-03-18T01:35:58.804906+0000","last_clean":"2024-03-18T01:35:58.804906+0000","last_became_active":"2024-03-18T01:35:01.146858+0000","last_became_peered":"2024-03-18T01:35:01.146858+0000","last_unstale":"2024-03-18T01:35:58.804906+0000","last_undegraded":"2024-03-18T01:35:58.804906+0000","last_fullsized":"2024-03-18T01:35:58.804906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:45:24.010746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802584+0000","last_change":"2024-03-18T01:34:55.658110+0000","last_active":"2024-03-18T01:35:58.802584+0000","last_peered":"2024-03-18T01:35:58.802584+0000","last_clean":"2024-03-18T01:35:58.802584+0000","last_became_active":"2024-03-18T01:34:55.657802+0000","last_became_peered":"2024-03-18T01:34:55.657802+0000","last_unstale":"2024-03-18T01:35:58.802584+0000","last_undegraded":"2024-03-18T01:35:58.802584+0000","last_fullsized":"2024-03-18T01:35:58.802584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:34.264115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.204986+0000","last_change":"2024-03-18T01:34:56.656689+0000","last_active":"2024-03-18T01:36:57.204986+0000","last_peered":"2024-03-18T01:36:57.204986+0000","last_clean":"2024-03-18T01:36:57.204986+0000","last_became_active":"2024-03-18T01:34:56.655653+0000","last_became_peered":"2024-03-18T01:34:56.655653+0000","last_unstale":"2024-03-18T01:36:57.204986+0000","last_undegraded":"2024-03-18T01:36:57.204986+0000","last_fullsized":"2024-03-18T01:36:57.204986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:58.370904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816471+0000","last_change":"2024-03-18T01:35:01.137419+0000","last_active":"2024-03-18T01:35:58.816471+0000","last_peered":"2024-03-18T01:35:58.816471+0000","last_clean":"2024-03-18T01:35:58.816471+0000","last_became_active":"2024-03-18T01:35:01.137033+0000","last_became_peered":"2024-03-18T01:35:01.137033+0000","last_unstale":"2024-03-18T01:35:58.816471+0000","last_undegraded":"2024-03-18T01:35:58.816471+0000","last_fullsized":"2024-03-18T01:35:58.816471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:13:48.382329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'14","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811325+0000","last_change":"2024-03-18T01:34:55.663134+0000","last_active":"2024-03-18T01:35:58.811325+0000","last_peered":"2024-03-18T01:35:58.811325+0000","last_clean":"2024-03-18T01:35:58.811325+0000","last_became_active":"2024-03-18T01:34:55.662083+0000","last_became_peered":"2024-03-18T01:34:55.662083+0000","last_unstale":"2024-03-18T01:35:58.811325+0000","last_undegraded":"2024-03-18T01:35:58.811325+0000","last_fullsized":"2024-03-18T01:35:58.811325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:01:09.186599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214781+0000","last_change":"2024-03-18T01:34:56.664761+0000","last_active":"2024-03-18T01:36:57.214781+0000","last_peered":"2024-03-18T01:36:57.214781+0000","last_clean":"2024-03-18T01:36:57.214781+0000","last_became_active":"2024-03-18T01:34:56.663657+0000","last_became_peered":"2024-03-18T01:34:56.663657+0000","last_unstale":"2024-03-18T01:36:57.214781+0000","last_undegraded":"2024-03-18T01:36:57.214781+0000","last_fullsized":"2024-03-18T01:36:57.214781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:26:53.075000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809944+0000","last_change":"2024-03-18T01:35:01.133002+0000","last_active":"2024-03-18T01:35:58.809944+0000","last_peered":"2024-03-18T01:35:58.809944+0000","last_clean":"2024-03-18T01:35:58.809944+0000","last_became_active":"2024-03-18T01:35:01.132542+0000","last_became_peered":"2024-03-18T01:35:01.132542+0000","last_unstale":"2024-03-18T01:35:58.809944+0000","last_undegraded":"2024-03-18T01:35:58.809944+0000","last_fullsized":"2024-03-18T01:35:58.809944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:23.823409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801907+0000","last_change":"2024-03-18T01:34:55.648924+0000","last_active":"2024-03-18T01:35:58.801907+0000","last_peered":"2024-03-18T01:35:58.801907+0000","last_clean":"2024-03-18T01:35:58.801907+0000","last_became_active":"2024-03-18T01:34:55.648548+0000","last_became_peered":"2024-03-18T01:34:55.648548+0000","last_unstale":"2024-03-18T01:35:58.801907+0000","last_undegraded":"2024-03-18T01:35:58.801907+0000","last_fullsized":"2024-03-18T01:35:58.801907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:26:38.916856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.221556+0000","last_change":"2024-03-18T01:34:56.663142+0000","last_active":"2024-03-18T01:36:57.221556+0000","last_peered":"2024-03-18T01:36:57.221556+0000","last_clean":"2024-03-18T01:36:57.221556+0000","last_became_active":"2024-03-18T01:34:56.660256+0000","last_became_peered":"2024-03-18T01:34:56.660256+0000","last_unstale":"2024-03-18T01:36:57.221556+0000","last_undegraded":"2024-03-18T01:36:57.221556+0000","last_fullsized":"2024-03-18T01:36:57.221556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:59.503277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801675+0000","last_change":"2024-03-18T01:35:01.136186+0000","last_active":"2024-03-18T01:35:58.801675+0000","last_peered":"2024-03-18T01:35:58.801675+0000","last_clean":"2024-03-18T01:35:58.801675+0000","last_became_active":"2024-03-18T01:35:01.135803+0000","last_became_peered":"2024-03-18T01:35:01.135803+0000","last_unstale":"2024-03-18T01:35:58.801675+0000","last_undegraded":"2024-03-18T01:35:58.801675+0000","last_fullsized":"2024-03-18T01:35:58.801675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:20.027448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806421+0000","last_change":"2024-03-18T01:34:55.663090+0000","last_active":"2024-03-18T01:35:58.806421+0000","last_peered":"2024-03-18T01:35:58.806421+0000","last_clean":"2024-03-18T01:35:58.806421+0000","last_became_active":"2024-03-18T01:34:55.662774+0000","last_became_peered":"2024-03-18T01:34:55.662774+0000","last_unstale":"2024-03-18T01:35:58.806421+0000","last_undegraded":"2024-03-18T01:35:58.806421+0000","last_fullsized":"2024-03-18T01:35:58.806421+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:30:35.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.154332+0000","last_change":"2024-03-18T01:34:56.639509+0000","last_active":"2024-03-18T01:36:57.154332+0000","last_peered":"2024-03-18T01:36:57.154332+0000","last_clean":"2024-03-18T01:36:57.154332+0000","last_became_active":"2024-03-18T01:34:56.639144+0000","last_became_peered":"2024-03-18T01:34:56.639144+0000","last_unstale":"2024-03-18T01:36:57.154332+0000","last_undegraded":"2024-03-18T01:36:57.154332+0000","last_fullsized":"2024-03-18T01:36:57.154332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:12:36.220793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812718+0000","last_change":"2024-03-18T01:35:01.141762+0000","last_active":"2024-03-18T01:35:58.812718+0000","last_peered":"2024-03-18T01:35:58.812718+0000","last_clean":"2024-03-18T01:35:58.812718+0000","last_became_active":"2024-03-18T01:35:01.141219+0000","last_became_peered":"2024-03-18T01:35:01.141219+0000","last_unstale":"2024-03-18T01:35:58.812718+0000","last_undegraded":"2024-03-18T01:35:58.812718+0000","last_fullsized":"2024-03-18T01:35:58.812718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:55.839120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.815857+0000","last_change":"2024-03-18T01:34:55.662530+0000","last_active":"2024-03-18T01:35:58.815857+0000","last_peered":"2024-03-18T01:35:58.815857+0000","last_clean":"2024-03-18T01:35:58.815857+0000","last_became_active":"2024-03-18T01:34:55.661178+0000","last_became_peered":"2024-03-18T01:34:55.661178+0000","last_unstale":"2024-03-18T01:35:58.815857+0000","last_undegraded":"2024-03-18T01:35:58.815857+0000","last_fullsized":"2024-03-18T01:35:58.815857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:47:16.678553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190194+0000","last_change":"2024-03-18T01:34:56.660936+0000","last_active":"2024-03-18T01:36:57.190194+0000","last_peered":"2024-03-18T01:36:57.190194+0000","last_clean":"2024-03-18T01:36:57.190194+0000","last_became_active":"2024-03-18T01:34:56.660513+0000","last_became_peered":"2024-03-18T01:34:56.660513+0000","last_unstale":"2024-03-18T01:36:57.190194+0000","last_undegraded":"2024-03-18T01:36:57.190194+0000","last_fullsized":"2024-03-18T01:36:57.190194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:35:21.037366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802003+0000","last_change":"2024-03-18T01:35:01.136914+0000","last_active":"2024-03-18T01:35:58.802003+0000","last_peered":"2024-03-18T01:35:58.802003+0000","last_clean":"2024-03-18T01:35:58.802003+0000","last_became_active":"2024-03-18T01:35:01.136480+0000","last_became_peered":"2024-03-18T01:35:01.136480+0000","last_unstale":"2024-03-18T01:35:58.802003+0000","last_undegraded":"2024-03-18T01:35:58.802003+0000","last_fullsized":"2024-03-18T01:35:58.802003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:49.262255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809429+0000","last_change":"2024-03-18T01:34:55.654938+0000","last_active":"2024-03-18T01:35:58.809429+0000","last_peered":"2024-03-18T01:35:58.809429+0000","last_clean":"2024-03-18T01:35:58.809429+0000","last_became_active":"2024-03-18T01:34:55.654611+0000","last_became_peered":"2024-03-18T01:34:55.654611+0000","last_unstale":"2024-03-18T01:35:58.809429+0000","last_undegraded":"2024-03-18T01:35:58.809429+0000","last_fullsized":"2024-03-18T01:35:58.809429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:23.210205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.219833+0000","last_change":"2024-03-18T01:34:56.650731+0000","last_active":"2024-03-18T01:36:57.219833+0000","last_peered":"2024-03-18T01:36:57.219833+0000","last_clean":"2024-03-18T01:36:57.219833+0000","last_became_active":"2024-03-18T01:34:56.650363+0000","last_became_peered":"2024-03-18T01:34:56.650363+0000","last_unstale":"2024-03-18T01:36:57.219833+0000","last_undegraded":"2024-03-18T01:36:57.219833+0000","last_fullsized":"2024-03-18T01:36:57.219833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:09:50.305019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801649+0000","last_change":"2024-03-18T01:35:01.135101+0000","last_active":"2024-03-18T01:35:58.801649+0000","last_peered":"2024-03-18T01:35:58.801649+0000","last_clean":"2024-03-18T01:35:58.801649+0000","last_became_active":"2024-03-18T01:35:01.134690+0000","last_became_peered":"2024-03-18T01:35:01.134690+0000","last_unstale":"2024-03-18T01:35:58.801649+0000","last_undegraded":"2024-03-18T01:35:58.801649+0000","last_fullsized":"2024-03-18T01:35:58.801649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:33.381629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802112+0000","last_change":"2024-03-18T01:34:55.648108+0000","last_active":"2024-03-18T01:35:58.802112+0000","last_peered":"2024-03-18T01:35:58.802112+0000","last_clean":"2024-03-18T01:35:58.802112+0000","last_became_active":"2024-03-18T01:34:55.647754+0000","last_became_peered":"2024-03-18T01:34:55.647754+0000","last_unstale":"2024-03-18T01:35:58.802112+0000","last_undegraded":"2024-03-18T01:35:58.802112+0000","last_fullsized":"2024-03-18T01:35:58.802112+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:05:48.508015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806678+0000","last_change":"2024-03-18T01:34:56.651589+0000","last_active":"2024-03-18T01:35:58.806678+0000","last_peered":"2024-03-18T01:35:58.806678+0000","last_clean":"2024-03-18T01:35:58.806678+0000","last_became_active":"2024-03-18T01:34:56.650929+0000","last_became_peered":"2024-03-18T01:34:56.650929+0000","last_unstale":"2024-03-18T01:35:58.806678+0000","last_undegraded":"2024-03-18T01:35:58.806678+0000","last_fullsized":"2024-03-18T01:35:58.806678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:43.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806777+0000","last_change":"2024-03-18T01:35:01.148347+0000","last_active":"2024-03-18T01:35:58.806777+0000","last_peered":"2024-03-18T01:35:58.806777+0000","last_clean":"2024-03-18T01:35:58.806777+0000","last_became_active":"2024-03-18T01:35:01.147936+0000","last_became_peered":"2024-03-18T01:35:01.147936+0000","last_unstale":"2024-03-18T01:35:58.806777+0000","last_undegraded":"2024-03-18T01:35:58.806777+0000","last_fullsized":"2024-03-18T01:35:58.806777+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:12.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:59.738160+0000","last_change":"2024-03-18T01:34:55.653625+0000","last_active":"2024-03-18T01:36:59.738160+0000","last_peered":"2024-03-18T01:36:59.738160+0000","last_clean":"2024-03-18T01:36:59.738160+0000","last_became_active":"2024-03-18T01:34:55.653186+0000","last_became_peered":"2024-03-18T01:34:55.653186+0000","last_unstale":"2024-03-18T01:36:59.738160+0000","last_undegraded":"2024-03-18T01:36:59.738160+0000","last_fullsized":"2024-03-18T01:36:59.738160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:10:51.193296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806920+0000","last_change":"2024-03-18T01:34:56.652703+0000","last_active":"2024-03-18T01:35:58.806920+0000","last_peered":"2024-03-18T01:35:58.806920+0000","last_clean":"2024-03-18T01:35:58.806920+0000","last_became_active":"2024-03-18T01:34:56.651768+0000","last_became_peered":"2024-03-18T01:34:56.651768+0000","last_unstale":"2024-03-18T01:35:58.806920+0000","last_undegraded":"2024-03-18T01:35:58.806920+0000","last_fullsized":"2024-03-18T01:35:58.806920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:59.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809291+0000","last_change":"2024-03-18T01:35:01.144900+0000","last_active":"2024-03-18T01:35:58.809291+0000","last_peered":"2024-03-18T01:35:58.809291+0000","last_clean":"2024-03-18T01:35:58.809291+0000","last_became_active":"2024-03-18T01:35:01.144431+0000","last_became_peered":"2024-03-18T01:35:01.144431+0000","last_unstale":"2024-03-18T01:35:58.809291+0000","last_undegraded":"2024-03-18T01:35:58.809291+0000","last_fullsized":"2024-03-18T01:35:58.809291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:36.199904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808737+0000","last_change":"2024-03-18T01:34:55.661259+0000","last_active":"2024-03-18T01:35:58.808737+0000","last_peered":"2024-03-18T01:35:58.808737+0000","last_clean":"2024-03-18T01:35:58.808737+0000","last_became_active":"2024-03-18T01:34:55.660964+0000","last_became_peered":"2024-03-18T01:34:55.660964+0000","last_unstale":"2024-03-18T01:35:58.808737+0000","last_undegraded":"2024-03-18T01:35:58.808737+0000","last_fullsized":"2024-03-18T01:35:58.808737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:14.723905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807058+0000","last_change":"2024-03-18T01:34:56.663075+0000","last_active":"2024-03-18T01:35:58.807058+0000","last_peered":"2024-03-18T01:35:58.807058+0000","last_clean":"2024-03-18T01:35:58.807058+0000","last_became_active":"2024-03-18T01:34:56.662458+0000","last_became_peered":"2024-03-18T01:34:56.662458+0000","last_unstale":"2024-03-18T01:35:58.807058+0000","last_undegraded":"2024-03-18T01:35:58.807058+0000","last_fullsized":"2024-03-18T01:35:58.807058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:31:31.837493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806819+0000","last_change":"2024-03-18T01:35:01.133317+0000","last_active":"2024-03-18T01:35:58.806819+0000","last_peered":"2024-03-18T01:35:58.806819+0000","last_clean":"2024-03-18T01:35:58.806819+0000","last_became_active":"2024-03-18T01:35:01.131798+0000","last_became_peered":"2024-03-18T01:35:01.131798+0000","last_unstale":"2024-03-18T01:35:58.806819+0000","last_undegraded":"2024-03-18T01:35:58.806819+0000","last_fullsized":"2024-03-18T01:35:58.806819+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:55.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819454+0000","last_change":"2024-03-18T01:34:55.660887+0000","last_active":"2024-03-18T01:35:58.819454+0000","last_peered":"2024-03-18T01:35:58.819454+0000","last_clean":"2024-03-18T01:35:58.819454+0000","last_became_active":"2024-03-18T01:34:55.660524+0000","last_became_peered":"2024-03-18T01:34:55.660524+0000","last_unstale":"2024-03-18T01:35:58.819454+0000","last_undegraded":"2024-03-18T01:35:58.819454+0000","last_fullsized":"2024-03-18T01:35:58.819454+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:24.816951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807252+0000","last_change":"2024-03-18T01:34:56.654107+0000","last_active":"2024-03-18T01:35:58.807252+0000","last_peered":"2024-03-18T01:35:58.807252+0000","last_clean":"2024-03-18T01:35:58.807252+0000","last_became_active":"2024-03-18T01:34:56.653742+0000","last_became_peered":"2024-03-18T01:34:56.653742+0000","last_unstale":"2024-03-18T01:35:58.807252+0000","last_undegraded":"2024-03-18T01:35:58.807252+0000","last_fullsized":"2024-03-18T01:35:58.807252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:58:46.093589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190644+0000","last_change":"2024-03-18T01:34:56.647818+0000","last_active":"2024-03-18T01:36:57.190644+0000","last_peered":"2024-03-18T01:36:57.190644+0000","last_clean":"2024-03-18T01:36:57.190644+0000","last_became_active":"2024-03-18T01:34:56.647174+0000","last_became_peered":"2024-03-18T01:34:56.647174+0000","last_unstale":"2024-03-18T01:36:57.190644+0000","last_undegraded":"2024-03-18T01:36:57.190644+0000","last_fullsized":"2024-03-18T01:36:57.190644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:55.656083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191353+0000","last_change":"2024-03-18T01:34:56.643221+0000","last_active":"2024-03-18T01:36:57.191353+0000","last_peered":"2024-03-18T01:36:57.191353+0000","last_clean":"2024-03-18T01:36:57.191353+0000","last_became_active":"2024-03-18T01:34:56.642871+0000","last_became_peered":"2024-03-18T01:34:56.642871+0000","last_unstale":"2024-03-18T01:36:57.191353+0000","last_undegraded":"2024-03-18T01:36:57.191353+0000","last_fullsized":"2024-03-18T01:36:57.191353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:23:25.106060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213467+0000","last_change":"2024-03-18T01:34:56.656227+0000","last_active":"2024-03-18T01:36:57.213467+0000","last_peered":"2024-03-18T01:36:57.213467+0000","last_clean":"2024-03-18T01:36:57.213467+0000","last_became_active":"2024-03-18T01:34:56.655859+0000","last_became_peered":"2024-03-18T01:34:56.655859+0000","last_unstale":"2024-03-18T01:36:57.213467+0000","last_undegraded":"2024-03-18T01:36:57.213467+0000","last_fullsized":"2024-03-18T01:36:57.213467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:51:54.424342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214947+0000","last_change":"2024-03-18T01:34:56.664655+0000","last_active":"2024-03-18T01:36:57.214947+0000","last_peered":"2024-03-18T01:36:57.214947+0000","last_clean":"2024-03-18T01:36:57.214947+0000","last_became_active":"2024-03-18T01:34:56.662526+0000","last_became_peered":"2024-03-18T01:34:56.662526+0000","last_unstale":"2024-03-18T01:36:57.214947+0000","last_undegraded":"2024-03-18T01:36:57.214947+0000","last_fullsized":"2024-03-18T01:36:57.214947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:45:06.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214683+0000","last_change":"2024-03-18T01:34:56.657045+0000","last_active":"2024-03-18T01:36:57.214683+0000","last_peered":"2024-03-18T01:36:57.214683+0000","last_clean":"2024-03-18T01:36:57.214683+0000","last_became_active":"2024-03-18T01:34:56.656339+0000","last_became_peered":"2024-03-18T01:34:56.656339+0000","last_unstale":"2024-03-18T01:36:57.214683+0000","last_undegraded":"2024-03-18T01:36:57.214683+0000","last_fullsized":"2024-03-18T01:36:57.214683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:41.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.184577+0000","last_change":"2024-03-18T01:34:56.647626+0000","last_active":"2024-03-18T01:36:57.184577+0000","last_peered":"2024-03-18T01:36:57.184577+0000","last_clean":"2024-03-18T01:36:57.184577+0000","last_became_active":"2024-03-18T01:34:56.647216+0000","last_became_peered":"2024-03-18T01:34:56.647216+0000","last_unstale":"2024-03-18T01:36:57.184577+0000","last_undegraded":"2024-03-18T01:36:57.184577+0000","last_fullsized":"2024-03-18T01:36:57.184577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:12:33.330925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.180015+0000","last_change":"2024-03-18T01:34:56.645091+0000","last_active":"2024-03-18T01:36:57.180015+0000","last_peered":"2024-03-18T01:36:57.180015+0000","last_clean":"2024-03-18T01:36:57.180015+0000","last_became_active":"2024-03-18T01:34:56.644725+0000","last_became_peered":"2024-03-18T01:34:56.644725+0000","last_unstale":"2024-03-18T01:36:57.180015+0000","last_undegraded":"2024-03-18T01:36:57.180015+0000","last_fullsized":"2024-03-18T01:36:57.180015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211315+0000","last_change":"2024-03-18T01:34:56.651282+0000","last_active":"2024-03-18T01:36:57.211315+0000","last_peered":"2024-03-18T01:36:57.211315+0000","last_clean":"2024-03-18T01:36:57.211315+0000","last_became_active":"2024-03-18T01:34:56.650872+0000","last_became_peered":"2024-03-18T01:34:56.650872+0000","last_unstale":"2024-03-18T01:36:57.211315+0000","last_undegraded":"2024-03-18T01:36:57.211315+0000","last_fullsized":"2024-03-18T01:36:57.211315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:14:45.106848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211037+0000","last_change":"2024-03-18T01:34:56.651347+0000","last_active":"2024-03-18T01:36:57.211037+0000","last_peered":"2024-03-18T01:36:57.211037+0000","last_clean":"2024-03-18T01:36:57.211037+0000","last_became_active":"2024-03-18T01:34:56.650935+0000","last_became_peered":"2024-03-18T01:34:56.650935+0000","last_unstale":"2024-03-18T01:36:57.211037+0000","last_undegraded":"2024-03-18T01:36:57.211037+0000","last_fullsized":"2024-03-18T01:36:57.211037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:41.849883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211551+0000","last_change":"2024-03-18T01:34:56.667333+0000","last_active":"2024-03-18T01:36:57.211551+0000","last_peered":"2024-03-18T01:36:57.211551+0000","last_clean":"2024-03-18T01:36:57.211551+0000","last_became_active":"2024-03-18T01:34:56.666845+0000","last_became_peered":"2024-03-18T01:34:56.666845+0000","last_unstale":"2024-03-18T01:36:57.211551+0000","last_undegraded":"2024-03-18T01:36:57.211551+0000","last_fullsized":"2024-03-18T01:36:57.211551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:49:23.813455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213487+0000","last_change":"2024-03-18T01:34:56.662677+0000","last_active":"2024-03-18T01:36:57.213487+0000","last_peered":"2024-03-18T01:36:57.213487+0000","last_clean":"2024-03-18T01:36:57.213487+0000","last_became_active":"2024-03-18T01:34:56.662306+0000","last_became_peered":"2024-03-18T01:34:56.662306+0000","last_unstale":"2024-03-18T01:36:57.213487+0000","last_undegraded":"2024-03-18T01:36:57.213487+0000","last_fullsized":"2024-03-18T01:36:57.213487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:53:56.868984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216369+0000","last_change":"2024-03-18T01:34:56.657721+0000","last_active":"2024-03-18T01:36:57.216369+0000","last_peered":"2024-03-18T01:36:57.216369+0000","last_clean":"2024-03-18T01:36:57.216369+0000","last_became_active":"2024-03-18T01:34:56.657072+0000","last_became_peered":"2024-03-18T01:34:56.657072+0000","last_unstale":"2024-03-18T01:36:57.216369+0000","last_undegraded":"2024-03-18T01:36:57.216369+0000","last_fullsized":"2024-03-18T01:36:57.216369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:21.120391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190883+0000","last_change":"2024-03-18T01:34:56.647327+0000","last_active":"2024-03-18T01:36:57.190883+0000","last_peered":"2024-03-18T01:36:57.190883+0000","last_clean":"2024-03-18T01:36:57.190883+0000","last_became_active":"2024-03-18T01:34:56.646910+0000","last_became_peered":"2024-03-18T01:34:56.646910+0000","last_unstale":"2024-03-18T01:36:57.190883+0000","last_undegraded":"2024-03-18T01:36:57.190883+0000","last_fullsized":"2024-03-18T01:36:57.190883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:37:25.970272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69275,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":924,"num_read_kb":538,"num_write":248,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":207684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":502083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1724416,"data_stored":1150712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":501159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1650688,"data_stored":1098238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1097728,"data_stored":559969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":560370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":501473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1093429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:37:03.220 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T01:37:03.220 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T01:37:03.220 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T01:37:03.220 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T01:37:03.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:03 smithi007 ceph-mon[31996]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:37:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:03 smithi203 ceph-mon[36345]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:37:03.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:03 smithi046 ceph-mon[36108]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:37:03.723 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:04.124 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:04 smithi007 ceph-mon[31996]: pgmap v384: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T01:37:04.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:04 smithi203 ceph-mon[36345]: pgmap v384: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T01:37:04.518 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:04 smithi046 ceph-mon[36108]: pgmap v384: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T01:37:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 -- 172.21.15.7:0/3235497971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a201044e0 msgr2=0x7f2a20104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 --2- 172.21.15.7:0/3235497971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a201044e0 0x7f2a20104960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0099c0 tx=0x7f2a0c009cd0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 -- 172.21.15.7:0/3235497971 shutdown_connections 2024-03-18T01:37:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 --2- 172.21.15.7:0/3235497971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a20104ea0 0x7f2a20111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 --2- 172.21.15.7:0/3235497971 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a2010a590 0x7f2a2010a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 --2- 172.21.15.7:0/3235497971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a201044e0 0x7f2a20104960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 -- 172.21.15.7:0/3235497971 >> 172.21.15.7:0/3235497971 conn(0x7f2a200fffb0 msgr2=0x7f2a201023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 -- 172.21.15.7:0/3235497971 shutdown_connections 2024-03-18T01:37:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.077+0000 7f2a25a00700 1 -- 172.21.15.7:0/3235497971 wait complete. 2024-03-18T01:37:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.079+0000 7f2a25a00700 1 Processor -- start 2024-03-18T01:37:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.079+0000 7f2a25a00700 1 -- start start 2024-03-18T01:37:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.079+0000 7f2a25a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a201044e0 0x7f2a201a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a25a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20104ea0 0x7f2a201a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a25a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a2010a590 0x7f2a2019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a25a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f2a201147d0 con 0x7f2a201044e0 2024-03-18T01:37:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a25a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2a20114650 con 0x7f2a20104ea0 2024-03-18T01:37:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a25a00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a20114950 con 0x7f2a2010a590 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1e000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20104ea0 0x7f2a201a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1e000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20104ea0 0x7f2a201a0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53562/0 (socket says 172.21.15.7:53562) 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1e000700 1 -- 172.21.15.7:0/559111902 learned_addr learned my addr 172.21.15.7:0/559111902 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1f400700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a2010a590 0x7f2a2019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1ea00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a201044e0 0x7f2a201a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1e000700 1 -- 172.21.15.7:0/559111902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a2010a590 msgr2=0x7f2a2019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1e000700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a2010a590 0x7f2a2019aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1e000700 1 -- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a201044e0 msgr2=0x7f2a201a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1e000700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a201044e0 0x7f2a201a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1e000700 1 -- 172.21.15.7:0/559111902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2019b170 con 0x7f2a20104ea0 2024-03-18T01:37:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.080+0000 7f2a1f400700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a2010a590 0x7f2a2019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.081+0000 7f2a1e000700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20104ea0 0x7f2a201a0d90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c006010 tx=0x7f2a0c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.081+0000 7f2a17400700 1 -- 172.21.15.7:0/559111902 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c003680 con 0x7f2a20104ea0 2024-03-18T01:37:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.081+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2019b400 con 0x7f2a20104ea0 2024-03-18T01:37:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.081+0000 7f2a17400700 1 -- 172.21.15.7:0/559111902 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a0c0055b0 con 0x7f2a20104ea0 2024-03-18T01:37:06.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.081+0000 7f2a17400700 1 -- 172.21.15.7:0/559111902 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c003cc0 con 0x7f2a20104ea0 2024-03-18T01:37:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.081+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a201a7720 con 0x7f2a20104ea0 2024-03-18T01:37:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.082+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a00005330 con 0x7f2a20104ea0 2024-03-18T01:37:06.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.084+0000 7f2a17400700 1 -- 172.21.15.7:0/559111902 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f2a0c00fb80 con 0x7f2a20104ea0 2024-03-18T01:37:06.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.087+0000 7f2a17400700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2a040a2a30 0x7f2a040a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:06.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.087+0000 7f2a17400700 1 -- 172.21.15.7:0/559111902 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2a0c0cf940 con 0x7f2a20104ea0 2024-03-18T01:37:06.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.087+0000 7f2a1ea00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2a040a2a30 0x7f2a040a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.088+0000 7f2a1ea00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2a040a2a30 0x7f2a040a4ef0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2a2019bb50 tx=0x7f2a10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:06.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.090+0000 7f2a17400700 1 -- 172.21.15.7:0/559111902 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a0c013070 con 0x7f2a20104ea0 2024-03-18T01:37:06.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.372+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2a00005620 con 0x7f2a20104ea0 2024-03-18T01:37:06.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.373+0000 7f2a17400700 1 -- 172.21.15.7:0/559111902 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f2a0c09c6b0 con 0x7f2a20104ea0 2024-03-18T01:37:06.373 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:06.373 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T01:37:06.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.376+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2a040a2a30 msgr2=0x7f2a040a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:06.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.376+0000 7f2a25a00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2a040a2a30 0x7f2a040a4ef0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2a2019bb50 tx=0x7f2a10009260 comp rx=0 tx=0).stop 2024-03-18T01:37:06.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.376+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20104ea0 msgr2=0x7f2a201a0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.376+0000 7f2a25a00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20104ea0 0x7f2a201a0d90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c006010 tx=0x7f2a0c00f690 comp rx=0 tx=0).stop 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.377+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 shutdown_connections 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.377+0000 7f2a25a00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f2a040a2a30 0x7f2a040a4ef0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.377+0000 7f2a25a00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a2010a590 0x7f2a2019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.377+0000 7f2a25a00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2a201044e0 0x7f2a201a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.377+0000 7f2a25a00700 1 --2- 172.21.15.7:0/559111902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a20104ea0 0x7f2a201a0d90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.377+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 >> 172.21.15.7:0/559111902 conn(0x7f2a200fffb0 msgr2=0x7f2a201014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.377+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 shutdown_connections 2024-03-18T01:37:06.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:06.377+0000 7f2a25a00700 1 -- 172.21.15.7:0/559111902 wait complete. 2024-03-18T01:37:06.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:06 smithi007 ceph-mon[31996]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:06.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:06 smithi203 ceph-mon[36345]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:06.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:06 smithi046 ceph-mon[36108]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:07.186 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T01:37:07.186 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:37:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.459+0000 7fd74f200640 1 -- 172.21.15.7:0/2824620356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748107370 msgr2=0x7fd748109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.459+0000 7fd74f200640 1 --2- 172.21.15.7:0/2824620356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748107370 0x7fd748109760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd744009be0 tx=0x7fd74402bef0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.459+0000 7fd74f200640 1 -- 172.21.15.7:0/2824620356 shutdown_connections 2024-03-18T01:37:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.459+0000 7fd74f200640 1 --2- 172.21.15.7:0/2824620356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748107370 0x7fd748109760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.459+0000 7fd74f200640 1 --2- 172.21.15.7:0/2824620356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748104a40 0x7fd748106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.459+0000 7fd74f200640 1 --2- 172.21.15.7:0/2824620356 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd748102110 0x7fd748104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.459+0000 7fd74f200640 1 -- 172.21.15.7:0/2824620356 >> 172.21.15.7:0/2824620356 conn(0x7fd7480fbaa0 msgr2=0x7fd7480fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:07.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.460+0000 7fd74f200640 1 -- 172.21.15.7:0/2824620356 shutdown_connections 2024-03-18T01:37:07.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.460+0000 7fd74f200640 1 -- 172.21.15.7:0/2824620356 wait complete. 2024-03-18T01:37:07.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74f200640 1 Processor -- start 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74f200640 1 -- start start 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748102110 0x7fd7481999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74f200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748104a40 0x7fd748199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74f200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd748107370 0x7fd7481a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74f200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810d2c0 con 0x7fd748107370 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810d140 con 0x7fd748102110 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74f200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810d440 con 0x7fd748104a40 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748104a40 0x7fd748199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748104a40 0x7fd748199f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39306/0 (socket says 172.21.15.7:39306) 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74d400640 1 -- 172.21.15.7:0/1803778089 learned_addr learned my addr 172.21.15.7:0/1803778089 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.461+0000 7fd74de00640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748102110 0x7fd7481999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd74e800640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd748107370 0x7fd7481a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd74d400640 1 -- 172.21.15.7:0/1803778089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748102110 msgr2=0x7fd7481999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd74d400640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748102110 0x7fd7481999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd74d400640 1 -- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd748107370 msgr2=0x7fd7481a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd74d400640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd748107370 0x7fd7481a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd74d400640 1 -- 172.21.15.7:0/1803778089 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7481a15f0 con 0x7fd748104a40 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd74d400640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748104a40 0x7fd748199f10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd73800e9a0 tx=0x7fd73800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd742a00640 1 -- 172.21.15.7:0/1803778089 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73800cd80 con 0x7fd748104a40 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd742a00640 1 -- 172.21.15.7:0/1803778089 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7380035f0 con 0x7fd748104a40 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd742a00640 1 -- 172.21.15.7:0/1803778089 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd738010670 con 0x7fd748104a40 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.462+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7481a18e0 con 0x7fd748104a40 2024-03-18T01:37:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.463+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7481a1e30 con 0x7fd748104a40 2024-03-18T01:37:07.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.464+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd748108e60 con 0x7fd748104a40 2024-03-18T01:37:07.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.466+0000 7fd742a00640 1 -- 172.21.15.7:0/1803778089 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fd7380037a0 con 0x7fd748104a40 2024-03-18T01:37:07.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.467+0000 7fd742a00640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7300a28a0 0x7fd7300a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:07.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.467+0000 7fd742a00640 1 -- 172.21.15.7:0/1803778089 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd7380cf070 con 0x7fd748104a40 2024-03-18T01:37:07.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.467+0000 7fd74de00640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7300a28a0 0x7fd7300a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:07.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.468+0000 7fd74de00640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7300a28a0 0x7fd7300a4d60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00bbe0 tx=0x7fd73c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.471+0000 7fd742a00640 1 -- 172.21.15.7:0/1803778089 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd73809bd50 con 0x7fd748104a40 2024-03-18T01:37:07.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:07 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/559111902' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T01:37:07.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:07 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/559111902' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T01:37:07.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.732+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7480623e0 con 0x7fd748104a40 2024-03-18T01:37:07.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.734+0000 7fd742a00640 1 -- 172.21.15.7:0/1803778089 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7fd73809b650 con 0x7fd748104a40 2024-03-18T01:37:07.734 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:07.734 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:37:07.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7300a28a0 msgr2=0x7fd7300a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:07.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7300a28a0 0x7fd7300a4d60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00bbe0 tx=0x7fd73c00b390 comp rx=0 tx=0).stop 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748104a40 msgr2=0x7fd748199f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748104a40 0x7fd748199f10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd73800e9a0 tx=0x7fd73800ee60 comp rx=0 tx=0).stop 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 shutdown_connections 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7300a28a0 0x7fd7300a4d60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd748107370 0x7fd7481a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd748104a40 0x7fd748199f10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 --2- 172.21.15.7:0/1803778089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748102110 0x7fd7481999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 >> 172.21.15.7:0/1803778089 conn(0x7fd7480fbaa0 msgr2=0x7fd748101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 shutdown_connections 2024-03-18T01:37:07.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.737+0000 7fd74f200640 1 -- 172.21.15.7:0/1803778089 wait complete. 2024-03-18T01:37:07.759 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:37:07.759 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:37:07.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:07 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/559111902' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 -- 172.21.15.7:0/1118068414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa480ffea0 msgr2=0x7ffa481022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 --2- 172.21.15.7:0/1118068414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa480ffea0 0x7ffa481022d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009900 tx=0x7ffa3802bc30 comp rx=0 tx=0).stop 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 -- 172.21.15.7:0/1118068414 shutdown_connections 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 --2- 172.21.15.7:0/1118068414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa481051c0 0x7ffa48105640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 --2- 172.21.15.7:0/1118068414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102810 0x7ffa48104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 --2- 172.21.15.7:0/1118068414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa480ffea0 0x7ffa481022d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 -- 172.21.15.7:0/1118068414 >> 172.21.15.7:0/1118068414 conn(0x7ffa480fbac0 msgr2=0x7ffa480fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 -- 172.21.15.7:0/1118068414 shutdown_connections 2024-03-18T01:37:07.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.970+0000 7ffa47e00640 1 -- 172.21.15.7:0/1118068414 wait complete. 2024-03-18T01:37:07.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.971+0000 7ffa47e00640 1 Processor -- start 2024-03-18T01:37:07.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47e00640 1 -- start start 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa480ffea0 0x7ffa4819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102810 0x7ffa4819e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa481051c0 0x7ffa481a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ffa481138d0 con 0x7ffa480ffea0 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48113750 con 0x7ffa481051c0 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffa48113a50 con 0x7ffa48102810 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa46a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa480ffea0 0x7ffa4819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa46000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102810 0x7ffa4819e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa481051c0 0x7ffa481a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa46000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102810 0x7ffa4819e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39322/0 (socket says 172.21.15.7:39322) 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa47400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa481051c0 0x7ffa481a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53592/0 (socket says 172.21.15.7:53592) 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa46a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa480ffea0 0x7ffa4819df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53260/0 (socket says 172.21.15.7:53260) 2024-03-18T01:37:07.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.972+0000 7ffa46000640 1 -- 172.21.15.7:0/2949978125 learned_addr learned my addr 172.21.15.7:0/2949978125 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa46000640 1 -- 172.21.15.7:0/2949978125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa481051c0 msgr2=0x7ffa481a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa46000640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa481051c0 0x7ffa481a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa46000640 1 -- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa480ffea0 msgr2=0x7ffa4819df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa46000640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa480ffea0 0x7ffa4819df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa46000640 1 -- 172.21.15.7:0/2949978125 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa481a5b20 con 0x7ffa48102810 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa46a00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa480ffea0 0x7ffa4819df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa47400640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa481051c0 0x7ffa481a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa46000640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102810 0x7ffa4819e490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400d950 tx=0x7ffa3400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa3f400640 1 -- 172.21.15.7:0/2949978125 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa34017030 con 0x7ffa48102810 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa3f400640 1 -- 172.21.15.7:0/2949978125 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa34004020 con 0x7ffa48102810 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa3f400640 1 -- 172.21.15.7:0/2949978125 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa340047a0 con 0x7ffa48102810 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.973+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa481a5e10 con 0x7ffa48102810 2024-03-18T01:37:07.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.974+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa481a6280 con 0x7ffa48102810 2024-03-18T01:37:07.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.975+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa480fd5b0 con 0x7ffa48102810 2024-03-18T01:37:07.980 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.980+0000 7ffa3f400640 1 -- 172.21.15.7:0/2949978125 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7ffa340041d0 con 0x7ffa48102810 2024-03-18T01:37:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.981+0000 7ffa3f400640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.981+0000 7ffa3f400640 1 -- 172.21.15.7:0/2949978125 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ffa340ce7c0 con 0x7ffa48102810 2024-03-18T01:37:07.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.981+0000 7ffa46a00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:07.982 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.982+0000 7ffa46a00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffa3802c240 tx=0x7ffa3802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:07.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:07.986+0000 7ffa3f400640 1 -- 172.21.15.7:0/2949978125 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffa3409b4a0 con 0x7ffa48102810 2024-03-18T01:37:08.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.191+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffa48062480 con 0x7ffa48102810 2024-03-18T01:37:08.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.192+0000 7ffa3f400640 1 -- 172.21.15.7:0/2949978125 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7ffa3409ada0 con 0x7ffa48102810 2024-03-18T01:37:08.192 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:37:08.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ffa2c0a2800 msgr2=0x7ffa2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:08.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffa3802c240 tx=0x7ffa3802df90 comp rx=0 tx=0).stop 2024-03-18T01:37:08.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102810 msgr2=0x7ffa4819e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:08.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102810 0x7ffa4819e490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400d950 tx=0x7ffa3400de10 comp rx=0 tx=0).stop 2024-03-18T01:37:08.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 shutdown_connections 2024-03-18T01:37:08.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa481051c0 0x7ffa481a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa48102810 0x7ffa4819e490 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 --2- 172.21.15.7:0/2949978125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ffa480ffea0 0x7ffa4819df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 >> 172.21.15.7:0/2949978125 conn(0x7ffa480fbac0 msgr2=0x7ffa48103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:08.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 shutdown_connections 2024-03-18T01:37:08.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.195+0000 7ffa47e00640 1 -- 172.21.15.7:0/2949978125 wait complete. 2024-03-18T01:37:08.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.440+0000 7f6e19800640 1 -- 172.21.15.7:0/1877100399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14107390 msgr2=0x7f6e14109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.440+0000 7f6e19800640 1 --2- 172.21.15.7:0/1877100399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14107390 0x7f6e14109780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6e0c009ca0 tx=0x7f6e0c02c100 comp rx=0 tx=0).stop 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.440+0000 7f6e19800640 1 -- 172.21.15.7:0/1877100399 shutdown_connections 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.440+0000 7f6e19800640 1 --2- 172.21.15.7:0/1877100399 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14107390 0x7f6e14109780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.440+0000 7f6e19800640 1 --2- 172.21.15.7:0/1877100399 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14104a60 0x7f6e14106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.440+0000 7f6e19800640 1 --2- 172.21.15.7:0/1877100399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e14102130 0x7f6e14104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.440+0000 7f6e19800640 1 -- 172.21.15.7:0/1877100399 >> 172.21.15.7:0/1877100399 conn(0x7f6e140fbac0 msgr2=0x7f6e140fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.440+0000 7f6e19800640 1 -- 172.21.15.7:0/1877100399 shutdown_connections 2024-03-18T01:37:08.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.441+0000 7f6e19800640 1 -- 172.21.15.7:0/1877100399 wait complete. 2024-03-18T01:37:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e19800640 1 Processor -- start 2024-03-18T01:37:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e19800640 1 -- start start 2024-03-18T01:37:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e19800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14102130 0x7f6e14199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e19800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e14104a60 0x7f6e1419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e19800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14107390 0x7f6e141a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e19800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1410d2c0 con 0x7f6e14107390 2024-03-18T01:37:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e19800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1410d140 con 0x7f6e14104a60 2024-03-18T01:37:08.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e19800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1410d440 con 0x7f6e14102130 2024-03-18T01:37:08.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e13e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14102130 0x7f6e14199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:08.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e13e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14102130 0x7f6e14199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39336/0 (socket says 172.21.15.7:39336) 2024-03-18T01:37:08.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e13e00640 1 -- 172.21.15.7:0/129001413 learned_addr learned my addr 172.21.15.7:0/129001413 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:08.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.442+0000 7f6e13400640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e14104a60 0x7f6e1419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:08.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e18e00640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14107390 0x7f6e141a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:08.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e13e00640 1 -- 172.21.15.7:0/129001413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e14104a60 msgr2=0x7f6e1419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:08.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e13e00640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e14104a60 0x7f6e1419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e13e00640 1 -- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14107390 msgr2=0x7f6e141a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e13e00640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14107390 0x7f6e141a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e13e00640 1 -- 172.21.15.7:0/129001413 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e141a1730 con 0x7f6e14102130 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e18e00640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14107390 0x7f6e141a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e13e00640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14102130 0x7f6e14199b10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400e9f0 tx=0x7f6e0400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e10c00640 1 -- 172.21.15.7:0/129001413 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0400cd80 con 0x7f6e14102130 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e10c00640 1 -- 172.21.15.7:0/129001413 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e04005500 con 0x7f6e14102130 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e10c00640 1 -- 172.21.15.7:0/129001413 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e04010690 con 0x7f6e14102130 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.443+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e141a1a20 con 0x7f6e14102130 2024-03-18T01:37:08.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.444+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e141a1f20 con 0x7f6e14102130 2024-03-18T01:37:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.445+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e14108e80 con 0x7f6e14102130 2024-03-18T01:37:08.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.447+0000 7f6e10c00640 1 -- 172.21.15.7:0/129001413 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f6e040046c0 con 0x7f6e14102130 2024-03-18T01:37:08.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.448+0000 7f6e10c00640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6df80a2880 0x7f6df80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:08.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.448+0000 7f6e10c00640 1 -- 172.21.15.7:0/129001413 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6e040cf250 con 0x7f6e14102130 2024-03-18T01:37:08.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.448+0000 7f6e13400640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6df80a2880 0x7f6df80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:08.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.449+0000 7f6e13400640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6df80a2880 0x7f6df80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6e1419b020 tx=0x7f6e00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:08.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.452+0000 7f6e10c00640 1 -- 172.21.15.7:0/129001413 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e0409bf30 con 0x7f6e14102130 2024-03-18T01:37:08.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:08 smithi007 ceph-mon[31996]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:08.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:08 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1803778089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:08.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:08 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2949978125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:08.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:08 smithi203 ceph-mon[36345]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:08.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:08 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1803778089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:08.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:08 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2949978125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:08.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.749+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6e14062410 con 0x7f6e14102130 2024-03-18T01:37:08.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.749+0000 7f6e10c00640 1 -- 172.21.15.7:0/129001413 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f6e0409b830 con 0x7f6e14102130 2024-03-18T01:37:08.749 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:37:08.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6df80a2880 msgr2=0x7f6df80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:08.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6df80a2880 0x7f6df80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6e1419b020 tx=0x7f6e00005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:08.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14102130 msgr2=0x7f6e14199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14102130 0x7f6e14199b10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6e0400e9f0 tx=0x7f6e0400edf0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 shutdown_connections 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6df80a2880 0x7f6df80a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6e14107390 0x7f6e141a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e14104a60 0x7f6e1419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 --2- 172.21.15.7:0/129001413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e14102130 0x7f6e14199b10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 >> 172.21.15.7:0/129001413 conn(0x7f6e140fbac0 msgr2=0x7f6e14101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 shutdown_connections 2024-03-18T01:37:08.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:08.753+0000 7f6e19800640 1 -- 172.21.15.7:0/129001413 wait complete. 2024-03-18T01:37:08.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:08 smithi046 ceph-mon[36108]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T01:37:08.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:08 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1803778089' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:08.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:08 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2949978125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:08.775 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:37:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.024+0000 7f35be800640 1 -- 172.21.15.7:0/1798794797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b8101650 msgr2=0x7f35b8101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.024+0000 7f35be800640 1 --2- 172.21.15.7:0/1798794797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b8101650 0x7f35b8101a70 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f35ac009be0 tx=0x7f35ac02bef0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.024+0000 7f35be800640 1 -- 172.21.15.7:0/1798794797 shutdown_connections 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.024+0000 7f35be800640 1 --2- 172.21.15.7:0/1798794797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b8103200 0x7f35b8109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.024+0000 7f35be800640 1 --2- 172.21.15.7:0/1798794797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35b8102840 0x7f35b8102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.024+0000 7f35be800640 1 --2- 172.21.15.7:0/1798794797 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b8101650 0x7f35b8101a70 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.024+0000 7f35be800640 1 -- 172.21.15.7:0/1798794797 >> 172.21.15.7:0/1798794797 conn(0x7f35b80fcbc0 msgr2=0x7f35b80ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.025+0000 7f35be800640 1 -- 172.21.15.7:0/1798794797 shutdown_connections 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.025+0000 7f35be800640 1 -- 172.21.15.7:0/1798794797 wait complete. 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.025+0000 7f35be800640 1 Processor -- start 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35be800640 1 -- start start 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35be800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35b8101650 0x7f35b8075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35be800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b8102840 0x7f35b8073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35be800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b8103200 0x7f35b8074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35be800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f35b810d2f0 con 0x7f35b8102840 2024-03-18T01:37:09.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35be800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f35b810d170 con 0x7f35b8103200 2024-03-18T01:37:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35be800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35b810d470 con 0x7f35b8101650 2024-03-18T01:37:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35bd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35b8101650 0x7f35b8075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35bd400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35b8101650 0x7f35b8075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39354/0 (socket says 172.21.15.7:39354) 2024-03-18T01:37:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35bd400640 1 -- 172.21.15.7:0/1752034327 learned_addr learned my addr 172.21.15.7:0/1752034327 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35bde00640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b8103200 0x7f35b8074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:09.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.026+0000 7f35bca00640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b8102840 0x7f35b8073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35bd400640 1 -- 172.21.15.7:0/1752034327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b8103200 msgr2=0x7f35b8074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35bd400640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b8103200 0x7f35b8074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35bd400640 1 -- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b8102840 msgr2=0x7f35b8073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35bd400640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b8102840 0x7f35b8073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35bd400640 1 -- 172.21.15.7:0/1752034327 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b80749a0 con 0x7f35b8101650 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35bde00640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b8103200 0x7f35b8074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35bd400640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35b8101650 0x7f35b8075590 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f35ac02c500 tx=0x7f35ac033580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35b6000640 1 -- 172.21.15.7:0/1752034327 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac043020 con 0x7f35b8101650 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35b6000640 1 -- 172.21.15.7:0/1752034327 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35ac003e50 con 0x7f35b8101650 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.027+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b81a1ae0 con 0x7f35b8101650 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.028+0000 7f35b6000640 1 -- 172.21.15.7:0/1752034327 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac034690 con 0x7f35b8101650 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.028+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b81a2030 con 0x7f35b8101650 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.029+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35b81089c0 con 0x7f35b8101650 2024-03-18T01:37:09.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.030+0000 7f35b6000640 1 -- 172.21.15.7:0/1752034327 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f35ac046020 con 0x7f35b8101650 2024-03-18T01:37:09.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.032+0000 7f35b6000640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35a00a2800 0x7f35a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:09.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.032+0000 7f35b6000640 1 -- 172.21.15.7:0/1752034327 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f35ac0f14f0 con 0x7f35b8101650 2024-03-18T01:37:09.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.032+0000 7f35bca00640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35a00a2800 0x7f35a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:09.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.033+0000 7f35bca00640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35a00a2800 0x7f35a00a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f35b8075300 tx=0x7f35a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:09.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.036+0000 7f35b6000640 1 -- 172.21.15.7:0/1752034327 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f35ac04d020 con 0x7f35b8101650 2024-03-18T01:37:09.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.310+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f35b80623e0 con 0x7f35b8101650 2024-03-18T01:37:09.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.311+0000 7f35b6000640 1 -- 172.21.15.7:0/1752034327 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f35ac03d3b0 con 0x7f35b8101650 2024-03-18T01:37:09.311 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:37:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35a00a2800 msgr2=0x7f35a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35a00a2800 0x7f35a00a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f35b8075300 tx=0x7f35a4009260 comp rx=0 tx=0).stop 2024-03-18T01:37:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35b8101650 msgr2=0x7f35b8075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35b8101650 0x7f35b8075590 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f35ac02c500 tx=0x7f35ac033580 comp rx=0 tx=0).stop 2024-03-18T01:37:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 shutdown_connections 2024-03-18T01:37:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35b8103200 0x7f35b8074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f35a00a2800 0x7f35a00a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f35b8102840 0x7f35b8073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 --2- 172.21.15.7:0/1752034327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35b8101650 0x7f35b8075590 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 >> 172.21.15.7:0/1752034327 conn(0x7f35b80fcbc0 msgr2=0x7f35b8105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 shutdown_connections 2024-03-18T01:37:09.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.314+0000 7f35be800640 1 -- 172.21.15.7:0/1752034327 wait complete. 2024-03-18T01:37:09.333 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:37:09.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:09 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/129001413' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 -- 172.21.15.7:0/2082811082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 msgr2=0x7fd894105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 --2- 172.21.15.7:0/2082811082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 0x7fd894105080 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009900 tx=0x7fd87c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 -- 172.21.15.7:0/2082811082 shutdown_connections 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 --2- 172.21.15.7:0/2082811082 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd8941055c0 0x7fd89410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 --2- 172.21.15.7:0/2082811082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 0x7fd894105080 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 --2- 172.21.15.7:0/2082811082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd894103a10 0x7fd894103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 -- 172.21.15.7:0/2082811082 >> 172.21.15.7:0/2082811082 conn(0x7fd8940fefa0 msgr2=0x7fd8941013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 -- 172.21.15.7:0/2082811082 shutdown_connections 2024-03-18T01:37:09.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.555+0000 7fd89b000640 1 -- 172.21.15.7:0/2082811082 wait complete. 2024-03-18T01:37:09.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.556+0000 7fd89b000640 1 Processor -- start 2024-03-18T01:37:09.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd89b000640 1 -- start start 2024-03-18T01:37:09.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd89b000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd894103a10 0x7fd894199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:09.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd89b000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 0x7fd89419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd89b000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8941055c0 0x7fd8941a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd89b000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd89410f6e0 con 0x7fd894103a10 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd89b000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd89410f560 con 0x7fd8941055c0 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd89b000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd89410f860 con 0x7fd894104c00 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd893e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd894103a10 0x7fd894199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd893400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 0x7fd89419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd899000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8941055c0 0x7fd8941a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd893e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd894103a10 0x7fd894199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53312/0 (socket says 172.21.15.7:53312) 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.557+0000 7fd893e00640 1 -- 172.21.15.7:0/2522831859 learned_addr learned my addr 172.21.15.7:0/2522831859 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd899000640 1 -- 172.21.15.7:0/2522831859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 msgr2=0x7fd89419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd899000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 0x7fd89419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd899000640 1 -- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd894103a10 msgr2=0x7fd894199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd899000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd894103a10 0x7fd894199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd899000640 1 -- 172.21.15.7:0/2522831859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8941a19b0 con 0x7fd8941055c0 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd893400640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 0x7fd89419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd893e00640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd894103a10 0x7fd894199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd899000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8941055c0 0x7fd8941a1310 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00e970 tx=0x7fd88c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd890c00640 1 -- 172.21.15.7:0/2522831859 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c00ccc0 con 0x7fd8941055c0 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8941a1ca0 con 0x7fd8941055c0 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd890c00640 1 -- 172.21.15.7:0/2522831859 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd88c005020 con 0x7fd8941055c0 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd890c00640 1 -- 172.21.15.7:0/2522831859 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c004a00 con 0x7fd8941055c0 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.558+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8941a21a0 con 0x7fd8941055c0 2024-03-18T01:37:09.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.560+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd89410ad30 con 0x7fd8941055c0 2024-03-18T01:37:09.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.562+0000 7fd890c00640 1 -- 172.21.15.7:0/2522831859 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fd88c0051d0 con 0x7fd8941055c0 2024-03-18T01:37:09.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.563+0000 7fd890c00640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd8780a2800 0x7fd8780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:09.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.564+0000 7fd890c00640 1 -- 172.21.15.7:0/2522831859 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd88c0cee10 con 0x7fd8941055c0 2024-03-18T01:37:09.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.564+0000 7fd893e00640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd8780a2800 0x7fd8780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:09.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.564+0000 7fd893e00640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd8780a2800 0x7fd8780a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd89419b270 tx=0x7fd884009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:09.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.572+0000 7fd890c00640 1 -- 172.21.15.7:0/2522831859 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd88c0d2030 con 0x7fd8941055c0 2024-03-18T01:37:09.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:09 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/129001413' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:09.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:09 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/129001413' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:09.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.804+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd89402d1b0 con 0x7fd8941055c0 2024-03-18T01:37:09.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.805+0000 7fd890c00640 1 -- 172.21.15.7:0/2522831859 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fd88c09baf0 con 0x7fd8941055c0 2024-03-18T01:37:09.805 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:37:09.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.807+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd8780a2800 msgr2=0x7fd8780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.807+0000 7fd89b000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd8780a2800 0x7fd8780a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd89419b270 tx=0x7fd884009500 comp rx=0 tx=0).stop 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.807+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8941055c0 msgr2=0x7fd8941a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.807+0000 7fd89b000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8941055c0 0x7fd8941a1310 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00e970 tx=0x7fd88c00ee30 comp rx=0 tx=0).stop 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.808+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 shutdown_connections 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.808+0000 7fd89b000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8941055c0 0x7fd8941a1310 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.808+0000 7fd89b000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd894104c00 0x7fd89419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.808+0000 7fd89b000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd8780a2800 0x7fd8780a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.808+0000 7fd89b000640 1 --2- 172.21.15.7:0/2522831859 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd894103a10 0x7fd894199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.808+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 >> 172.21.15.7:0/2522831859 conn(0x7fd8940fefa0 msgr2=0x7fd894107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.808+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 shutdown_connections 2024-03-18T01:37:09.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:09.808+0000 7fd89b000640 1 -- 172.21.15.7:0/2522831859 wait complete. 2024-03-18T01:37:09.827 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:37:10.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 -- 172.21.15.7:0/3689456789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 msgr2=0x7fb57c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 --2- 172.21.15.7:0/3689456789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 0x7fb57c105600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb574009ca0 tx=0x7fb57402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 -- 172.21.15.7:0/3689456789 shutdown_connections 2024-03-18T01:37:10.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 --2- 172.21.15.7:0/3689456789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 0x7fb57c105600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 --2- 172.21.15.7:0/3689456789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb57c1027d0 0x7fb57c104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 --2- 172.21.15.7:0/3689456789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb57c0ffe60 0x7fb57c102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 -- 172.21.15.7:0/3689456789 >> 172.21.15.7:0/3689456789 conn(0x7fb57c0fbaa0 msgr2=0x7fb57c0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:10.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 -- 172.21.15.7:0/3689456789 shutdown_connections 2024-03-18T01:37:10.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.024+0000 7fb57be00640 1 -- 172.21.15.7:0/3689456789 wait complete. 2024-03-18T01:37:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.025+0000 7fb57be00640 1 Processor -- start 2024-03-18T01:37:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.025+0000 7fb57be00640 1 -- start start 2024-03-18T01:37:10.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb57c0ffe60 0x7fb57c19ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57be00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb57c1027d0 0x7fb57c19e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 0x7fb57c1a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57be00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c113860 con 0x7fb57c1027d0 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57be00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1136e0 con 0x7fb57c105180 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57be00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c1139e0 con 0x7fb57c0ffe60 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb57c0ffe60 0x7fb57c19ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb57c0ffe60 0x7fb57c19ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39400/0 (socket says 172.21.15.7:39400) 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57aa00640 1 -- 172.21.15.7:0/357905885 learned_addr learned my addr 172.21.15.7:0/357905885 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:10.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb57b400640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 0x7fb57c1a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.026+0000 7fb573e00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb57c1027d0 0x7fb57c19e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb57aa00640 1 -- 172.21.15.7:0/357905885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 msgr2=0x7fb57c1a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb57aa00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 0x7fb57c1a5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb57aa00640 1 -- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb57c1027d0 msgr2=0x7fb57c19e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb57aa00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb57c1027d0 0x7fb57c19e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb57aa00640 1 -- 172.21.15.7:0/357905885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c1a59d0 con 0x7fb57c0ffe60 2024-03-18T01:37:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb573e00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb57c1027d0 0x7fb57c19e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:37:10.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb57b400640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 0x7fb57c1a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb57aa00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb57c0ffe60 0x7fb57c19ddb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00e9f0 tx=0x7fb56c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb573400640 1 -- 172.21.15.7:0/357905885 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c00cd80 con 0x7fb57c0ffe60 2024-03-18T01:37:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb573400640 1 -- 172.21.15.7:0/357905885 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb56c010910 con 0x7fb57c0ffe60 2024-03-18T01:37:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57c1a5cc0 con 0x7fb57c0ffe60 2024-03-18T01:37:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.027+0000 7fb573400640 1 -- 172.21.15.7:0/357905885 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c018990 con 0x7fb57c0ffe60 2024-03-18T01:37:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.028+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb57c1a61b0 con 0x7fb57c0ffe60 2024-03-18T01:37:10.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.029+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb57c0fd590 con 0x7fb57c0ffe60 2024-03-18T01:37:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.032+0000 7fb573400640 1 -- 172.21.15.7:0/357905885 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fb56c010ac0 con 0x7fb57c0ffe60 2024-03-18T01:37:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.033+0000 7fb573400640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb5600a2920 0x7fb5600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:10.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.034+0000 7fb573400640 1 -- 172.21.15.7:0/357905885 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb56c0cf160 con 0x7fb57c0ffe60 2024-03-18T01:37:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.034+0000 7fb573e00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb5600a2920 0x7fb5600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.035+0000 7fb573e00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb5600a2920 0x7fb5600a4de0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb57c19f2c0 tx=0x7fb564005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:10.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.037+0000 7fb573400640 1 -- 172.21.15.7:0/357905885 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb56c09be40 con 0x7fb57c0ffe60 2024-03-18T01:37:10.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.280+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb57c0624e0 con 0x7fb57c0ffe60 2024-03-18T01:37:10.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.281+0000 7fb573400640 1 -- 172.21.15.7:0/357905885 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11918 (secure 0 0 0) 0x7fb56c09b740 con 0x7fb57c0ffe60 2024-03-18T01:37:10.282 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:10.282 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:35:30.634148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24542,24523,24544,24596,24562]},"id":1}]} 2024-03-18T01:37:10.282 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb5600a2920 msgr2=0x7fb5600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb5600a2920 0x7fb5600a4de0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb57c19f2c0 tx=0x7fb564005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb57c0ffe60 msgr2=0x7fb57c19ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb57c0ffe60 0x7fb57c19ddb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00e9f0 tx=0x7fb56c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 shutdown_connections 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb57c105180 0x7fb57c1a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb5600a2920 0x7fb5600a4de0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb57c1027d0 0x7fb57c19e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 --2- 172.21.15.7:0/357905885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb57c0ffe60 0x7fb57c19ddb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 >> 172.21.15.7:0/357905885 conn(0x7fb57c0fbaa0 msgr2=0x7fb57c100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 shutdown_connections 2024-03-18T01:37:10.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.285+0000 7fb57be00640 1 -- 172.21.15.7:0/357905885 wait complete. 2024-03-18T01:37:10.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:37:10.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 -- 172.21.15.7:0/3264759524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80761a0 msgr2=0x7f54f80765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 --2- 172.21.15.7:0/3264759524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80761a0 0x7f54f80765c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009900 tx=0x7f54ec02bc50 comp rx=0 tx=0).stop 2024-03-18T01:37:10.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 -- 172.21.15.7:0/3264759524 shutdown_connections 2024-03-18T01:37:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 --2- 172.21.15.7:0/3264759524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54f8077d80 0x7f54f8078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 --2- 172.21.15.7:0/3264759524 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f80773e0 0x7f54f8077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 --2- 172.21.15.7:0/3264759524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80761a0 0x7f54f80765c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 -- 172.21.15.7:0/3264759524 >> 172.21.15.7:0/3264759524 conn(0x7f54f80fed70 msgr2=0x7f54f81011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 -- 172.21.15.7:0/3264759524 shutdown_connections 2024-03-18T01:37:10.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.497+0000 7f54fee00640 1 -- 172.21.15.7:0/3264759524 wait complete. 2024-03-18T01:37:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.498+0000 7f54fee00640 1 Processor -- start 2024-03-18T01:37:10.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.498+0000 7f54fee00640 1 -- start start 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.498+0000 7f54fee00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f80761a0 0x7f54f819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.498+0000 7f54fee00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80773e0 0x7f54f819e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.498+0000 7f54fee00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54f8077d80 0x7f54f81a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.498+0000 7f54fee00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f54f81138d0 con 0x7f54f80761a0 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.498+0000 7f54fee00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f54f8113750 con 0x7f54f8077d80 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.498+0000 7f54fee00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f54f8113a50 con 0x7f54f80773e0 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fda00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f80761a0 0x7f54f819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80773e0 0x7f54f819e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fda00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f80761a0 0x7f54f819ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53360/0 (socket says 172.21.15.7:53360) 2024-03-18T01:37:10.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fda00640 1 -- 172.21.15.7:0/3146643577 learned_addr learned my addr 172.21.15.7:0/3146643577 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fe400640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54f8077d80 0x7f54f81a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fd000640 1 -- 172.21.15.7:0/3146643577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54f8077d80 msgr2=0x7f54f81a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fd000640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54f8077d80 0x7f54f81a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fd000640 1 -- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f80761a0 msgr2=0x7f54f819ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fd000640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f80761a0 0x7f54f819ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fd000640 1 -- 172.21.15.7:0/3146643577 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f81a5a10 con 0x7f54f80773e0 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.499+0000 7f54fd000640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80773e0 0x7f54f819e330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f54e400e9f0 tx=0x7f54e400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.500+0000 7f54f6000640 1 -- 172.21.15.7:0/3146643577 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e400ccc0 con 0x7f54f80773e0 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.500+0000 7f54f6000640 1 -- 172.21.15.7:0/3146643577 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54e4010910 con 0x7f54f80773e0 2024-03-18T01:37:10.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.500+0000 7f54f6000640 1 -- 172.21.15.7:0/3146643577 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4018960 con 0x7f54f80773e0 2024-03-18T01:37:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.500+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f81a5d00 con 0x7f54f80773e0 2024-03-18T01:37:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.500+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f81a61a0 con 0x7f54f80773e0 2024-03-18T01:37:10.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.501+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f8110170 con 0x7f54f80773e0 2024-03-18T01:37:10.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.504+0000 7f54f6000640 1 -- 172.21.15.7:0/3146643577 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f54e4010ac0 con 0x7f54f80773e0 2024-03-18T01:37:10.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.505+0000 7f54f6000640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f54e00a2920 0x7f54e00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:10.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.505+0000 7f54f6000640 1 -- 172.21.15.7:0/3146643577 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f54e40cf110 con 0x7f54f80773e0 2024-03-18T01:37:10.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.505+0000 7f54fda00640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f54e00a2920 0x7f54e00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:10.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.506+0000 7f54fda00640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f54e00a2920 0x7f54e00a4de0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f54ec0096c0 tx=0x7f54ec000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:10.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.509+0000 7f54f6000640 1 -- 172.21.15.7:0/3146643577 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f54e409b7a0 con 0x7f54f80773e0 2024-03-18T01:37:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:10 smithi007 ceph-mon[31996]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T01:37:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1752034327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:10.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2522831859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:10.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/357905885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:10 smithi203 ceph-mon[36345]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T01:37:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:10 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1752034327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:10 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2522831859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:10 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/357905885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:10.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.711+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54f802d3d0 con 0x7f54f80773e0 2024-03-18T01:37:10.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.713+0000 7f54f6000640 1 -- 172.21.15.7:0/3146643577 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7f54e409f840 con 0x7f54f80773e0 2024-03-18T01:37:10.714 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:10.714 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:37:10.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f54e00a2920 msgr2=0x7f54e00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f54e00a2920 0x7f54e00a4de0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f54ec0096c0 tx=0x7f54ec000bc0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80773e0 msgr2=0x7f54f819e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80773e0 0x7f54f819e330 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f54e400e9f0 tx=0x7f54e400eeb0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 shutdown_connections 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54f8077d80 0x7f54f81a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54f80773e0 0x7f54f819e330 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f54e00a2920 0x7f54e00a4de0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 --2- 172.21.15.7:0/3146643577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f54f80761a0 0x7f54f819ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 >> 172.21.15.7:0/3146643577 conn(0x7f54f80fed70 msgr2=0x7f54f8100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 shutdown_connections 2024-03-18T01:37:10.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:10.716+0000 7f54fee00640 1 -- 172.21.15.7:0/3146643577 wait complete. 2024-03-18T01:37:10.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:37:10.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:37:10.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:10 smithi046 ceph-mon[36108]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T01:37:10.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:10 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1752034327' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:10.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:10 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2522831859' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:10.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:10 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/357905885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.001+0000 7fd94b200640 1 -- 172.21.15.7:0/883845054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 msgr2=0x7fd944103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.001+0000 7fd94b200640 1 --2- 172.21.15.7:0/883845054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 0x7fd944103c40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd938009900 tx=0x7fd93802bc30 comp rx=0 tx=0).stop 2024-03-18T01:37:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.002+0000 7fd94b200640 1 -- 172.21.15.7:0/883845054 shutdown_connections 2024-03-18T01:37:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.002+0000 7fd94b200640 1 --2- 172.21.15.7:0/883845054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9441053d0 0x7fd94410bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.002+0000 7fd94b200640 1 --2- 172.21.15.7:0/883845054 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd944104a10 0x7fd944104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.002+0000 7fd94b200640 1 --2- 172.21.15.7:0/883845054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 0x7fd944103c40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.002+0000 7fd94b200640 1 -- 172.21.15.7:0/883845054 >> 172.21.15.7:0/883845054 conn(0x7fd9440fedb0 msgr2=0x7fd9441011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.002+0000 7fd94b200640 1 -- 172.21.15.7:0/883845054 shutdown_connections 2024-03-18T01:37:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.002+0000 7fd94b200640 1 -- 172.21.15.7:0/883845054 wait complete. 2024-03-18T01:37:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.003+0000 7fd94b200640 1 Processor -- start 2024-03-18T01:37:11.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.004+0000 7fd94b200640 1 -- start start 2024-03-18T01:37:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.004+0000 7fd94b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 0x7fd944199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.004+0000 7fd94b200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd944104a10 0x7fd94419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.004+0000 7fd94b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9441053d0 0x7fd9441a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.004+0000 7fd94b200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd94410f4c0 con 0x7fd944104a10 2024-03-18T01:37:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.004+0000 7fd94b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd94410f340 con 0x7fd944103820 2024-03-18T01:37:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.004+0000 7fd94b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd94410f640 con 0x7fd9441053d0 2024-03-18T01:37:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 0x7fd944199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 0x7fd944199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53718/0 (socket says 172.21.15.7:53718) 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 -- 172.21.15.7:0/91378049 learned_addr learned my addr 172.21.15.7:0/91378049 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd94a800640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9441053d0 0x7fd9441a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949400640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd944104a10 0x7fd94419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 -- 172.21.15.7:0/91378049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9441053d0 msgr2=0x7fd9441a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9441053d0 0x7fd9441a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 -- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd944104a10 msgr2=0x7fd94419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd944104a10 0x7fd94419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 -- 172.21.15.7:0/91378049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9441a17b0 con 0x7fd944103820 2024-03-18T01:37:11.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949400640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd944104a10 0x7fd94419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:37:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.005+0000 7fd949e00640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 0x7fd944199b90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd938000c00 tx=0x7fd938005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.006+0000 7fd93ea00640 1 -- 172.21.15.7:0/91378049 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938043020 con 0x7fd944103820 2024-03-18T01:37:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.006+0000 7fd93ea00640 1 -- 172.21.15.7:0/91378049 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9380054e0 con 0x7fd944103820 2024-03-18T01:37:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.006+0000 7fd93ea00640 1 -- 172.21.15.7:0/91378049 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938003e60 con 0x7fd944103820 2024-03-18T01:37:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.006+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9441a1a40 con 0x7fd944103820 2024-03-18T01:37:11.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.006+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9441a1f40 con 0x7fd944103820 2024-03-18T01:37:11.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.007+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd94410abe0 con 0x7fd944103820 2024-03-18T01:37:11.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.009+0000 7fd93ea00640 1 -- 172.21.15.7:0/91378049 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fd938046020 con 0x7fd944103820 2024-03-18T01:37:11.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.010+0000 7fd93ea00640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.010+0000 7fd93ea00640 1 -- 172.21.15.7:0/91378049 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd9380f0fe0 con 0x7fd944103820 2024-03-18T01:37:11.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.011+0000 7fd949400640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.012+0000 7fd949400640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd94419b0a0 tx=0x7fd934005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:11.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.015+0000 7fd93ea00640 1 -- 172.21.15.7:0/91378049 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd93804d020 con 0x7fd944103820 2024-03-18T01:37:11.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.217+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd9440623e0 con 0x7fd944103820 2024-03-18T01:37:11.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.218+0000 7fd93ea00640 1 -- 172.21.15.7:0/91378049 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fd9380bdd00 con 0x7fd944103820 2024-03-18T01:37:11.218 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:37:11.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd92c0a2800 msgr2=0x7fd92c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd94419b0a0 tx=0x7fd934005e50 comp rx=0 tx=0).stop 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 msgr2=0x7fd944199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 0x7fd944199b90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd938000c00 tx=0x7fd938005dd0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 shutdown_connections 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9441053d0 0x7fd9441a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd92c0a2800 0x7fd92c0a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd944104a10 0x7fd94419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 --2- 172.21.15.7:0/91378049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd944103820 0x7fd944199b90 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 >> 172.21.15.7:0/91378049 conn(0x7fd9440fedb0 msgr2=0x7fd944107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 shutdown_connections 2024-03-18T01:37:11.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.221+0000 7fd94b200640 1 -- 172.21.15.7:0/91378049 wait complete. 2024-03-18T01:37:11.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.476+0000 7f460e400640 1 -- 172.21.15.7:0/2216997740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46081031e0 msgr2=0x7f4608109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.476+0000 7f460e400640 1 --2- 172.21.15.7:0/2216997740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46081031e0 0x7f4608109a70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4600009ca0 tx=0x7f460002c0f0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.476+0000 7f460e400640 1 -- 172.21.15.7:0/2216997740 shutdown_connections 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.476+0000 7f460e400640 1 --2- 172.21.15.7:0/2216997740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f46081031e0 0x7f4608109a70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.476+0000 7f460e400640 1 --2- 172.21.15.7:0/2216997740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4608102820 0x7f4608102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.476+0000 7f460e400640 1 --2- 172.21.15.7:0/2216997740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4608101630 0x7f4608101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.476+0000 7f460e400640 1 -- 172.21.15.7:0/2216997740 >> 172.21.15.7:0/2216997740 conn(0x7f46080fcba0 msgr2=0x7f46080fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.477+0000 7f460e400640 1 -- 172.21.15.7:0/2216997740 shutdown_connections 2024-03-18T01:37:11.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.477+0000 7f460e400640 1 -- 172.21.15.7:0/2216997740 wait complete. 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460e400640 1 Processor -- start 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460e400640 1 -- start start 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460e400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4608101630 0x7f4608199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460e400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4608102820 0x7f4608199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460e400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46081031e0 0x7f46081a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460e400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f460810d290 con 0x7f46081031e0 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460e400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f460810d110 con 0x7f4608102820 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460e400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f460810d410 con 0x7f4608101630 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4608101630 0x7f4608199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4608101630 0x7f4608199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39450/0 (socket says 172.21.15.7:39450) 2024-03-18T01:37:11.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f460d000640 1 -- 172.21.15.7:0/797885831 learned_addr learned my addr 172.21.15.7:0/797885831 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.478+0000 7f45ffe00640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4608102820 0x7f4608199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f460da00640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46081031e0 0x7f46081a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f460d000640 1 -- 172.21.15.7:0/797885831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4608102820 msgr2=0x7f4608199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f460d000640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4608102820 0x7f4608199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f460d000640 1 -- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46081031e0 msgr2=0x7f46081a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f460d000640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46081031e0 0x7f46081a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f460d000640 1 -- 172.21.15.7:0/797885831 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46081a16d0 con 0x7f4608101630 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f460da00640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46081031e0 0x7f46081a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f460d000640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4608101630 0x7f4608199ab0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f45f800e9c0 tx=0x7f45f800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.479+0000 7f4606000640 1 -- 172.21.15.7:0/797885831 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f800cd80 con 0x7f4608101630 2024-03-18T01:37:11.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.480+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46081a19c0 con 0x7f4608101630 2024-03-18T01:37:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.480+0000 7f4606000640 1 -- 172.21.15.7:0/797885831 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45f80035f0 con 0x7f4608101630 2024-03-18T01:37:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.480+0000 7f4606000640 1 -- 172.21.15.7:0/797885831 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f8010690 con 0x7f4608101630 2024-03-18T01:37:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.480+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46081a1f10 con 0x7f4608101630 2024-03-18T01:37:11.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.481+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46081a21a0 con 0x7f4608101630 2024-03-18T01:37:11.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.484+0000 7f4606000640 1 -- 172.21.15.7:0/797885831 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f45f80037a0 con 0x7f4608101630 2024-03-18T01:37:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.485+0000 7f4606000640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f45ec0a2800 0x7f45ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.485+0000 7f4606000640 1 -- 172.21.15.7:0/797885831 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f45f80cf280 con 0x7f4608101630 2024-03-18T01:37:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.485+0000 7f45ffe00640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f45ec0a2800 0x7f45ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.486+0000 7f45ffe00640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f45ec0a2800 0x7f45ec0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f460819afc0 tx=0x7f45f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:11.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.487+0000 7f4606000640 1 -- 172.21.15.7:0/797885831 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f45f809bf60 con 0x7f4608101630 2024-03-18T01:37:11.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:11 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3146643577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:11.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:11 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/91378049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:11.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:11 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3146643577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:11.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:11 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/91378049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:11.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.690+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f46081a2370 con 0x7f4608101630 2024-03-18T01:37:11.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.691+0000 7f4606000640 1 -- 172.21.15.7:0/797885831 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f45f809b860 con 0x7f4608101630 2024-03-18T01:37:11.691 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:37:11.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f45ec0a2800 msgr2=0x7f45ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f45ec0a2800 0x7f45ec0a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f460819afc0 tx=0x7f45f0005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:11.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4608101630 msgr2=0x7f4608199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4608101630 0x7f4608199ab0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f45f800e9c0 tx=0x7f45f800ee80 comp rx=0 tx=0).stop 2024-03-18T01:37:11.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 shutdown_connections 2024-03-18T01:37:11.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f45ec0a2800 0x7f45ec0a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f46081031e0 0x7f46081a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4608102820 0x7f4608199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 --2- 172.21.15.7:0/797885831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4608101630 0x7f4608199ab0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 >> 172.21.15.7:0/797885831 conn(0x7f46080fcba0 msgr2=0x7f4608073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:11.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 shutdown_connections 2024-03-18T01:37:11.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.695+0000 7f460e400640 1 -- 172.21.15.7:0/797885831 wait complete. 2024-03-18T01:37:11.718 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:37:11.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:11 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3146643577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:11.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:11 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/91378049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 -- 172.21.15.7:0/3640357969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a81031e0 msgr2=0x7ff5a8109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/3640357969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a81031e0 0x7ff5a8109a70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009900 tx=0x7ff5a002bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 -- 172.21.15.7:0/3640357969 shutdown_connections 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/3640357969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a81031e0 0x7ff5a8109a70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/3640357969 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a8102820 0x7ff5a8102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/3640357969 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a8101630 0x7ff5a8101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 -- 172.21.15.7:0/3640357969 >> 172.21.15.7:0/3640357969 conn(0x7ff5a80fcba0 msgr2=0x7ff5a80fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 -- 172.21.15.7:0/3640357969 shutdown_connections 2024-03-18T01:37:11.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.962+0000 7ff5a7e00640 1 -- 172.21.15.7:0/3640357969 wait complete. 2024-03-18T01:37:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.964+0000 7ff5a7e00640 1 Processor -- start 2024-03-18T01:37:11.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.964+0000 7ff5a7e00640 1 -- start start 2024-03-18T01:37:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.964+0000 7ff5a7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a8101630 0x7ff5a81073b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8102820 0x7ff5a81078f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a7e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a81031e0 0x7ff5a8103f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a7e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a810d2a0 con 0x7ff5a81031e0 2024-03-18T01:37:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a7e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a810d120 con 0x7ff5a8102820 2024-03-18T01:37:11.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a7e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a810d420 con 0x7ff5a8101630 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a8101630 0x7ff5a81073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a7400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a81031e0 0x7ff5a8103f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8102820 0x7ff5a81078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a8101630 0x7ff5a81073b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39468/0 (socket says 172.21.15.7:39468) 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8102820 0x7ff5a81078f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53766/0 (socket says 172.21.15.7:53766) 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6a00640 1 -- 172.21.15.7:0/1679900278 learned_addr learned my addr 172.21.15.7:0/1679900278 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6000640 1 -- 172.21.15.7:0/1679900278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a8101630 msgr2=0x7ff5a81073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6000640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a8101630 0x7ff5a81073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6000640 1 -- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a81031e0 msgr2=0x7ff5a8103f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6000640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a81031e0 0x7ff5a8103f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6000640 1 -- 172.21.15.7:0/1679900278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a81047e0 con 0x7ff5a8102820 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.965+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a8101630 0x7ff5a81073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.966+0000 7ff5a6000640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8102820 0x7ff5a81078f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff59000e750 tx=0x7ff59000ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:11.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.966+0000 7ff59f400640 1 -- 172.21.15.7:0/1679900278 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59000ccc0 con 0x7ff5a8102820 2024-03-18T01:37:11.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.966+0000 7ff59f400640 1 -- 172.21.15.7:0/1679900278 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff59000ee60 con 0x7ff5a8102820 2024-03-18T01:37:11.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.966+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a81a5f50 con 0x7ff5a8102820 2024-03-18T01:37:11.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.966+0000 7ff59f400640 1 -- 172.21.15.7:0/1679900278 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff590004a00 con 0x7ff5a8102820 2024-03-18T01:37:11.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.966+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a81a6300 con 0x7ff5a8102820 2024-03-18T01:37:11.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.967+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a804f540 con 0x7ff5a8102820 2024-03-18T01:37:11.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.970+0000 7ff59f400640 1 -- 172.21.15.7:0/1679900278 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7ff590004ba0 con 0x7ff5a8102820 2024-03-18T01:37:11.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.971+0000 7ff59f400640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a2880 0x7ff58c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:11.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.971+0000 7ff59f400640 1 -- 172.21.15.7:0/1679900278 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff5900cec60 con 0x7ff5a8102820 2024-03-18T01:37:11.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.971+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a2880 0x7ff58c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:11.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.972+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a2880 0x7ff58c0a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff598005c40 tx=0x7ff598009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:11.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:11.977+0000 7ff59f400640 1 -- 172.21.15.7:0/1679900278 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff59009b940 con 0x7ff5a8102820 2024-03-18T01:37:12.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.181+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff5a81a6830 con 0x7ff5a8102820 2024-03-18T01:37:12.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.182+0000 7ff59f400640 1 -- 172.21.15.7:0/1679900278 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7ff59009b240 con 0x7ff5a8102820 2024-03-18T01:37:12.182 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:37:12.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a2880 msgr2=0x7ff58c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a2880 0x7ff58c0a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff598005c40 tx=0x7ff598009500 comp rx=0 tx=0).stop 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8102820 msgr2=0x7ff5a81078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8102820 0x7ff5a81078f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff59000e750 tx=0x7ff59000ec10 comp rx=0 tx=0).stop 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 shutdown_connections 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a2880 0x7ff58c0a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a81031e0 0x7ff5a8103f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8102820 0x7ff5a81078f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 --2- 172.21.15.7:0/1679900278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a8101630 0x7ff5a81073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 >> 172.21.15.7:0/1679900278 conn(0x7ff5a80fcba0 msgr2=0x7ff5a8073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 shutdown_connections 2024-03-18T01:37:12.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.186+0000 7ff5a7e00640 1 -- 172.21.15.7:0/1679900278 wait complete. 2024-03-18T01:37:12.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.438+0000 7f13e1800640 1 -- 172.21.15.7:0/902059040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 msgr2=0x7f13dc0746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.438+0000 7f13e1800640 1 --2- 172.21.15.7:0/902059040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 0x7f13dc0746c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009860 tx=0x7f13c802bb70 comp rx=0 tx=0).stop 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.439+0000 7f13e1800640 1 -- 172.21.15.7:0/902059040 shutdown_connections 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.439+0000 7f13e1800640 1 --2- 172.21.15.7:0/902059040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc10dbf0 0x7f13dc10ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.439+0000 7f13e1800640 1 --2- 172.21.15.7:0/902059040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 0x7f13dc0746c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.439+0000 7f13e1800640 1 --2- 172.21.15.7:0/902059040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13dc075820 0x7f13dc073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.439+0000 7f13e1800640 1 -- 172.21.15.7:0/902059040 >> 172.21.15.7:0/902059040 conn(0x7f13dc0fcb90 msgr2=0x7f13dc0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.439+0000 7f13e1800640 1 -- 172.21.15.7:0/902059040 shutdown_connections 2024-03-18T01:37:12.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.439+0000 7f13e1800640 1 -- 172.21.15.7:0/902059040 wait complete. 2024-03-18T01:37:12.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.440+0000 7f13e1800640 1 Processor -- start 2024-03-18T01:37:12.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.440+0000 7f13e1800640 1 -- start start 2024-03-18T01:37:12.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.440+0000 7f13e1800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 0x7f13dc19df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:12.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.440+0000 7f13e1800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13dc075820 0x7f13dc19e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:12.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.440+0000 7f13e1800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc10dbf0 0x7f13dc1a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:12.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.440+0000 7f13e1800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc113870 con 0x7f13dc10dbf0 2024-03-18T01:37:12.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.440+0000 7f13e1800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1136f0 con 0x7f13dc075820 2024-03-18T01:37:12.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.440+0000 7f13e1800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f13dc1139f0 con 0x7f13dc074240 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13db400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13dc075820 0x7f13dc19e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 0x7f13dc19df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13db400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13dc075820 0x7f13dc19e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53776/0 (socket says 172.21.15.7:53776) 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 0x7f13dc19df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39482/0 (socket says 172.21.15.7:39482) 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13db400640 1 -- 172.21.15.7:0/1627136486 learned_addr learned my addr 172.21.15.7:0/1627136486 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13e0e00640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc10dbf0 0x7f13dc1a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13dbe00640 1 -- 172.21.15.7:0/1627136486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13dc075820 msgr2=0x7f13dc19e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13dbe00640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13dc075820 0x7f13dc19e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13dbe00640 1 -- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc10dbf0 msgr2=0x7f13dc1a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13dbe00640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc10dbf0 0x7f13dc1a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13dbe00640 1 -- 172.21.15.7:0/1627136486 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc1a5b40 con 0x7f13dc074240 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13db400640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13dc075820 0x7f13dc19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:12.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.441+0000 7f13dbe00640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 0x7f13dc19df20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f13cc005fd0 tx=0x7f13cc00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:12.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.442+0000 7f13d8c00640 1 -- 172.21.15.7:0/1627136486 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc00cd00 con 0x7f13dc074240 2024-03-18T01:37:12.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.442+0000 7f13d8c00640 1 -- 172.21.15.7:0/1627136486 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13cc0035f0 con 0x7f13dc074240 2024-03-18T01:37:12.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.442+0000 7f13d8c00640 1 -- 172.21.15.7:0/1627136486 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc010690 con 0x7f13dc074240 2024-03-18T01:37:12.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.442+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13dc1a5e30 con 0x7f13dc074240 2024-03-18T01:37:12.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.442+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13dc1a62a0 con 0x7f13dc074240 2024-03-18T01:37:12.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.443+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13dc103dc0 con 0x7f13dc074240 2024-03-18T01:37:12.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.444+0000 7f13d8c00640 1 -- 172.21.15.7:0/1627136486 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f13cc0037a0 con 0x7f13dc074240 2024-03-18T01:37:12.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.447+0000 7f13d8c00640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f13c00a2800 0x7f13c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:12.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.447+0000 7f13d8c00640 1 -- 172.21.15.7:0/1627136486 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f13cc0cf060 con 0x7f13dc074240 2024-03-18T01:37:12.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.447+0000 7f13db400640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f13c00a2800 0x7f13c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:12.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.448+0000 7f13db400640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f13c00a2800 0x7f13c00a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f13dc19f430 tx=0x7f13c802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:12.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.450+0000 7f13d8c00640 1 -- 172.21.15.7:0/1627136486 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f13cc09bd40 con 0x7f13dc074240 2024-03-18T01:37:12.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:12 smithi007 ceph-mon[31996]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T01:37:12.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:12 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/797885831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:12.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:12 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1679900278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:12 smithi203 ceph-mon[36345]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T01:37:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:12 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/797885831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:12 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1679900278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:12.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.655+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f13dc062480 con 0x7f13dc074240 2024-03-18T01:37:12.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.655+0000 7f13d8c00640 1 -- 172.21.15.7:0/1627136486 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f13cc097030 con 0x7f13dc074240 2024-03-18T01:37:12.656 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:37:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f13c00a2800 msgr2=0x7f13c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f13c00a2800 0x7f13c00a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f13dc19f430 tx=0x7f13c802e040 comp rx=0 tx=0).stop 2024-03-18T01:37:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 msgr2=0x7f13dc19df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 0x7f13dc19df20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f13cc005fd0 tx=0x7f13cc00ee50 comp rx=0 tx=0).stop 2024-03-18T01:37:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 shutdown_connections 2024-03-18T01:37:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f13c00a2800 0x7f13c00a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f13dc10dbf0 0x7f13dc1a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f13dc075820 0x7f13dc19e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 --2- 172.21.15.7:0/1627136486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13dc074240 0x7f13dc19df20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 >> 172.21.15.7:0/1627136486 conn(0x7f13dc0fcb90 msgr2=0x7f13dc100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 shutdown_connections 2024-03-18T01:37:12.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.659+0000 7f13e1800640 1 -- 172.21.15.7:0/1627136486 wait complete. 2024-03-18T01:37:12.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:37:12.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:12 smithi046 ceph-mon[36108]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T01:37:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:12 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/797885831' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:12 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1679900278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:12.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 -- 172.21.15.7:0/1790596330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa70101630 msgr2=0x7faa70101a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 --2- 172.21.15.7:0/1790596330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa70101630 0x7faa70101a50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faa64009900 tx=0x7faa6402bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:12.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 -- 172.21.15.7:0/1790596330 shutdown_connections 2024-03-18T01:37:12.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 --2- 172.21.15.7:0/1790596330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa701031e0 0x7faa70109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 --2- 172.21.15.7:0/1790596330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa70102820 0x7faa70102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 --2- 172.21.15.7:0/1790596330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa70101630 0x7faa70101a50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 -- 172.21.15.7:0/1790596330 >> 172.21.15.7:0/1790596330 conn(0x7faa700fcb80 msgr2=0x7faa700fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:12.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 -- 172.21.15.7:0/1790596330 shutdown_connections 2024-03-18T01:37:12.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.942+0000 7faa76800640 1 -- 172.21.15.7:0/1790596330 wait complete. 2024-03-18T01:37:12.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.943+0000 7faa76800640 1 Processor -- start 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.943+0000 7faa76800640 1 -- start start 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.943+0000 7faa76800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa70101630 0x7faa70199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa76800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa70102820 0x7faa70199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa76800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa701031e0 0x7faa701a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa76800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010d300 con 0x7faa70101630 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa76800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010d180 con 0x7faa70102820 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa76800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa7010d480 con 0x7faa701031e0 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa70101630 0x7faa70199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa70101630 0x7faa70199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53460/0 (socket says 172.21.15.7:53460) 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75400640 1 -- 172.21.15.7:0/729464541 learned_addr learned my addr 172.21.15.7:0/729464541 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75e00640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa701031e0 0x7faa701a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa74a00640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa70102820 0x7faa70199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:12.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75400640 1 -- 172.21.15.7:0/729464541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa701031e0 msgr2=0x7faa701a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75400640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa701031e0 0x7faa701a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75400640 1 -- 172.21.15.7:0/729464541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa70102820 msgr2=0x7faa70199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:12.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75400640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa70102820 0x7faa70199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:12.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75400640 1 -- 172.21.15.7:0/729464541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa701a16a0 con 0x7faa70101630 2024-03-18T01:37:12.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa75e00640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa701031e0 0x7faa701a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:12.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.944+0000 7faa74a00640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa70102820 0x7faa70199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:12.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.945+0000 7faa75400640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa70101630 0x7faa70199a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa6402efd0 tx=0x7faa64005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:12.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.945+0000 7faa6e000640 1 -- 172.21.15.7:0/729464541 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64041070 con 0x7faa70101630 2024-03-18T01:37:12.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.945+0000 7faa6e000640 1 -- 172.21.15.7:0/729464541 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa640044d0 con 0x7faa70101630 2024-03-18T01:37:12.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.945+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa701a1930 con 0x7faa70101630 2024-03-18T01:37:12.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.945+0000 7faa6e000640 1 -- 172.21.15.7:0/729464541 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa64034440 con 0x7faa70101630 2024-03-18T01:37:12.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.945+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa701a1df0 con 0x7faa70101630 2024-03-18T01:37:12.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.946+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa701a20b0 con 0x7faa70101630 2024-03-18T01:37:12.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.948+0000 7faa6e000640 1 -- 172.21.15.7:0/729464541 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7faa64044070 con 0x7faa70101630 2024-03-18T01:37:12.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.950+0000 7faa6e000640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faa580a28a0 0x7faa580a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:12.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.950+0000 7faa6e000640 1 -- 172.21.15.7:0/729464541 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7faa640f09b0 con 0x7faa70101630 2024-03-18T01:37:12.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.950+0000 7faa74a00640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faa580a28a0 0x7faa580a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:12.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.951+0000 7faa74a00640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faa580a28a0 0x7faa580a4d60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faa7019af90 tx=0x7faa5c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:12.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:12.955+0000 7faa6e000640 1 -- 172.21.15.7:0/729464541 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa640bd040 con 0x7faa70101630 2024-03-18T01:37:13.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.200+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa701a2440 con 0x7faa70101630 2024-03-18T01:37:13.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.203+0000 7faa6e000640 1 -- 172.21.15.7:0/729464541 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11918 (secure 0 0 0) 0x7faa640c10e0 con 0x7faa70101630 2024-03-18T01:37:13.203 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:13.204 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:35:30.634148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24542,24523,24544,24596,24562]},"id":1}]} 2024-03-18T01:37:13.204 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2024-03-18T01:37:13.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.205+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faa580a28a0 msgr2=0x7faa580a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.205+0000 7faa76800640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faa580a28a0 0x7faa580a4d60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faa7019af90 tx=0x7faa5c00b390 comp rx=0 tx=0).stop 2024-03-18T01:37:13.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.205+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa70101630 msgr2=0x7faa70199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.205+0000 7faa76800640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa70101630 0x7faa70199a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa6402efd0 tx=0x7faa64005d50 comp rx=0 tx=0).stop 2024-03-18T01:37:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.206+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 shutdown_connections 2024-03-18T01:37:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.206+0000 7faa76800640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa701031e0 0x7faa701a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.206+0000 7faa76800640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faa70102820 0x7faa70199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.206+0000 7faa76800640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7faa580a28a0 0x7faa580a4d60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.206+0000 7faa76800640 1 --2- 172.21.15.7:0/729464541 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7faa70101630 0x7faa70199a80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.206+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 >> 172.21.15.7:0/729464541 conn(0x7faa700fcb80 msgr2=0x7faa70073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:13.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.206+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 shutdown_connections 2024-03-18T01:37:13.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.206+0000 7faa76800640 1 -- 172.21.15.7:0/729464541 wait complete. 2024-03-18T01:37:13.227 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:37:13.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 -- 172.21.15.7:0/4119609359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1022b0 msgr2=0x7ff2cc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 --2- 172.21.15.7:0/4119609359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1022b0 0x7ff2cc108b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4009900 tx=0x7ff2c402bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:13.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 -- 172.21.15.7:0/4119609359 shutdown_connections 2024-03-18T01:37:13.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 --2- 172.21.15.7:0/4119609359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1022b0 0x7ff2cc108b40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 --2- 172.21.15.7:0/4119609359 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2cc1018f0 0x7ff2cc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 --2- 172.21.15.7:0/4119609359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2cc100700 0x7ff2cc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 -- 172.21.15.7:0/4119609359 >> 172.21.15.7:0/4119609359 conn(0x7ff2cc0fbc70 msgr2=0x7ff2cc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:13.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 -- 172.21.15.7:0/4119609359 shutdown_connections 2024-03-18T01:37:13.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.438+0000 7ff2d3000640 1 -- 172.21.15.7:0/4119609359 wait complete. 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.439+0000 7ff2d3000640 1 Processor -- start 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.439+0000 7ff2d3000640 1 -- start start 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2d3000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2cc100700 0x7ff2cc10b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2d3000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1018f0 0x7ff2cc1096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2d3000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2cc1022b0 0x7ff2cc109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2d3000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc10d4c0 con 0x7ff2cc1022b0 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2d3000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc10d340 con 0x7ff2cc1018f0 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2d3000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc10d640 con 0x7ff2cc100700 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1018f0 0x7ff2cc1096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:13.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1018f0 0x7ff2cc1096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53824/0 (socket says 172.21.15.7:53824) 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cb400640 1 -- 172.21.15.7:0/2056442037 learned_addr learned my addr 172.21.15.7:0/2056442037 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2d1000640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2cc1022b0 0x7ff2cc109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cbe00640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2cc100700 0x7ff2cc10b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cb400640 1 -- 172.21.15.7:0/2056442037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2cc100700 msgr2=0x7ff2cc10b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cb400640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2cc100700 0x7ff2cc10b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cb400640 1 -- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2cc1022b0 msgr2=0x7ff2cc109c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cb400640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2cc1022b0 0x7ff2cc109c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cb400640 1 -- 172.21.15.7:0/2056442037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc10a490 con 0x7ff2cc1018f0 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.440+0000 7ff2cbe00640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2cc100700 0x7ff2cc10b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.441+0000 7ff2cb400640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1018f0 0x7ff2cc1096c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400ebc0 tx=0x7ff2b400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:13.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.441+0000 7ff2c8c00640 1 -- 172.21.15.7:0/2056442037 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4010630 con 0x7ff2cc1018f0 2024-03-18T01:37:13.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.441+0000 7ff2c8c00640 1 -- 172.21.15.7:0/2056442037 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b4010cb0 con 0x7ff2cc1018f0 2024-03-18T01:37:13.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.441+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2cc1a1ba0 con 0x7ff2cc1018f0 2024-03-18T01:37:13.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.441+0000 7ff2c8c00640 1 -- 172.21.15.7:0/2056442037 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4018bd0 con 0x7ff2cc1018f0 2024-03-18T01:37:13.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.441+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2cc1a2140 con 0x7ff2cc1018f0 2024-03-18T01:37:13.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.442+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2cc107950 con 0x7ff2cc1018f0 2024-03-18T01:37:13.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.445+0000 7ff2c8c00640 1 -- 172.21.15.7:0/2056442037 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7ff2b40107d0 con 0x7ff2cc1018f0 2024-03-18T01:37:13.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.448+0000 7ff2c8c00640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff2b00a2880 0x7ff2b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:13.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.448+0000 7ff2cbe00640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff2b00a2880 0x7ff2b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:13.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.449+0000 7ff2c8c00640 1 -- 172.21.15.7:0/2056442037 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff2b40cf350 con 0x7ff2cc1018f0 2024-03-18T01:37:13.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.449+0000 7ff2cbe00640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff2b00a2880 0x7ff2b00a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc101750 tx=0x7ff2bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:13.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.449+0000 7ff2c8c00640 1 -- 172.21.15.7:0/2056442037 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff2b4098050 con 0x7ff2cc1018f0 2024-03-18T01:37:13.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:13 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1627136486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:13.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:13 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/729464541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:13 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1627136486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:13 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/729464541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:13.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.655+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff2cc02d3d0 con 0x7ff2cc1018f0 2024-03-18T01:37:13.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.656+0000 7ff2c8c00640 1 -- 172.21.15.7:0/2056442037 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7ff2b409b9e0 con 0x7ff2cc1018f0 2024-03-18T01:37:13.657 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:13.657 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:37:13.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff2b00a2880 msgr2=0x7ff2b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff2b00a2880 0x7ff2b00a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc101750 tx=0x7ff2bc005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1018f0 msgr2=0x7ff2cc1096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1018f0 0x7ff2cc1096c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff2b400ebc0 tx=0x7ff2b400c6a0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 shutdown_connections 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff2b00a2880 0x7ff2b00a4d40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff2cc1022b0 0x7ff2cc109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2cc1018f0 0x7ff2cc1096c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 --2- 172.21.15.7:0/2056442037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2cc100700 0x7ff2cc10b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 >> 172.21.15.7:0/2056442037 conn(0x7ff2cc0fbc70 msgr2=0x7ff2cc104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 shutdown_connections 2024-03-18T01:37:13.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.659+0000 7ff2d3000640 1 -- 172.21.15.7:0/2056442037 wait complete. 2024-03-18T01:37:13.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:37:13.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:13 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1627136486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:13.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:13 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/729464541' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.888+0000 7ff1a6e00640 1 -- 172.21.15.7:0/1610435749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01017e0 msgr2=0x7ff1a0101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.888+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/1610435749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01017e0 0x7ff1a0101c00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff190009860 tx=0x7ff19002bb70 comp rx=0 tx=0).stop 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.888+0000 7ff1a6e00640 1 -- 172.21.15.7:0/1610435749 shutdown_connections 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.888+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/1610435749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a0103390 0x7ff1a0109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.888+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/1610435749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1a01029d0 0x7ff1a0102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.888+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/1610435749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01017e0 0x7ff1a0101c00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.888+0000 7ff1a6e00640 1 -- 172.21.15.7:0/1610435749 >> 172.21.15.7:0/1610435749 conn(0x7ff1a00fcd50 msgr2=0x7ff1a00ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.888+0000 7ff1a6e00640 1 -- 172.21.15.7:0/1610435749 shutdown_connections 2024-03-18T01:37:13.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.889+0000 7ff1a6e00640 1 -- 172.21.15.7:0/1610435749 wait complete. 2024-03-18T01:37:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.889+0000 7ff1a6e00640 1 Processor -- start 2024-03-18T01:37:13.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff1a6e00640 1 -- start start 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff1a6e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a01017e0 0x7ff1a0195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff1a6e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01029d0 0x7ff1a0195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff1a6e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1a0103390 0x7ff1a019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff1a6e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a010b280 con 0x7ff1a0103390 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff1a6e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a010b100 con 0x7ff1a01017e0 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff1a6e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a010b400 con 0x7ff1a01029d0 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff19fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a01017e0 0x7ff1a0195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff19fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a01017e0 0x7ff1a0195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53842/0 (socket says 172.21.15.7:53842) 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff19fe00640 1 -- 172.21.15.7:0/3191877998 learned_addr learned my addr 172.21.15.7:0/3191877998 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff197e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01029d0 0x7ff1a0195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.890+0000 7ff1a4e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1a0103390 0x7ff1a019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19fe00640 1 -- 172.21.15.7:0/3191877998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01029d0 msgr2=0x7ff1a0195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19fe00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01029d0 0x7ff1a0195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19fe00640 1 -- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1a0103390 msgr2=0x7ff1a019ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19fe00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1a0103390 0x7ff1a019ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19fe00640 1 -- 172.21.15.7:0/3191877998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a019d520 con 0x7ff1a01017e0 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff197e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01029d0 0x7ff1a0195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19fe00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a01017e0 0x7ff1a0195900 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff19002c0b0 tx=0x7ff190032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19d600640 1 -- 172.21.15.7:0/3191877998 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190041020 con 0x7ff1a01017e0 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a019d7b0 con 0x7ff1a01017e0 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19d600640 1 -- 172.21.15.7:0/3191877998 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff19002c9f0 con 0x7ff1a01017e0 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff19d600640 1 -- 172.21.15.7:0/3191877998 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190039040 con 0x7ff1a01017e0 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.891+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a019dca0 con 0x7ff1a01017e0 2024-03-18T01:37:13.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.892+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a0108b80 con 0x7ff1a01017e0 2024-03-18T01:37:13.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.895+0000 7ff19d600640 1 -- 172.21.15.7:0/3191877998 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7ff190044020 con 0x7ff1a01017e0 2024-03-18T01:37:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.896+0000 7ff19d600640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff1840a2800 0x7ff1840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:13.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.896+0000 7ff19d600640 1 -- 172.21.15.7:0/3191877998 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff1900f0a40 con 0x7ff1a01017e0 2024-03-18T01:37:13.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.896+0000 7ff197e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff1840a2800 0x7ff1840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:13.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.897+0000 7ff197e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff1840a2800 0x7ff1840a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0196e10 tx=0x7ff188005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:13.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:13.901+0000 7ff19d600640 1 -- 172.21.15.7:0/3191877998 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff19004b020 con 0x7ff1a01017e0 2024-03-18T01:37:14.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.143+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1a002d1b0 con 0x7ff1a01017e0 2024-03-18T01:37:14.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.144+0000 7ff19d600640 1 -- 172.21.15.7:0/3191877998 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11918 (secure 0 0 0) 0x7ff1900bd780 con 0x7ff1a01017e0 2024-03-18T01:37:14.145 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:14.145 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:35:30.634148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24542,24523,24544,24596,24562]},"id":1}]} 2024-03-18T01:37:14.145 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2024-03-18T01:37:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.147+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff1840a2800 msgr2=0x7ff1840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.147+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff1840a2800 0x7ff1840a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0196e10 tx=0x7ff188005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.147+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a01017e0 msgr2=0x7ff1a0195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a01017e0 0x7ff1a0195900 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff19002c0b0 tx=0x7ff190032430 comp rx=0 tx=0).stop 2024-03-18T01:37:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 shutdown_connections 2024-03-18T01:37:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff1840a2800 0x7ff1840a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.148 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff1a0103390 0x7ff1a019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1a01029d0 0x7ff1a0195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 --2- 172.21.15.7:0/3191877998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1a01017e0 0x7ff1a0195900 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 >> 172.21.15.7:0/3191877998 conn(0x7ff1a00fcd50 msgr2=0x7ff1a0105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 shutdown_connections 2024-03-18T01:37:14.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.148+0000 7ff1a6e00640 1 -- 172.21.15.7:0/3191877998 wait complete. 2024-03-18T01:37:14.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T01:34:56.125856+0000', 'modified': '2024-03-18T01:35:30.634148+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24596, 'mds_1': 24562, 'mds_2': 24544, 'mds_3': 24542, 'mds_4': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24448': {'gid': 24448, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/2100401567', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 2100401567}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 2100401567}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.46:6833/1027889667', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1027889667}, {'type': 'v1', 'addr': '172.21.15.46:6833', 'nonce': 1027889667}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.7:6835/49989672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6834', 'nonce': 49989672}, {'type': 'v1', 'addr': '172.21.15.7:6835', 'nonce': 49989672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/848080613', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 848080613}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 848080613}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.46:6835/2878820671', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 2878820671}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2878820671}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.203:6837/2344597644', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 2344597644}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 2344597644}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.46:6837/1509246906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6836', 'nonce': 1509246906}, {'type': 'v1', 'addr': '172.21.15.46:6837', 'nonce': 1509246906}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.203:6839/3801010143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 3801010143}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 3801010143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'g', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.7:6839/3166631604', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6838', 'nonce': 3166631604}, {'type': 'v1', 'addr': '172.21.15.7:6839', 'nonce': 3166631604}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24596': {'gid': 24596, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6839/1986652812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6838', 'nonce': 1986652812}, {'type': 'v1', 'addr': '172.21.15.46:6839', 'nonce': 1986652812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24596, 'qdb_cluster': [24542, 24523, 24544, 24596, 24562]} 2024-03-18T01:37:14.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T01:37:14.169 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T01:37:14.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.388+0000 7f7dc8e00640 1 -- 172.21.15.7:0/2388956807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc4108e30 msgr2=0x7f7dc41092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.388+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/2388956807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc4108e30 0x7f7dc41092b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009ca0 tx=0x7f7db402c100 comp rx=0 tx=0).stop 2024-03-18T01:37:14.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.388+0000 7f7dc8e00640 1 -- 172.21.15.7:0/2388956807 shutdown_connections 2024-03-18T01:37:14.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.388+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/2388956807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dc41097f0 0x7f7dc4110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.388+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/2388956807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc4108e30 0x7f7dc41092b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.388+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/2388956807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4107c40 0x7f7dc4108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.388+0000 7f7dc8e00640 1 -- 172.21.15.7:0/2388956807 >> 172.21.15.7:0/2388956807 conn(0x7f7dc4076bd0 msgr2=0x7f7dc4079010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.389+0000 7f7dc8e00640 1 -- 172.21.15.7:0/2388956807 shutdown_connections 2024-03-18T01:37:14.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.389+0000 7f7dc8e00640 1 -- 172.21.15.7:0/2388956807 wait complete. 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.390+0000 7f7dc8e00640 1 Processor -- start 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.390+0000 7f7dc8e00640 1 -- start start 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.390+0000 7f7dc8e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4107c40 0x7f7dc419df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.390+0000 7f7dc8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dc4108e30 0x7f7dc419e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.390+0000 7f7dc8e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc41097f0 0x7f7dc41a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.390+0000 7f7dc8e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc41138c0 con 0x7f7dc4107c40 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.390+0000 7f7dc8e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4113740 con 0x7f7dc41097f0 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.390+0000 7f7dc8e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc4113a40 con 0x7f7dc4108e30 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4107c40 0x7f7dc419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dc4108e30 0x7f7dc419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc41097f0 0x7f7dc41a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dc4108e30 0x7f7dc419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39528/0 (socket says 172.21.15.7:39528) 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4107c40 0x7f7dc419df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53490/0 (socket says 172.21.15.7:53490) 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 -- 172.21.15.7:0/3428322016 learned_addr learned my addr 172.21.15.7:0/3428322016 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 -- 172.21.15.7:0/3428322016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc41097f0 msgr2=0x7f7dc41a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc41097f0 0x7f7dc41a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 -- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4107c40 msgr2=0x7f7dc419df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4107c40 0x7f7dc419df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 -- 172.21.15.7:0/3428322016 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc41a5b50 con 0x7f7dc4108e30 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.391+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dc4108e30 0x7f7dc419e470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7db40325f0 tx=0x7f7db4035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.392+0000 7f7dbbe00640 1 -- 172.21.15.7:0/3428322016 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4039040 con 0x7f7dc4108e30 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.392+0000 7f7dbbe00640 1 -- 172.21.15.7:0/3428322016 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db4044070 con 0x7f7dc4108e30 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.392+0000 7f7dbbe00640 1 -- 172.21.15.7:0/3428322016 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40044d0 con 0x7f7dc4108e30 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.392+0000 7f7dc8e00640 1 -- 172.21.15.7:0/3428322016 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc41a5de0 con 0x7f7dc4108e30 2024-03-18T01:37:14.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.392+0000 7f7dc8e00640 1 -- 172.21.15.7:0/3428322016 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc41a61f0 con 0x7f7dc4108e30 2024-03-18T01:37:14.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.396+0000 7f7dbbe00640 1 -- 172.21.15.7:0/3428322016 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f7db4004670 con 0x7f7dc4108e30 2024-03-18T01:37:14.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.396+0000 7f7dcd01a740 1 Processor -- start 2024-03-18T01:37:14.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.396+0000 7f7dcd01a740 1 -- start start 2024-03-18T01:37:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e6473760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6478830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6478ea0 0x55b6e647b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b6e63d7c50 con 0x55b6e6478ea0 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b6e63809d0 con 0x55b6e6473d30 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b6e63d3cf0 con 0x55b6e6473340 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6478830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e6473760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6478830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53868/0 (socket says 172.21.15.7:53868) 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc3400640 1 -- 172.21.15.7:0/2973489737 learned_addr learned my addr 172.21.15.7:0/2973489737 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/2973489737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6478ea0 0x55b6e647b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc3400640 1 -- 172.21.15.7:0/2973489737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 msgr2=0x55b6e6473760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc3400640 1 --2- 172.21.15.7:0/2973489737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e6473760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc3400640 1 -- 172.21.15.7:0/2973489737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6478ea0 msgr2=0x55b6e647b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.397+0000 7f7dc3400640 1 --2- 172.21.15.7:0/2973489737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6478ea0 0x55b6e647b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.398+0000 7f7dc3400640 1 -- 172.21.15.7:0/2973489737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6e639af10 con 0x55b6e6473d30 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.398+0000 7f7dc3400640 1 --2- 172.21.15.7:0/2973489737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6478830 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7dac009c80 tx=0x7f7dac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d61674f5e05b1d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.398+0000 7f7dbbe00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7da80a2880 0x7f7da80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.398+0000 7f7db8c00640 1 -- 172.21.15.7:0/2973489737 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dac01e070 con 0x55b6e6473d30 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.398+0000 7f7db8c00640 1 -- 172.21.15.7:0/2973489737 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dac009e30 con 0x55b6e6473d30 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.398+0000 7f7db8c00640 1 -- 172.21.15.7:0/2973489737 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dac00ea80 con 0x55b6e6473d30 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.398+0000 7f7dbbe00640 1 -- 172.21.15.7:0/3428322016 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7db40f12d0 con 0x7f7dc4108e30 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.398+0000 7f7dc3e00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7da80a2880 0x7f7da80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 -- 172.21.15.7:0/2973489737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 msgr2=0x55b6e6478830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/2973489737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6478830 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7dac009c80 tx=0x7f7dac011810 comp rx=0 tx=0).stop 2024-03-18T01:37:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 -- 172.21.15.7:0/2973489737 shutdown_connections 2024-03-18T01:37:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/2973489737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6478ea0 0x55b6e647b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/2973489737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6478830 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/2973489737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e6473760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 -- 172.21.15.7:0/2973489737 >> 172.21.15.7:0/2973489737 conn(0x55b6e646e8b0 msgr2=0x55b6e646ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 -- 172.21.15.7:0/2973489737 shutdown_connections 2024-03-18T01:37:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dc3e00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7da80a2880 0x7f7da80a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55b6e6474590 tx=0x7f7dbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.399+0000 7f7dcd01a740 1 -- 172.21.15.7:0/2973489737 wait complete. 2024-03-18T01:37:14.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.400+0000 7f7dcd01a740 1 -- start start 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.400+0000 7f7dcd01a740 1 Processor -- start 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473d30 0x55b6e647da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 0x55b6e6484aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b6e647be50 con 0x55b6e6473340 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b6e6393e20 con 0x55b6e6478ea0 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b6e647bfd0 con 0x55b6e6473d30 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dcd01a740 10 client.0 _ll_register_callbacks cb 0x55b6e63937f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 0x55b6e6484aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473d30 0x55b6e647da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473d30 0x55b6e647da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39554/0 (socket says 172.21.15.7:39554) 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc2a00640 1 -- 172.21.15.7:0/554692815 learned_addr learned my addr 172.21.15.7:0/554692815 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc2a00640 1 -- 172.21.15.7:0/554692815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 msgr2=0x55b6e6484aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 0x55b6e6484aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc2a00640 1 -- 172.21.15.7:0/554692815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 msgr2=0x55b6e647d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.401+0000 7f7dc2a00640 1 -- 172.21.15.7:0/554692815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6e60ec2d0 con 0x55b6e6473d30 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.402+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473d30 0x55b6e647da60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7db4000f80 tx=0x7f7db4035f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=102b8450f14a04a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.402+0000 7f7db1600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.402+0000 7f7db1600640 1 -- 172.21.15.7:0/554692815 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40b9b50 con 0x55b6e6473d30 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.402+0000 7f7db1600640 1 -- 172.21.15.7:0/554692815 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7db40a80b0 con 0x55b6e6473d30 2024-03-18T01:37:14.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.402+0000 7f7dcd01a740 1 -- 172.21.15.7:0/554692815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dac009e30 con 0x55b6e6473d30 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.402+0000 7f7db1600640 1 -- 172.21.15.7:0/554692815 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7db40bd210 con 0x55b6e6473d30 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.402+0000 7f7dcd01a740 10 client.25162 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.402+0000 7f7dcd01a740 1 -- 172.21.15.7:0/554692815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b6e61b66b0 con 0x55b6e6473d30 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7db1600640 1 -- 172.21.15.7:0/554692815 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dac009e30 con 0x55b6e6473d30 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7db1600640 1 -- 172.21.15.7:0/554692815 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7db403c030 con 0x55b6e6473d30 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7dcd01a740 10 client.25162 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7dcd01a740 10 client.25162 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7dcd01a740 20 client.25162 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55b6e648a2e0 0x55b6e648b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7dcd01a740 4 client.25162 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7dcd01a740 1 -- 172.21.15.7:0/554692815 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6e64352b0 con 0x55b6e648a2e0 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.403+0000 7f7dc3400640 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55b6e648a2e0 0x55b6e648b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.404+0000 7f7dc3400640 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55b6e648a2e0 0x55b6e648b750 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7c420f83b90acfdc server_cookie=14a820f981798b76 in_seq=0 out_seq=0 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.404+0000 7f7db1600640 10 client.25162 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:37:14.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7db1600640 1 -- 172.21.15.7:0/554692815 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b6e64352b0 con 0x55b6e648a2e0 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7db1600640 10 client.25162 handle_command_reply: tid=0 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 client.25162 shutdown 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 -- 172.21.15.7:0/554692815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473d30 msgr2=0x55b6e647da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473d30 0x55b6e647da60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7db4000f80 tx=0x7f7db4035f80 comp rx=0 tx=0).stop 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 -- 172.21.15.7:0/554692815 shutdown_connections 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55b6e648a2e0 0x55b6e648b750 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 0x55b6e6484aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473d30 0x55b6e647da60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/554692815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 -- 172.21.15.7:0/554692815 >> 172.21.15.7:0/554692815 conn(0x55b6e646e8b0 msgr2=0x55b6e646da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7db1600640 0 client.25162 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 -- 172.21.15.7:0/554692815 shutdown_connections 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 1 -- 172.21.15.7:0/554692815 wait complete. 2024-03-18T01:37:14.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.405+0000 7f7dcd01a740 20 client.25162 trim_cache size 0 max 16384 2024-03-18T01:37:14.425 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dcd01a740 1 Processor -- start 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dcd01a740 1 -- start start 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6484a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6478ea0 0x55b6e647da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b6e6489a90 con 0x55b6e6473340 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b6e648a160 con 0x55b6e6473d30 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b6e648a830 con 0x55b6e6478ea0 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.425+0000 7f7dc3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6478ea0 0x55b6e647da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6478ea0 0x55b6e647da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39560/0 (socket says 172.21.15.7:39560) 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc3e00640 1 -- 172.21.15.7:0/1998295806 learned_addr learned my addr 172.21.15.7:0/1998295806 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6484a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc3400640 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc2a00640 1 -- 172.21.15.7:0/1998295806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6478ea0 msgr2=0x55b6e647da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6478ea0 0x55b6e647da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc2a00640 1 -- 172.21.15.7:0/1998295806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 msgr2=0x55b6e647d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc2a00640 1 -- 172.21.15.7:0/1998295806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6e646f620 con 0x55b6e6473d30 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc3400640 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6484a60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7db402c710 tx=0x7f7db40f11f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78202947ae6d8a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7db3e00640 1 -- 172.21.15.7:0/1998295806 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40bd3f0 con 0x55b6e6473d30 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7db3e00640 1 -- 172.21.15.7:0/1998295806 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db40f4900 con 0x55b6e6473d30 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.426+0000 7f7db3e00640 1 -- 172.21.15.7:0/1998295806 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db40bd680 con 0x55b6e6473d30 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1998295806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 msgr2=0x55b6e6484a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6484a60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7db402c710 tx=0x7f7db40f11f0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1998295806 shutdown_connections 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6478ea0 0x55b6e647da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6473d30 0x55b6e6484a60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1998295806 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473340 0x55b6e647d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1998295806 >> 172.21.15.7:0/1998295806 conn(0x55b6e646e8b0 msgr2=0x55b6e6478510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1998295806 shutdown_connections 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.427+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1998295806 wait complete. 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 1 -- start start 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 1 Processor -- start 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e64805b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473d30 0x55b6e6480af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 0x55b6e6483b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b6e648b5d0 con 0x55b6e6473d30 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b6e648af00 con 0x55b6e6478ea0 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b6e648bca0 con 0x55b6e6473340 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dcd01a740 10 client.0 _ll_register_callbacks cb 0x7f7db4003ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e64805b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dc2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473d30 0x55b6e6480af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e64805b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39568/0 (socket says 172.21.15.7:39568) 2024-03-18T01:37:14.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.428+0000 7f7dc3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 0x55b6e6483b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7dc3400640 1 -- 172.21.15.7:0/1669798229 learned_addr learned my addr 172.21.15.7:0/1669798229 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7dc3400640 1 -- 172.21.15.7:0/1669798229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 msgr2=0x55b6e6483b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7dc3400640 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 0x55b6e6483b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7dc3400640 1 -- 172.21.15.7:0/1669798229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473d30 msgr2=0x55b6e6480af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7dc3400640 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473d30 0x55b6e6480af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7dc3400640 1 -- 172.21.15.7:0/1669798229 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6e6484100 con 0x55b6e6473340 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7dc3400640 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e64805b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7dac01c040 tx=0x7f7dac009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=330c4f4e0e25119 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7db1600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:14.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7db1600640 1 -- 172.21.15.7:0/1669798229 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dac032060 con 0x55b6e6473340 2024-03-18T01:37:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7db1600640 1 -- 172.21.15.7:0/1669798229 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7dac009e30 con 0x55b6e6473340 2024-03-18T01:37:14.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.429+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1669798229 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7db40f4900 con 0x55b6e6473340 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7db1600640 1 -- 172.21.15.7:0/1669798229 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7dac039050 con 0x55b6e6473340 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7dcd01a740 10 client.25168 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1669798229 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b6e64f0490 con 0x55b6e6473340 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7db1600640 1 -- 172.21.15.7:0/1669798229 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db40f4900 con 0x55b6e6473340 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7db1600640 1 -- 172.21.15.7:0/1669798229 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7dac02c310 con 0x55b6e6473340 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7dcd01a740 10 client.25168 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7dcd01a740 10 client.25168 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7dcd01a740 20 client.25168 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55b6e64f4e70 0x55b6e64f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7dcd01a740 4 client.25168 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:14.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.430+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1669798229 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "status"}) -- 0x55b6e63e5190 con 0x55b6e64f4e70 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.431+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55b6e64f4e70 0x55b6e64f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.431+0000 7f7dc2a00640 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55b6e64f4e70 0x55b6e64f3b80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f4562b0938f21a03 server_cookie=2fcecc71a465e1a5 in_seq=0 out_seq=0 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.431+0000 7f7db1600640 10 client.25168 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.431+0000 7f7db1600640 1 -- 172.21.15.7:0/1669798229 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55b6e63e5190 con 0x55b6e64f4e70 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7db1600640 10 client.25168 handle_command_reply: tid=0 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 client.25168 shutdown 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1669798229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 msgr2=0x55b6e64805b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e64805b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7dac01c040 tx=0x7f7dac009bf0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1669798229 shutdown_connections 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55b6e64f4e70 0x55b6e64f3b80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b6e6478ea0 0x55b6e6483b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b6e6473d30 0x55b6e6480af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7db1600640 0 client.25168 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:37:14.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 --2- 172.21.15.7:0/1669798229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b6e6473340 0x55b6e64805b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1669798229 >> 172.21.15.7:0/1669798229 conn(0x55b6e646e8b0 msgr2=0x55b6e647d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1669798229 shutdown_connections 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 1 -- 172.21.15.7:0/1669798229 wait complete. 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.432+0000 7f7dcd01a740 20 client.25168 trim_cache size 0 max 16384 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 4, 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24523, 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:14.437 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:14.438 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:14.438 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 104.07211981, 2024-03-18T01:37:14.438 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:14.438 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:14.438 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:14.438 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 161.58113657199999 2024-03-18T01:37:14.438 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 -- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7da80a2880 msgr2=0x7f7da80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7da80a2880 0x7f7da80a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55b6e6474590 tx=0x7f7dbc005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 -- 172.21.15.7:0/3428322016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dc4108e30 msgr2=0x7f7dc419e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dc4108e30 0x7f7dc419e470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7db40325f0 tx=0x7f7db4035d10 comp rx=0 tx=0).stop 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 -- 172.21.15.7:0/3428322016 shutdown_connections 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc41097f0 0x7f7dc41a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dc4108e30 0x7f7dc419e470 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7da80a2880 0x7f7da80a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 --2- 172.21.15.7:0/3428322016 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7dc4107c40 0x7f7dc419df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 -- 172.21.15.7:0/3428322016 >> 172.21.15.7:0/3428322016 conn(0x7f7dc4076bd0 msgr2=0x7f7dc410c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 -- 172.21.15.7:0/3428322016 shutdown_connections 2024-03-18T01:37:14.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.433+0000 7f7dc8e00640 1 -- 172.21.15.7:0/3428322016 wait complete. 2024-03-18T01:37:14.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:14 smithi007 ceph-mon[31996]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:14.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:14 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2056442037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:14.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:14 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3191877998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:14.451 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T01:37:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:14 smithi203 ceph-mon[36345]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:14 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2056442037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:14.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:14 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3191877998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:14.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.671+0000 7fb048e00640 1 -- 172.21.15.7:0/3834097529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044075820 msgr2=0x7fb044073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.671+0000 7fb048e00640 1 --2- 172.21.15.7:0/3834097529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044075820 0x7fb044073c70 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb034009860 tx=0x7fb03402bb70 comp rx=0 tx=0).stop 2024-03-18T01:37:14.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.672+0000 7fb048e00640 1 -- 172.21.15.7:0/3834097529 shutdown_connections 2024-03-18T01:37:14.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.672+0000 7fb048e00640 1 --2- 172.21.15.7:0/3834097529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb04410dde0 0x7fb0441101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.672+0000 7fb048e00640 1 --2- 172.21.15.7:0/3834097529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb044074240 0x7fb0440746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.672+0000 7fb048e00640 1 --2- 172.21.15.7:0/3834097529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044075820 0x7fb044073c70 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.672+0000 7fb048e00640 1 -- 172.21.15.7:0/3834097529 >> 172.21.15.7:0/3834097529 conn(0x7fb0440fcda0 msgr2=0x7fb0440ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.672+0000 7fb048e00640 1 -- 172.21.15.7:0/3834097529 shutdown_connections 2024-03-18T01:37:14.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.672+0000 7fb048e00640 1 -- 172.21.15.7:0/3834097529 wait complete. 2024-03-18T01:37:14.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.673+0000 7fb048e00640 1 Processor -- start 2024-03-18T01:37:14.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.673+0000 7fb048e00640 1 -- start start 2024-03-18T01:37:14.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.673+0000 7fb048e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044074240 0x7fb04419e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb048e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb044075820 0x7fb04419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb048e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb04410dde0 0x7fb0441a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb048e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb044113a60 con 0x7fb04410dde0 2024-03-18T01:37:14.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb048e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441138e0 con 0x7fb044075820 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb048e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb044113be0 con 0x7fb044074240 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb044075820 0x7fb04419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb044075820 0x7fb04419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53920/0 (socket says 172.21.15.7:53920) 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041400640 1 -- 172.21.15.7:0/892423798 learned_addr learned my addr 172.21.15.7:0/892423798 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044074240 0x7fb04419e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb042800640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb04410dde0 0x7fb0441a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041400640 1 -- 172.21.15.7:0/892423798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044074240 msgr2=0x7fb04419e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041400640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044074240 0x7fb04419e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041400640 1 -- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb04410dde0 msgr2=0x7fb0441a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041400640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb04410dde0 0x7fb0441a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041400640 1 -- 172.21.15.7:0/892423798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0441a5d30 con 0x7fb044075820 2024-03-18T01:37:14.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.674+0000 7fb041e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044074240 0x7fb04419e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:14.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.675+0000 7fb041400640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb044075820 0x7fb04419e6a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb03000e9c0 tx=0x7fb03000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.675+0000 7fb03aa00640 1 -- 172.21.15.7:0/892423798 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03000cde0 con 0x7fb044075820 2024-03-18T01:37:14.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.675+0000 7fb03aa00640 1 -- 172.21.15.7:0/892423798 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0300035f0 con 0x7fb044075820 2024-03-18T01:37:14.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.675+0000 7fb03aa00640 1 -- 172.21.15.7:0/892423798 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030010690 con 0x7fb044075820 2024-03-18T01:37:14.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.675+0000 7fb048e00640 1 -- 172.21.15.7:0/892423798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0441a5fc0 con 0x7fb044075820 2024-03-18T01:37:14.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.675+0000 7fb048e00640 1 -- 172.21.15.7:0/892423798 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0441a6490 con 0x7fb044075820 2024-03-18T01:37:14.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.678+0000 7fb04ba6d740 1 Processor -- start 2024-03-18T01:37:14.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb04ba6d740 1 -- start start 2024-03-18T01:37:14.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019e0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019e5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e5cc0 0x55a6019e8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a601944c50 con 0x55a6019e5cc0 2024-03-18T01:37:14.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a6018ed9d0 con 0x55a6019e0b50 2024-03-18T01:37:14.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a601940cf0 con 0x55a6019e0160 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb042800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019e0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb042800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019e0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39594/0 (socket says 172.21.15.7:39594) 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb042800640 1 -- 172.21.15.7:0/610746798 learned_addr learned my addr 172.21.15.7:0/610746798 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.679+0000 7fb041e00640 1 --2- 172.21.15.7:0/610746798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb042800640 1 -- 172.21.15.7:0/610746798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 msgr2=0x55a6019e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb042800640 1 --2- 172.21.15.7:0/610746798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb042800640 1 -- 172.21.15.7:0/610746798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e5cc0 msgr2=0x55a6019e8150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb042800640 1 --2- 172.21.15.7:0/610746798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e5cc0 0x55a6019e8150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb042800640 1 -- 172.21.15.7:0/610746798 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a601907f10 con 0x55a6019e0160 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb03aa00640 1 -- 172.21.15.7:0/892423798 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fb0300037a0 con 0x7fb044075820 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb042800640 1 --2- 172.21.15.7:0/610746798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019e0580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009fd0 tx=0x7fb03c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1f253ebd1017e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb02f400640 1 -- 172.21.15.7:0/610746798 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c004d20 con 0x55a6019e0160 2024-03-18T01:37:14.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb02f400640 1 -- 172.21.15.7:0/610746798 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb03c004050 con 0x55a6019e0160 2024-03-18T01:37:14.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.680+0000 7fb02f400640 1 -- 172.21.15.7:0/610746798 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c005260 con 0x55a6019e0160 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 -- 172.21.15.7:0/610746798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 msgr2=0x55a6019e0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/610746798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019e0580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb03c009fd0 tx=0x7fb03c0118d0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 -- 172.21.15.7:0/610746798 shutdown_connections 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/610746798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e5cc0 0x55a6019e8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/610746798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/610746798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019e0580 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 -- 172.21.15.7:0/610746798 >> 172.21.15.7:0/610746798 conn(0x55a6019db6d0 msgr2=0x55a6019dbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 -- 172.21.15.7:0/610746798 shutdown_connections 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb03aa00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb0280a2790 0x7fb0280a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.681+0000 7fb04ba6d740 1 -- 172.21.15.7:0/610746798 wait complete. 2024-03-18T01:37:14.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.682+0000 7fb03aa00640 1 -- 172.21.15.7:0/892423798 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb0300cf090 con 0x7fb044075820 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.682+0000 7fb041e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb0280a2790 0x7fb0280a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.682+0000 7fb04ba6d740 1 -- start start 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.682+0000 7fb04ba6d740 1 Processor -- start 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0160 0x55a6019ea270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a6019ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 0x55a6019f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a6019e8c70 con 0x55a6019e0b50 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a601900e20 con 0x55a6019e0160 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a6019e8df0 con 0x55a6019e5cc0 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb041e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb0280a2790 0x7fb0280a4c50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb034009830 tx=0x7fb03402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb04ba6d740 10 client.0 _ll_register_callbacks cb 0x55a6019007f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:14.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb041e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0160 0x55a6019ea270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb041400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a6019ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb041e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0160 0x55a6019ea270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53934/0 (socket says 172.21.15.7:53934) 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb041e00640 1 -- 172.21.15.7:0/605433580 learned_addr learned my addr 172.21.15.7:0/605433580 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.683+0000 7fb042800640 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 0x55a6019f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb041400640 1 -- 172.21.15.7:0/605433580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 msgr2=0x55a6019f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb041400640 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 0x55a6019f17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb041400640 1 -- 172.21.15.7:0/605433580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0160 msgr2=0x55a6019ea270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb041400640 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0160 0x55a6019ea270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb041400640 1 -- 172.21.15.7:0/605433580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6016592d0 con 0x55a6019e0b50 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb041e00640 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0160 0x55a6019ea270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb041400640 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a6019ea7b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb0300cc7d0 tx=0x7fb03009afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a45e46fdc44c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb023e00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb023e00640 1 -- 172.21.15.7:0/605433580 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0300b4060 con 0x55a6019e0b50 2024-03-18T01:37:14.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb023e00640 1 -- 172.21.15.7:0/605433580 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb0300b4200 con 0x55a6019e0b50 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb04ba6d740 1 -- 172.21.15.7:0/605433580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb03c004050 con 0x55a6019e0b50 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb023e00640 1 -- 172.21.15.7:0/605433580 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0300bb040 con 0x55a6019e0b50 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb04ba6d740 10 client.15036 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.684+0000 7fb04ba6d740 1 -- 172.21.15.7:0/605433580 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a6017236b0 con 0x55a6019e0b50 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb023e00640 1 -- 172.21.15.7:0/605433580 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb03c004050 con 0x55a6019e0b50 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb023e00640 1 -- 172.21.15.7:0/605433580 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb03009ab50 con 0x55a6019e0b50 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb04ba6d740 10 client.15036 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb04ba6d740 10 client.15036 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb04ba6d740 20 client.15036 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55a6019f7040 0x55a6019f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb04ba6d740 4 client.15036 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb04ba6d740 1 -- 172.21.15.7:0/605433580 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6019a22b0 con 0x55a6019f7040 2024-03-18T01:37:14.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.685+0000 7fb041e00640 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55a6019f7040 0x55a6019f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.686+0000 7fb041e00640 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55a6019f7040 0x55a6019f84b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7c662a323d21d1f2 server_cookie=9f946c1494386c4e in_seq=0 out_seq=0 2024-03-18T01:37:14.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.686+0000 7fb023e00640 10 client.15036 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb023e00640 1 -- 172.21.15.7:0/605433580 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a6019a22b0 con 0x55a6019f7040 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb023e00640 10 client.15036 handle_command_reply: tid=0 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 client.15036 shutdown 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 -- 172.21.15.7:0/605433580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 msgr2=0x55a6019ea7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a6019ea7b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb0300cc7d0 tx=0x7fb03009afd0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 -- 172.21.15.7:0/605433580 shutdown_connections 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55a6019f7040 0x55a6019f84b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 0x55a6019f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a6019ea7b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/605433580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0160 0x55a6019ea270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 -- 172.21.15.7:0/605433580 >> 172.21.15.7:0/605433580 conn(0x55a6019db6d0 msgr2=0x55a6019daa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb023e00640 0 client.15036 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:37:14.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 -- 172.21.15.7:0/605433580 shutdown_connections 2024-03-18T01:37:14.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.687+0000 7fb04ba6d740 1 -- 172.21.15.7:0/605433580 wait complete. 2024-03-18T01:37:14.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.688+0000 7fb04ba6d740 20 client.15036 trim_cache size 0 max 16384 2024-03-18T01:37:14.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.709+0000 7fb04ba6d740 1 Processor -- start 2024-03-18T01:37:14.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb04ba6d740 1 -- start start 2024-03-18T01:37:14.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0160 0x55a6019e9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 0x55a6019ea270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a6019f67f0 con 0x55a6019e0160 2024-03-18T01:37:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a6019f6ec0 con 0x55a6019e0b50 2024-03-18T01:37:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a6019f7590 con 0x55a6019e5cc0 2024-03-18T01:37:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb041400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb041e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0160 0x55a6019e9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb041400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019f17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53936/0 (socket says 172.21.15.7:53936) 2024-03-18T01:37:14.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb041400640 1 -- 172.21.15.7:0/2087212125 learned_addr learned my addr 172.21.15.7:0/2087212125 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.710+0000 7fb042800640 1 --2- 172.21.15.7:0/2087212125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 0x55a6019ea270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb041400640 1 -- 172.21.15.7:0/2087212125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 msgr2=0x55a6019ea270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb041400640 1 --2- 172.21.15.7:0/2087212125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 0x55a6019ea270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb041400640 1 -- 172.21.15.7:0/2087212125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0160 msgr2=0x55a6019e9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb041400640 1 --2- 172.21.15.7:0/2087212125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0160 0x55a6019e9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb041400640 1 -- 172.21.15.7:0/2087212125 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6019b1750 con 0x55a6019e0b50 2024-03-18T01:37:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb041400640 1 --2- 172.21.15.7:0/2087212125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019f17b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb03009b600 tx=0x7fb03009b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eebc5c04658cadd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb02ea00640 1 -- 172.21.15.7:0/2087212125 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03009bd70 con 0x55a6019e0b50 2024-03-18T01:37:14.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb02ea00640 1 -- 172.21.15.7:0/2087212125 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb03009f7c0 con 0x55a6019e0b50 2024-03-18T01:37:14.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.711+0000 7fb02ea00640 1 -- 172.21.15.7:0/2087212125 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03009a790 con 0x55a6019e0b50 2024-03-18T01:37:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 -- 172.21.15.7:0/2087212125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 msgr2=0x55a6019f17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/2087212125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019f17b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb03009b600 tx=0x7fb03009b7f0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 -- 172.21.15.7:0/2087212125 shutdown_connections 2024-03-18T01:37:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/2087212125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e5cc0 0x55a6019ea270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/2087212125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e0b50 0x55a6019f17b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/2087212125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0160 0x55a6019e9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 -- 172.21.15.7:0/2087212125 >> 172.21.15.7:0/2087212125 conn(0x55a6019db6d0 msgr2=0x55a6019d6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 -- 172.21.15.7:0/2087212125 shutdown_connections 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.712+0000 7fb04ba6d740 1 -- 172.21.15.7:0/2087212125 wait complete. 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.713+0000 7fb04ba6d740 1 -- start start 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.713+0000 7fb04ba6d740 1 Processor -- start 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.713+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a601a58d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb04ba6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e5cc0 0x55a601a5b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a6019f8330 con 0x55a6019e0b50 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a6019f7c60 con 0x55a6019e5cc0 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb04ba6d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a6019f8a00 con 0x55a6019e0160 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb04ba6d740 10 client.0 _ll_register_callbacks cb 0x7fb03009b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39620/0 (socket says 172.21.15.7:39620) 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a601a58d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 -- 172.21.15.7:0/1950776051 learned_addr learned my addr 172.21.15.7:0/1950776051 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb042800640 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e5cc0 0x55a601a5b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 -- 172.21.15.7:0/1950776051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e5cc0 msgr2=0x55a601a5b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e5cc0 0x55a601a5b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 -- 172.21.15.7:0/1950776051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 msgr2=0x55a601a58d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a601a58d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 -- 172.21.15.7:0/1950776051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a601a5b820 con 0x55a6019e0160 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb041e00640 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019ef7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb034009860 tx=0x7fb03402bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c34cc317bc08c41c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.714+0000 7fb023e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.715+0000 7fb023e00640 1 -- 172.21.15.7:0/1950776051 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034039040 con 0x55a6019e0160 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.715+0000 7fb023e00640 1 -- 172.21.15.7:0/1950776051 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb034052350 con 0x55a6019e0160 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.715+0000 7fb04ba6d740 1 -- 172.21.15.7:0/1950776051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb03009f7c0 con 0x55a6019e0160 2024-03-18T01:37:14.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.715+0000 7fb023e00640 1 -- 172.21.15.7:0/1950776051 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb034060050 con 0x55a6019e0160 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.715+0000 7fb04ba6d740 10 client.25186 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.715+0000 7fb04ba6d740 1 -- 172.21.15.7:0/1950776051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a601a5ca50 con 0x55a6019e0160 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.715+0000 7fb023e00640 1 -- 172.21.15.7:0/1950776051 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb03009f7c0 con 0x55a6019e0160 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.716+0000 7fb023e00640 1 -- 172.21.15.7:0/1950776051 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fb034032760 con 0x55a6019e0160 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.716+0000 7fb04ba6d740 10 client.25186 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.716+0000 7fb04ba6d740 10 client.25186 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.716+0000 7fb04ba6d740 20 client.25186 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.716+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55a601a616a0 0x55a601a5dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.716+0000 7fb04ba6d740 4 client.25186 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.716+0000 7fb04ba6d740 1 -- 172.21.15.7:0/1950776051 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "status"}) -- 0x55a601952190 con 0x55a601a616a0 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.716+0000 7fb041400640 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55a601a616a0 0x55a601a5dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb041400640 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55a601a616a0 0x55a601a5dae0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=de0a68ce02a729cb server_cookie=6b87e5851fbd1dfd in_seq=0 out_seq=0 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb023e00640 10 client.25186 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:37:14.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb023e00640 1 -- 172.21.15.7:0/1950776051 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55a601952190 con 0x55a601a616a0 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb023e00640 10 client.25186 handle_command_reply: tid=0 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 client.25186 shutdown 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 -- 172.21.15.7:0/1950776051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 msgr2=0x55a6019ef7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019ef7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb034009860 tx=0x7fb03402bb70 comp rx=0 tx=0).stop 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 -- 172.21.15.7:0/1950776051 shutdown_connections 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55a601a616a0 0x55a601a5dae0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6019e5cc0 0x55a601a5b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a6019e0b50 0x55a601a58d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb023e00640 0 client.25186 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 --2- 172.21.15.7:0/1950776051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a6019e0160 0x55a6019ef7a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 -- 172.21.15.7:0/1950776051 >> 172.21.15.7:0/1950776051 conn(0x55a6019db6d0 msgr2=0x55a6019ea250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 -- 172.21.15.7:0/1950776051 shutdown_connections 2024-03-18T01:37:14.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 1 -- 172.21.15.7:0/1950776051 wait complete. 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.717+0000 7fb04ba6d740 20 client.25186 trim_cache size 0 max 16384 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 3, 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24542, 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 105.365708317, 2024-03-18T01:37:14.723 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:14.724 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:14.724 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:14.724 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 158.928592708 2024-03-18T01:37:14.724 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:14.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 -- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb0280a2790 msgr2=0x7fb0280a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb0280a2790 0x7fb0280a4c50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb034009830 tx=0x7fb03402e040 comp rx=0 tx=0).stop 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 -- 172.21.15.7:0/892423798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb044075820 msgr2=0x7fb04419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb044075820 0x7fb04419e6a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb03000e9c0 tx=0x7fb03000ee80 comp rx=0 tx=0).stop 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 -- 172.21.15.7:0/892423798 shutdown_connections 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb0280a2790 0x7fb0280a4c50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb04410dde0 0x7fb0441a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb044075820 0x7fb04419e6a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 --2- 172.21.15.7:0/892423798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb044074240 0x7fb04419e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 -- 172.21.15.7:0/892423798 >> 172.21.15.7:0/892423798 conn(0x7fb0440fcda0 msgr2=0x7fb044101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 -- 172.21.15.7:0/892423798 shutdown_connections 2024-03-18T01:37:14.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.718+0000 7fb048e00640 1 -- 172.21.15.7:0/892423798 wait complete. 2024-03-18T01:37:14.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T01:37:14.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:14 smithi046 ceph-mon[36108]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:14.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:14 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2056442037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:14.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:14 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3191877998' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:14.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 -- 172.21.15.7:0/3010168662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4750103820 msgr2=0x7f4750103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 --2- 172.21.15.7:0/3010168662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4750103820 0x7f4750103c40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4740009900 tx=0x7f474002bc50 comp rx=0 tx=0).stop 2024-03-18T01:37:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 -- 172.21.15.7:0/3010168662 shutdown_connections 2024-03-18T01:37:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 --2- 172.21.15.7:0/3010168662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f47501053d0 0x7f475010bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 --2- 172.21.15.7:0/3010168662 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4750104a10 0x7f4750104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 --2- 172.21.15.7:0/3010168662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4750103820 0x7f4750103c40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 -- 172.21.15.7:0/3010168662 >> 172.21.15.7:0/3010168662 conn(0x7f47500fedb0 msgr2=0x7f47501011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 -- 172.21.15.7:0/3010168662 shutdown_connections 2024-03-18T01:37:14.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.951+0000 7f474fe00640 1 -- 172.21.15.7:0/3010168662 wait complete. 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.952+0000 7f474fe00640 1 Processor -- start 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.952+0000 7f474fe00640 1 -- start start 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4750103820 0x7f4750199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4750104a10 0x7f475019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47501053d0 0x7f47501a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474fe00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f475010f4c0 con 0x7f4750104a10 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474fe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f475010f340 con 0x7f4750103820 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f475010f640 con 0x7f47501053d0 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4750103820 0x7f4750199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4750103820 0x7f4750199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53974/0 (socket says 172.21.15.7:53974) 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474ea00640 1 -- 172.21.15.7:0/4239598823 learned_addr learned my addr 172.21.15.7:0/4239598823 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474f400640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47501053d0 0x7f47501a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474e000640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4750104a10 0x7f475019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474ea00640 1 -- 172.21.15.7:0/4239598823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47501053d0 msgr2=0x7f47501a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474ea00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47501053d0 0x7f47501a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474ea00640 1 -- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4750104a10 msgr2=0x7f475019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.953+0000 7f474ea00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4750104a10 0x7f475019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.954+0000 7f474ea00640 1 -- 172.21.15.7:0/4239598823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47501a1760 con 0x7f4750103820 2024-03-18T01:37:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.954+0000 7f474ea00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4750103820 0x7f4750199b40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4740005ab0 tx=0x7f474002c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.954+0000 7f4747400640 1 -- 172.21.15.7:0/4239598823 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740041070 con 0x7f4750103820 2024-03-18T01:37:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.954+0000 7f4747400640 1 -- 172.21.15.7:0/4239598823 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4740004450 con 0x7f4750103820 2024-03-18T01:37:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.954+0000 7f474fe00640 1 -- 172.21.15.7:0/4239598823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47501a19f0 con 0x7f4750103820 2024-03-18T01:37:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.954+0000 7f4747400640 1 -- 172.21.15.7:0/4239598823 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740034440 con 0x7f4750103820 2024-03-18T01:37:14.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.954+0000 7f474fe00640 1 -- 172.21.15.7:0/4239598823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47501a1f40 con 0x7f4750103820 2024-03-18T01:37:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.957+0000 7f475882b740 1 Processor -- start 2024-03-18T01:37:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.957+0000 7f4747400640 1 -- 172.21.15.7:0/4239598823 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f4740044070 con 0x7f4750103820 2024-03-18T01:37:14.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f475882b740 1 -- start start 2024-03-18T01:37:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f475882b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1b3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f475882b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1b8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f475882b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f475882b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f117c50 con 0x55bd0f1b3340 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f475882b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f0c09d0 con 0x55bd0f1b3d30 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f475882b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f113cf0 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f474ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f474f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f474f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53608/0 (socket says 172.21.15.7:53608) 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f474e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.958+0000 7f474f400640 1 -- 172.21.15.7:0/1161851058 learned_addr learned my addr 172.21.15.7:0/1161851058 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f474ea00640 1 -- 172.21.15.7:0/1161851058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 msgr2=0x55bd0f1bb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f474ea00640 1 --2- 172.21.15.7:0/1161851058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f474ea00640 1 -- 172.21.15.7:0/1161851058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 msgr2=0x55bd0f1b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f474ea00640 1 --2- 172.21.15.7:0/1161851058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1b3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f474ea00640 1 -- 172.21.15.7:0/1161851058 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd0f0daf10 con 0x55bd0f1b3d30 2024-03-18T01:37:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f474ea00640 1 --2- 172.21.15.7:0/1161851058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1b8830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f47400ee090 tx=0x7f47400f6b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8555d1d3753035f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f4747400640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f47340a2800 0x7f47340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f474f400640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f47340a2800 0x7f47340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.959+0000 7f473be00640 1 -- 172.21.15.7:0/1161851058 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47400ff070 con 0x55bd0f1b3d30 2024-03-18T01:37:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.960+0000 7f473be00640 1 -- 172.21.15.7:0/1161851058 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47400c5730 con 0x55bd0f1b3d30 2024-03-18T01:37:14.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.960+0000 7f473be00640 1 -- 172.21.15.7:0/1161851058 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47400bc9a0 con 0x55bd0f1b3d30 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.960+0000 7f4747400640 1 -- 172.21.15.7:0/4239598823 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f47400f09b0 con 0x7f4750103820 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.960+0000 7f474f400640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f47340a2800 0x7f47340a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x55bd0f1b4590 tx=0x7f4748009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.960+0000 7f475882b740 1 -- 172.21.15.7:0/1161851058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 msgr2=0x55bd0f1b8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.960+0000 7f475882b740 1 --2- 172.21.15.7:0/1161851058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1b8830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f47400ee090 tx=0x7f47400f6b20 comp rx=0 tx=0).stop 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.961+0000 7f475882b740 1 -- 172.21.15.7:0/1161851058 shutdown_connections 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.961+0000 7f475882b740 1 --2- 172.21.15.7:0/1161851058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.961+0000 7f475882b740 1 --2- 172.21.15.7:0/1161851058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1b8830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.961+0000 7f475882b740 1 --2- 172.21.15.7:0/1161851058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1b3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.961+0000 7f475882b740 1 -- 172.21.15.7:0/1161851058 >> 172.21.15.7:0/1161851058 conn(0x55bd0f1ae8b0 msgr2=0x55bd0f1aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.961+0000 7f475882b740 1 -- 172.21.15.7:0/1161851058 shutdown_connections 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.961+0000 7f475882b740 1 -- 172.21.15.7:0/1161851058 wait complete. 2024-03-18T01:37:14.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.961+0000 7f475882b740 1 -- start start 2024-03-18T01:37:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.962+0000 7f475882b740 1 Processor -- start 2024-03-18T01:37:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.962+0000 7f475882b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1bd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.962+0000 7f475882b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1bd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.962+0000 7f475882b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.962+0000 7f475882b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f1bbc10 con 0x55bd0f1b3340 2024-03-18T01:37:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.962+0000 7f475882b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f0d3e20 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.962+0000 7f475882b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f1bbd90 con 0x55bd0f1b3d30 2024-03-18T01:37:14.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.962+0000 7f475882b740 10 client.0 _ll_register_callbacks cb 0x55bd0f0d37f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1bd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53990/0 (socket says 172.21.15.7:53990) 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 -- 172.21.15.7:0/1623313546 learned_addr learned my addr 172.21.15.7:0/1623313546 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474ea00640 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1bd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 -- 172.21.15.7:0/1623313546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 msgr2=0x55bd0f1bd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1bd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 -- 172.21.15.7:0/1623313546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 msgr2=0x55bd0f1bd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1bd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 -- 172.21.15.7:0/1623313546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd0ee2c2d0 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474e000640 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1bd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474ea00640 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1bd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:14.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f474f400640 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c4700 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f474800f700 tx=0x7f4748004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbeb8a30d9cfd31d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.963+0000 7f4738c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.964+0000 7f4738c00640 1 -- 172.21.15.7:0/1623313546 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4748003970 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.964+0000 7f4738c00640 1 -- 172.21.15.7:0/1623313546 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4748017030 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.964+0000 7f475882b740 1 -- 172.21.15.7:0/1623313546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47400c5730 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.964+0000 7f4738c00640 1 -- 172.21.15.7:0/1623313546 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4748032050 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.964+0000 7f475882b740 10 client.25196 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.964+0000 7f475882b740 1 -- 172.21.15.7:0/1623313546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bd0eef66b0 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.964+0000 7f4738c00640 1 -- 172.21.15.7:0/1623313546 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47400c5730 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.965+0000 7f4738c00640 1 -- 172.21.15.7:0/1623313546 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f47480042b0 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.965+0000 7f475882b740 10 client.25196 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.965+0000 7f475882b740 10 client.25196 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.965+0000 7f475882b740 20 client.25196 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.965+0000 7f475882b740 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55bd0f1c9fd0 0x55bd0f1cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.965+0000 7f475882b740 4 client.25196 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:14.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.965+0000 7f475882b740 1 -- 172.21.15.7:0/1623313546 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd0f1752b0 con 0x55bd0f1c9fd0 2024-03-18T01:37:14.968 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.965+0000 7f474ea00640 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55bd0f1c9fd0 0x55bd0f1cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.966+0000 7f474ea00640 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55bd0f1c9fd0 0x55bd0f1cb440 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b42c055b1b79e34e server_cookie=a9aa5d26b8ec5bda in_seq=0 out_seq=0 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.966+0000 7f4738c00640 10 client.25196 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.966+0000 7f4738c00640 1 -- 172.21.15.7:0/1623313546 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd0f1752b0 con 0x55bd0f1c9fd0 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.966+0000 7f4738c00640 10 client.25196 handle_command_reply: tid=0 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.966+0000 7f475882b740 1 client.25196 shutdown 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 -- 172.21.15.7:0/1623313546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 msgr2=0x55bd0f1c4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c4700 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f474800f700 tx=0x7f4748004ab0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 -- 172.21.15.7:0/1623313546 shutdown_connections 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c4700 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55bd0f1c9fd0 0x55bd0f1cb440 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1bd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 --2- 172.21.15.7:0/1623313546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1bd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 -- 172.21.15.7:0/1623313546 >> 172.21.15.7:0/1623313546 conn(0x55bd0f1ae8b0 msgr2=0x55bd0f1ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f4738c00640 0 client.25196 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:37:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 -- 172.21.15.7:0/1623313546 shutdown_connections 2024-03-18T01:37:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 1 -- 172.21.15.7:0/1623313546 wait complete. 2024-03-18T01:37:14.971 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.967+0000 7f475882b740 20 client.25196 trim_cache size 0 max 16384 2024-03-18T01:37:14.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.987+0000 7f475882b740 1 Processor -- start 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.987+0000 7f475882b740 1 -- start start 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.987+0000 7f475882b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1a9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.987+0000 7f475882b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.987+0000 7f475882b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.987+0000 7f475882b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f1c9780 con 0x55bd0f1b3340 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.987+0000 7f475882b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f1c9e50 con 0x55bd0f1b3d30 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.987+0000 7f475882b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f1ca520 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474ea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:53998/0 (socket says 172.21.15.7:53998) 2024-03-18T01:37:14.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39664/0 (socket says 172.21.15.7:39664) 2024-03-18T01:37:14.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474e000640 1 -- 172.21.15.7:0/1064577815 learned_addr learned my addr 172.21.15.7:0/1064577815 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474ea00640 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1a9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53622/0 (socket says 172.21.15.7:53622) 2024-03-18T01:37:14.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474f400640 1 -- 172.21.15.7:0/1064577815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 msgr2=0x55bd0f1c46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474f400640 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474f400640 1 -- 172.21.15.7:0/1064577815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 msgr2=0x55bd0f1a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474f400640 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474f400640 1 -- 172.21.15.7:0/1064577815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd0f1af620 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474ea00640 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.988+0000 7f474f400640 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bd180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4748018040 tx=0x7f4748005c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea8a807c2bcc5a8f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f473b400640 1 -- 172.21.15.7:0/1064577815 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474801f230 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f473b400640 1 -- 172.21.15.7:0/1064577815 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f474801f3d0 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f473b400640 1 -- 172.21.15.7:0/1064577815 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474801e420 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f475882b740 1 -- 172.21.15.7:0/1064577815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 msgr2=0x55bd0f1bd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f475882b740 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bd180 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4748018040 tx=0x7f4748005c10 comp rx=0 tx=0).stop 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f475882b740 1 -- 172.21.15.7:0/1064577815 shutdown_connections 2024-03-18T01:37:14.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f475882b740 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1bd180 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f475882b740 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f475882b740 1 --2- 172.21.15.7:0/1064577815 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.989+0000 7f475882b740 1 -- 172.21.15.7:0/1064577815 >> 172.21.15.7:0/1064577815 conn(0x55bd0f1ae8b0 msgr2=0x55bd0f1c8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.990+0000 7f475882b740 1 -- 172.21.15.7:0/1064577815 shutdown_connections 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.990+0000 7f475882b740 1 -- 172.21.15.7:0/1064577815 wait complete. 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.990+0000 7f475882b740 1 -- start start 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.990+0000 7f475882b740 1 Processor -- start 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.990+0000 7f475882b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1c00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.990+0000 7f475882b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.990+0000 7f475882b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f475882b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f1cb2c0 con 0x55bd0f1b3340 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f475882b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f1cabf0 con 0x55bd0f1b8ea0 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f475882b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bd0f1cb990 con 0x55bd0f1b3d30 2024-03-18T01:37:14.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f475882b740 10 client.0 _ll_register_callbacks cb 0x7f474800f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39676/0 (socket says 172.21.15.7:39676) 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474e000640 1 -- 172.21.15.7:0/2252908009 learned_addr learned my addr 172.21.15.7:0/2252908009 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474ea00640 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1c00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474e000640 1 -- 172.21.15.7:0/2252908009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 msgr2=0x55bd0f1c3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474e000640 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474e000640 1 -- 172.21.15.7:0/2252908009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 msgr2=0x55bd0f1c00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474e000640 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1c00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474e000640 1 -- 172.21.15.7:0/2252908009 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd0f1c3bf0 con 0x55bd0f1b3d30 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.991+0000 7f474f400640 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f474e000640 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c05e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f473c0038a0 tx=0x7f473c003d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46bfd7579ea754f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:14.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f4738c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f4738c00640 1 -- 172.21.15.7:0/2252908009 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473c012b50 con 0x55bd0f1b3d30 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f475882b740 1 -- 172.21.15.7:0/2252908009 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f474801f3d0 con 0x55bd0f1b3d30 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f4738c00640 1 -- 172.21.15.7:0/2252908009 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f473c012cf0 con 0x55bd0f1b3d30 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f4738c00640 1 -- 172.21.15.7:0/2252908009 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f473c02b050 con 0x55bd0f1b3d30 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f475882b740 10 client.25204 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f475882b740 1 -- 172.21.15.7:0/2252908009 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bd0f230100 con 0x55bd0f1b3d30 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f4738c00640 1 -- 172.21.15.7:0/2252908009 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f474801f3d0 con 0x55bd0f1b3d30 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.992+0000 7f4738c00640 1 -- 172.21.15.7:0/2252908009 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f473c010430 con 0x55bd0f1b3d30 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f475882b740 10 client.25204 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f475882b740 10 client.25204 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f475882b740 20 client.25204 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f475882b740 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55bd0f234d40 0x55bd0f233a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f475882b740 4 client.25204 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:14.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f475882b740 1 -- 172.21.15.7:0/2252908009 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "status"}) -- 0x55bd0f125190 con 0x55bd0f234d40 2024-03-18T01:37:14.997 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f474ea00640 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55bd0f234d40 0x55bd0f233a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f474ea00640 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55bd0f234d40 0x55bd0f233a50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a12ba90dee2dfc8 server_cookie=d196438e7a04da2a in_seq=0 out_seq=0 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f4738c00640 10 client.25204 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.993+0000 7f4738c00640 1 -- 172.21.15.7:0/2252908009 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55bd0f125190 con 0x55bd0f234d40 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f4738c00640 10 client.25204 handle_command_reply: tid=0 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 client.25204 shutdown 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 -- 172.21.15.7:0/2252908009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 msgr2=0x55bd0f1c05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c05e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f473c0038a0 tx=0x7f473c003d60 comp rx=0 tx=0).stop 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 -- 172.21.15.7:0/2252908009 shutdown_connections 2024-03-18T01:37:14.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bd0f1b8ea0 0x55bd0f1c3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55bd0f234d40 0x55bd0f233a50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bd0f1b3d30 0x55bd0f1c05e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 --2- 172.21.15.7:0/2252908009 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bd0f1b3340 0x55bd0f1c00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 -- 172.21.15.7:0/2252908009 >> 172.21.15.7:0/2252908009 conn(0x55bd0f1ae8b0 msgr2=0x55bd0f1a9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f4738c00640 0 client.25204 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 -- 172.21.15.7:0/2252908009 shutdown_connections 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 1 -- 172.21.15.7:0/2252908009 wait complete. 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.994+0000 7f475882b740 20 client.25204 trim_cache size 0 max 16384 2024-03-18T01:37:14.999 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 2, 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24544, 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 106.65358029399999, 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 156.40486293500001 2024-03-18T01:37:15.000 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:15.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 -- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f47340a2800 msgr2=0x7f47340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f47340a2800 0x7f47340a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x55bd0f1b4590 tx=0x7f4748009500 comp rx=0 tx=0).stop 2024-03-18T01:37:15.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 -- 172.21.15.7:0/4239598823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4750103820 msgr2=0x7f4750199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4750103820 0x7f4750199b40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4740005ab0 tx=0x7f474002c6e0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 -- 172.21.15.7:0/4239598823 shutdown_connections 2024-03-18T01:37:15.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47501053d0 0x7f47501a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f47340a2800 0x7f47340a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4750104a10 0x7f475019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 --2- 172.21.15.7:0/4239598823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4750103820 0x7f4750199b40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 -- 172.21.15.7:0/4239598823 >> 172.21.15.7:0/4239598823 conn(0x7f47500fedb0 msgr2=0x7f4750107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 -- 172.21.15.7:0/4239598823 shutdown_connections 2024-03-18T01:37:15.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:14.995+0000 7f474fe00640 1 -- 172.21.15.7:0/4239598823 wait complete. 2024-03-18T01:37:15.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-03-18T01:37:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 -- 172.21.15.7:0/1402279457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc2c103840 msgr2=0x7fbc2c103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 --2- 172.21.15.7:0/1402279457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc2c103840 0x7fbc2c103c60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009900 tx=0x7fbc2002bc30 comp rx=0 tx=0).stop 2024-03-18T01:37:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 -- 172.21.15.7:0/1402279457 shutdown_connections 2024-03-18T01:37:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 --2- 172.21.15.7:0/1402279457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc2c1053f0 0x7fbc2c10bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 --2- 172.21.15.7:0/1402279457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc2c104a30 0x7fbc2c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 --2- 172.21.15.7:0/1402279457 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc2c103840 0x7fbc2c103c60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 -- 172.21.15.7:0/1402279457 >> 172.21.15.7:0/1402279457 conn(0x7fbc2c0fedb0 msgr2=0x7fbc2c1011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 -- 172.21.15.7:0/1402279457 shutdown_connections 2024-03-18T01:37:15.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.245+0000 7fbc32200640 1 -- 172.21.15.7:0/1402279457 wait complete. 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.247+0000 7fbc32200640 1 Processor -- start 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.247+0000 7fbc32200640 1 -- start start 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.247+0000 7fbc32200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc2c103840 0x7fbc2c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.247+0000 7fbc32200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc2c104a30 0x7fbc2c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.247+0000 7fbc32200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc2c1053f0 0x7fbc2c1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.247+0000 7fbc32200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c10f4e0 con 0x7fbc2c104a30 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.247+0000 7fbc32200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c10f360 con 0x7fbc2c1053f0 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.247+0000 7fbc32200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c10f660 con 0x7fbc2c103840 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc2c103840 0x7fbc2c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc31800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc2c1053f0 0x7fbc2c1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc2c103840 0x7fbc2c199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39694/0 (socket says 172.21.15.7:39694) 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 -- 172.21.15.7:0/340066758 learned_addr learned my addr 172.21.15.7:0/340066758 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc2be00640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc2c104a30 0x7fbc2c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 -- 172.21.15.7:0/340066758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc2c1053f0 msgr2=0x7fbc2c1a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc2c1053f0 0x7fbc2c1a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 -- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc2c104a30 msgr2=0x7fbc2c19a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc2c104a30 0x7fbc2c19a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 -- 172.21.15.7:0/340066758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c1a17a0 con 0x7fbc2c103840 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc31800640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc2c1053f0 0x7fbc2c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.248+0000 7fbc30e00640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc2c103840 0x7fbc2c199b80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbc20000c00 tx=0x7fbc2002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.249+0000 7fbc29600640 1 -- 172.21.15.7:0/340066758 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20043020 con 0x7fbc2c103840 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.249+0000 7fbc29600640 1 -- 172.21.15.7:0/340066758 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc200054e0 con 0x7fbc2c103840 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.249+0000 7fbc32200640 1 -- 172.21.15.7:0/340066758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2c1a1a30 con 0x7fbc2c103840 2024-03-18T01:37:15.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.249+0000 7fbc29600640 1 -- 172.21.15.7:0/340066758 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20003bd0 con 0x7fbc2c103840 2024-03-18T01:37:15.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.249+0000 7fbc32200640 1 -- 172.21.15.7:0/340066758 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2c1a1fd0 con 0x7fbc2c103840 2024-03-18T01:37:15.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.252+0000 7fbc36595740 1 Processor -- start 2024-03-18T01:37:15.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.252+0000 7fbc29600640 1 -- 172.21.15.7:0/340066758 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fbc20046020 con 0x7fbc2c103840 2024-03-18T01:37:15.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.252+0000 7fbc36595740 1 -- start start 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.252+0000 7fbc36595740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4ea760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc36595740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ead30 0x55d00d4ef830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc36595740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4f2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc36595740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d00d44ec50 con 0x55d00d4efea0 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc36595740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d00d3f79d0 con 0x55d00d4ea340 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc36595740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d00d44acf0 con 0x55d00d4ead30 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc30e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ead30 0x55d00d4ef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc31800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc31800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4ea760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54048/0 (socket says 172.21.15.7:54048) 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc31800640 1 -- 172.21.15.7:0/662909300 learned_addr learned my addr 172.21.15.7:0/662909300 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc2be00640 1 --2- 172.21.15.7:0/662909300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc31800640 1 -- 172.21.15.7:0/662909300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ead30 msgr2=0x55d00d4ef830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc31800640 1 --2- 172.21.15.7:0/662909300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ead30 0x55d00d4ef830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc31800640 1 -- 172.21.15.7:0/662909300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 msgr2=0x55d00d4f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc31800640 1 --2- 172.21.15.7:0/662909300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.253+0000 7fbc31800640 1 -- 172.21.15.7:0/662909300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d00d411f10 con 0x55d00d4ea340 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.254+0000 7fbc31800640 1 --2- 172.21.15.7:0/662909300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4ea760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc24011810 tx=0x7fbc24011cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6204e72cd7b84526 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.254+0000 7fbc29600640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbc140a2880 0x7fbc140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.254+0000 7fbc1e000640 1 -- 172.21.15.7:0/662909300 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24011310 con 0x55d00d4ea340 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.254+0000 7fbc1e000640 1 -- 172.21.15.7:0/662909300 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc2400f9a0 con 0x55d00d4ea340 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.254+0000 7fbc1e000640 1 -- 172.21.15.7:0/662909300 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc240056d0 con 0x55d00d4ea340 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.254+0000 7fbc30e00640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbc140a2880 0x7fbc140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.254+0000 7fbc29600640 1 -- 172.21.15.7:0/340066758 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbc200f0da0 con 0x7fbc2c103840 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 -- 172.21.15.7:0/662909300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 msgr2=0x55d00d4ea760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 --2- 172.21.15.7:0/662909300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4ea760 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc24011810 tx=0x7fbc24011cd0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 -- 172.21.15.7:0/662909300 shutdown_connections 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 --2- 172.21.15.7:0/662909300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 --2- 172.21.15.7:0/662909300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ead30 0x55d00d4ef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 --2- 172.21.15.7:0/662909300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4ea760 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 -- 172.21.15.7:0/662909300 >> 172.21.15.7:0/662909300 conn(0x55d00d4e58b0 msgr2=0x55d00d4e5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc30e00640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbc140a2880 0x7fbc140a4d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbc20006010 tx=0x7fbc200dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 -- 172.21.15.7:0/662909300 shutdown_connections 2024-03-18T01:37:15.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 -- 172.21.15.7:0/662909300 wait complete. 2024-03-18T01:37:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.255+0000 7fbc36595740 1 -- start start 2024-03-18T01:37:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc36595740 1 Processor -- start 2024-03-18T01:37:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc36595740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc36595740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc36595740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4efea0 0x55d00d4fba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc36595740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d00d4f2e50 con 0x55d00d4ead30 2024-03-18T01:37:15.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc36595740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d00d40ae20 con 0x55d00d4efea0 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc36595740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d00d4f2fd0 con 0x55d00d4ea340 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc36595740 10 client.0 _ll_register_callbacks cb 0x55d00d40a7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.256+0000 7fbc31800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc31800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39726/0 (socket says 172.21.15.7:39726) 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc31800640 1 -- 172.21.15.7:0/4130355556 learned_addr learned my addr 172.21.15.7:0/4130355556 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc31800640 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4efea0 0x55d00d4fba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc2be00640 1 -- 172.21.15.7:0/4130355556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 msgr2=0x55d00d4f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc2be00640 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc2be00640 1 -- 172.21.15.7:0/4130355556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4efea0 msgr2=0x55d00d4fba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc2be00640 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4efea0 0x55d00d4fba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc2be00640 1 -- 172.21.15.7:0/4130355556 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d00d1632d0 con 0x55d00d4ead30 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc31800640 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc2be00640 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4f4a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c19b090 tx=0x7fbc18012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef2773fa51eb77f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.257+0000 7fbc12a00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.258+0000 7fbc12a00640 1 -- 172.21.15.7:0/4130355556 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1800f8c0 con 0x55d00d4ead30 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.258+0000 7fbc12a00640 1 -- 172.21.15.7:0/4130355556 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbc1800fa60 con 0x55d00d4ead30 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.258+0000 7fbc36595740 1 -- 172.21.15.7:0/4130355556 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc2400f9a0 con 0x55d00d4ead30 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.258+0000 7fbc12a00640 1 -- 172.21.15.7:0/4130355556 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbc1800fc60 con 0x55d00d4ead30 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.258+0000 7fbc36595740 10 client.15072 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.258+0000 7fbc36595740 1 -- 172.21.15.7:0/4130355556 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d00d22d6b0 con 0x55d00d4ead30 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.258+0000 7fbc12a00640 1 -- 172.21.15.7:0/4130355556 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc2400f9a0 con 0x55d00d4ead30 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.258+0000 7fbc12a00640 1 -- 172.21.15.7:0/4130355556 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbc1801c420 con 0x55d00d4ead30 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.259+0000 7fbc36595740 10 client.15072 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.259+0000 7fbc36595740 10 client.15072 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.259+0000 7fbc36595740 20 client.15072 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.259+0000 7fbc36595740 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d00d501380 0x55d00d5027f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.259+0000 7fbc36595740 4 client.15072 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.259+0000 7fbc36595740 1 -- 172.21.15.7:0/4130355556 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d00d4ac2b0 con 0x55d00d501380 2024-03-18T01:37:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.259+0000 7fbc31800640 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d00d501380 0x55d00d5027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.260+0000 7fbc31800640 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d00d501380 0x55d00d5027f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ece5a157fc17927 server_cookie=51f070d939414605 in_seq=0 out_seq=0 2024-03-18T01:37:15.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.260+0000 7fbc12a00640 10 client.15072 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc12a00640 1 -- 172.21.15.7:0/4130355556 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d00d4ac2b0 con 0x55d00d501380 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc12a00640 10 client.15072 handle_command_reply: tid=0 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 client.15072 shutdown 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 -- 172.21.15.7:0/4130355556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 msgr2=0x55d00d4f4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4f4a00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c19b090 tx=0x7fbc18012b60 comp rx=0 tx=0).stop 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 -- 172.21.15.7:0/4130355556 shutdown_connections 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d00d501380 0x55d00d5027f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4efea0 0x55d00d4fba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4f4a00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 --2- 172.21.15.7:0/4130355556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 -- 172.21.15.7:0/4130355556 >> 172.21.15.7:0/4130355556 conn(0x55d00d4e58b0 msgr2=0x55d00d4e4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc12a00640 0 client.15072 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:37:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 -- 172.21.15.7:0/4130355556 shutdown_connections 2024-03-18T01:37:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 1 -- 172.21.15.7:0/4130355556 wait complete. 2024-03-18T01:37:15.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.261+0000 7fbc36595740 20 client.15072 trim_cache size 0 max 16384 2024-03-18T01:37:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc36595740 1 Processor -- start 2024-03-18T01:37:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc36595740 1 -- start start 2024-03-18T01:37:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc36595740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4e48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc36595740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4fba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc36595740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4efea0 0x55d00d4eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc36595740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d00d500b30 con 0x55d00d4ead30 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc36595740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d00d501200 con 0x55d00d4ea340 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc36595740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d00d5018d0 con 0x55d00d4efea0 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4e48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4e48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54064/0 (socket says 172.21.15.7:54064) 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 -- 172.21.15.7:0/868892598 learned_addr learned my addr 172.21.15.7:0/868892598 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 --2- 172.21.15.7:0/868892598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4efea0 0x55d00d4eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 -- 172.21.15.7:0/868892598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4efea0 msgr2=0x55d00d4eb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 --2- 172.21.15.7:0/868892598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4efea0 0x55d00d4eb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 -- 172.21.15.7:0/868892598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 msgr2=0x55d00d4fba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 --2- 172.21.15.7:0/868892598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4fba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.282+0000 7fbc31800640 1 -- 172.21.15.7:0/868892598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d00d4dfec0 con 0x55d00d4ea340 2024-03-18T01:37:15.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.283+0000 7fbc31800640 1 --2- 172.21.15.7:0/868892598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4e48f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55d00d502f30 tx=0x7fbc24011cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4115e18302f269a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.283+0000 7fbc1d600640 1 -- 172.21.15.7:0/868892598 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2401e040 con 0x55d00d4ea340 2024-03-18T01:37:15.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.283+0000 7fbc1d600640 1 -- 172.21.15.7:0/868892598 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc24003e50 con 0x55d00d4ea340 2024-03-18T01:37:15.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.283+0000 7fbc1d600640 1 -- 172.21.15.7:0/868892598 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc240057c0 con 0x55d00d4ea340 2024-03-18T01:37:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 -- 172.21.15.7:0/868892598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 msgr2=0x55d00d4e48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 --2- 172.21.15.7:0/868892598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4e48f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x55d00d502f30 tx=0x7fbc24011cd0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 -- 172.21.15.7:0/868892598 shutdown_connections 2024-03-18T01:37:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 --2- 172.21.15.7:0/868892598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4efea0 0x55d00d4eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 --2- 172.21.15.7:0/868892598 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4ead30 0x55d00d4fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 --2- 172.21.15.7:0/868892598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ea340 0x55d00d4e48f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 -- 172.21.15.7:0/868892598 >> 172.21.15.7:0/868892598 conn(0x55d00d4e58b0 msgr2=0x55d00d4e41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 -- 172.21.15.7:0/868892598 shutdown_connections 2024-03-18T01:37:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 -- 172.21.15.7:0/868892598 wait complete. 2024-03-18T01:37:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 -- start start 2024-03-18T01:37:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.284+0000 7fbc36595740 1 Processor -- start 2024-03-18T01:37:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc36595740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc36595740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ead30 0x55d00d4f7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc36595740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4fa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc36595740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d00d502670 con 0x55d00d4efea0 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc36595740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d00d501fa0 con 0x55d00d4ead30 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc36595740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d00d502d40 con 0x55d00d4ea340 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc36595740 10 client.0 _ll_register_callbacks cb 0x7fbc18009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ead30 0x55d00d4f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc31800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc2be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ead30 0x55d00d4f7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54078/0 (socket says 172.21.15.7:54078) 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc2be00640 1 -- 172.21.15.7:0/280439153 learned_addr learned my addr 172.21.15.7:0/280439153 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc31800640 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4fa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc2be00640 1 -- 172.21.15.7:0/280439153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 msgr2=0x55d00d4f7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc2be00640 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc2be00640 1 -- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 msgr2=0x55d00d4fa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc2be00640 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4fa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc2be00640 1 -- 172.21.15.7:0/280439153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d00d4fac90 con 0x55d00d4ead30 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.285+0000 7fbc31800640 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4fa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc2be00640 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ead30 0x55d00d4f7740 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55d00d4e9fa0 tx=0x7fbc18009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e644642c8f0b78b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc12a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc12a00640 1 -- 172.21.15.7:0/280439153 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1802f060 con 0x55d00d4ead30 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc36595740 1 -- 172.21.15.7:0/280439153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc24003e50 con 0x55d00d4ead30 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc12a00640 1 -- 172.21.15.7:0/280439153 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbc1801cad0 con 0x55d00d4ead30 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc12a00640 1 -- 172.21.15.7:0/280439153 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbc1800faa0 con 0x55d00d4ead30 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc36595740 10 client.25220 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc36595740 1 -- 172.21.15.7:0/280439153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d00d567190 con 0x55d00d4ead30 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.286+0000 7fbc12a00640 1 -- 172.21.15.7:0/280439153 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc24003e50 con 0x55d00d4ead30 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.287+0000 7fbc12a00640 1 -- 172.21.15.7:0/280439153 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fbc18003860 con 0x55d00d4ead30 2024-03-18T01:37:15.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.287+0000 7fbc36595740 10 client.25220 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.287+0000 7fbc36595740 10 client.25220 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.287+0000 7fbc36595740 20 client.25220 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.287+0000 7fbc36595740 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d00d56bd80 0x55d00d56aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.287+0000 7fbc36595740 4 client.25220 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.287+0000 7fbc36595740 1 -- 172.21.15.7:0/280439153 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "status"}) -- 0x55d00d45c190 con 0x55d00d56bd80 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.287+0000 7fbc31800640 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d00d56bd80 0x55d00d56aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc31800640 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d00d56bd80 0x55d00d56aa90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1b43f31aa70b746d server_cookie=9071476bb84d0747 in_seq=0 out_seq=0 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc12a00640 10 client.25220 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc12a00640 1 -- 172.21.15.7:0/280439153 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d00d45c190 con 0x55d00d56bd80 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc12a00640 10 client.25220 handle_command_reply: tid=0 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 client.25220 shutdown 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 -- 172.21.15.7:0/280439153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ead30 msgr2=0x55d00d4f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ead30 0x55d00d4f7740 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55d00d4e9fa0 tx=0x7fbc18009460 comp rx=0 tx=0).stop 2024-03-18T01:37:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 -- 172.21.15.7:0/280439153 shutdown_connections 2024-03-18T01:37:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d00d56bd80 0x55d00d56aa90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d00d4efea0 0x55d00d4fa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d00d4ead30 0x55d00d4f7740 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 --2- 172.21.15.7:0/280439153 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d00d4ea340 0x55d00d4f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc12a00640 0 client.25220 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:37:15.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 -- 172.21.15.7:0/280439153 >> 172.21.15.7:0/280439153 conn(0x55d00d4e58b0 msgr2=0x55d00d4e4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 -- 172.21.15.7:0/280439153 shutdown_connections 2024-03-18T01:37:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 1 -- 172.21.15.7:0/280439153 wait complete. 2024-03-18T01:37:15.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.288+0000 7fbc36595740 20 client.25220 trim_cache size 0 max 16384 2024-03-18T01:37:15.298 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:15.298 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 1, 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24562, 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 107.961128629, 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 152.33078424300001 2024-03-18T01:37:15.299 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 -- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbc140a2880 msgr2=0x7fbc140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbc140a2880 0x7fbc140a4d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbc20006010 tx=0x7fbc200dd040 comp rx=0 tx=0).stop 2024-03-18T01:37:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 -- 172.21.15.7:0/340066758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc2c103840 msgr2=0x7fbc2c199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc2c103840 0x7fbc2c199b80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbc20000c00 tx=0x7fbc2002c650 comp rx=0 tx=0).stop 2024-03-18T01:37:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 -- 172.21.15.7:0/340066758 shutdown_connections 2024-03-18T01:37:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc2c1053f0 0x7fbc2c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fbc140a2880 0x7fbc140a4d40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fbc2c104a30 0x7fbc2c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 --2- 172.21.15.7:0/340066758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc2c103840 0x7fbc2c199b80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 -- 172.21.15.7:0/340066758 >> 172.21.15.7:0/340066758 conn(0x7fbc2c0fedb0 msgr2=0x7fbc2c107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 -- 172.21.15.7:0/340066758 shutdown_connections 2024-03-18T01:37:15.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.289+0000 7fbc32200640 1 -- 172.21.15.7:0/340066758 wait complete. 2024-03-18T01:37:15.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.516+0000 7f6ecc600640 1 -- 172.21.15.7:0/434030638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 msgr2=0x7f6ec40746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.516+0000 7f6ecc600640 1 --2- 172.21.15.7:0/434030638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 0x7f6ec40746c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009930 tx=0x7f6eb002bc20 comp rx=0 tx=0).stop 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.517+0000 7f6ecc600640 1 -- 172.21.15.7:0/434030638 shutdown_connections 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.517+0000 7f6ecc600640 1 --2- 172.21.15.7:0/434030638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec410ddb0 0x7f6ec41101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.517+0000 7f6ecc600640 1 --2- 172.21.15.7:0/434030638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 0x7f6ec40746c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.517+0000 7f6ecc600640 1 --2- 172.21.15.7:0/434030638 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ec4075820 0x7f6ec4073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.517+0000 7f6ecc600640 1 -- 172.21.15.7:0/434030638 >> 172.21.15.7:0/434030638 conn(0x7f6ec40fcd30 msgr2=0x7f6ec40ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.517+0000 7f6ecc600640 1 -- 172.21.15.7:0/434030638 shutdown_connections 2024-03-18T01:37:15.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.517+0000 7f6ecc600640 1 -- 172.21.15.7:0/434030638 wait complete. 2024-03-18T01:37:15.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.518+0000 7f6ecc600640 1 Processor -- start 2024-03-18T01:37:15.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.518+0000 7f6ecc600640 1 -- start start 2024-03-18T01:37:15.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ecc600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 0x7f6ec419e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ecc600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec4075820 0x7f6ec419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ecc600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ec410ddb0 0x7f6ec41a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ecc600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec4113a80 con 0x7f6ec410ddb0 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ecc600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec4113900 con 0x7f6ec4074240 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ecc600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec4113c00 con 0x7f6ec4075820 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 0x7f6ec419e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec4075820 0x7f6ec419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 0x7f6ec419e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54100/0 (socket says 172.21.15.7:54100) 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9c00640 1 -- 172.21.15.7:0/1052334251 learned_addr learned my addr 172.21.15.7:0/1052334251 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6eca600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ec410ddb0 0x7f6ec41a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9200640 1 -- 172.21.15.7:0/1052334251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 msgr2=0x7f6ec419e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9200640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 0x7f6ec419e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9200640 1 -- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ec410ddb0 msgr2=0x7f6ec41a5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9200640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ec410ddb0 0x7f6ec41a5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.519+0000 7f6ec9200640 1 -- 172.21.15.7:0/1052334251 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec41a5d90 con 0x7f6ec4075820 2024-03-18T01:37:15.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.520+0000 7f6eca600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ec410ddb0 0x7f6ec41a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:15.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.520+0000 7f6ec9200640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec4075820 0x7f6ec419e6b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6eb002c160 tx=0x7f6eb0004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.520+0000 7f6ec2000640 1 -- 172.21.15.7:0/1052334251 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0043020 con 0x7f6ec4075820 2024-03-18T01:37:15.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.520+0000 7f6ec2000640 1 -- 172.21.15.7:0/1052334251 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb0004440 con 0x7f6ec4075820 2024-03-18T01:37:15.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.520+0000 7f6ec2000640 1 -- 172.21.15.7:0/1052334251 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0003d40 con 0x7f6ec4075820 2024-03-18T01:37:15.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.520+0000 7f6ecc600640 1 -- 172.21.15.7:0/1052334251 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec41a6020 con 0x7f6ec4075820 2024-03-18T01:37:15.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.520+0000 7f6ecc600640 1 -- 172.21.15.7:0/1052334251 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec41a64f0 con 0x7f6ec4075820 2024-03-18T01:37:15.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.523+0000 7f6ec2000640 1 -- 172.21.15.7:0/1052334251 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f6eb0046020 con 0x7f6ec4075820 2024-03-18T01:37:15.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.523+0000 7f6ecf22f740 1 Processor -- start 2024-03-18T01:37:15.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ecf22f740 1 -- start start 2024-03-18T01:37:15.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37ab580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 0x5569c37b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5569c370fc50 con 0x5569c37b0cc0 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5569c36b89d0 con 0x5569c37ab160 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5569c370bcf0 con 0x5569c37abb50 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6eca600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39794/0 (socket says 172.21.15.7:39794) 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6ec9c00640 1 -- 172.21.15.7:0/1736206694 learned_addr learned my addr 172.21.15.7:0/1736206694 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.524+0000 7f6eca600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37ab580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54106/0 (socket says 172.21.15.7:54106) 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec9200640 1 --2- 172.21.15.7:0/1736206694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 0x5569c37b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec9c00640 1 -- 172.21.15.7:0/1736206694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 msgr2=0x5569c37ab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/1736206694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37ab580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec9c00640 1 -- 172.21.15.7:0/1736206694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 msgr2=0x5569c37b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/1736206694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 0x5569c37b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec9c00640 1 -- 172.21.15.7:0/1736206694 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c36d2f10 con 0x5569c37abb50 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec2000640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6eac0a2880 0x7f6eac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/1736206694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b0650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b890 tx=0x7f6eb800bd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5730ed83bdb09431 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6ec2000640 1 -- 172.21.15.7:0/1052334251 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6eb00f0f50 con 0x7f6ec4075820 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.525+0000 7f6eca600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6eac0a2880 0x7f6eac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6eb6a00640 1 -- 172.21.15.7:0/1736206694 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8003dd0 con 0x5569c37abb50 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6eb6a00640 1 -- 172.21.15.7:0/1736206694 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb8015070 con 0x5569c37abb50 2024-03-18T01:37:15.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6eb6a00640 1 -- 172.21.15.7:0/1736206694 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb80118a0 con 0x5569c37abb50 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 -- 172.21.15.7:0/1736206694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 msgr2=0x5569c37b0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6eca600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6eac0a2880 0x7f6eac0a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x5569c37ac3b0 tx=0x7f6ebc009780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/1736206694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b0650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b890 tx=0x7f6eb800bd50 comp rx=0 tx=0).stop 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 -- 172.21.15.7:0/1736206694 shutdown_connections 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/1736206694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 0x5569c37b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/1736206694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b0650 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/1736206694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 -- 172.21.15.7:0/1736206694 >> 172.21.15.7:0/1736206694 conn(0x5569c37a66d0 msgr2=0x5569c37a6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 -- 172.21.15.7:0/1736206694 shutdown_connections 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.526+0000 7f6ecf22f740 1 -- 172.21.15.7:0/1736206694 wait complete. 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.527+0000 7f6ecf22f740 1 -- start start 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.527+0000 7f6ecf22f740 1 Processor -- start 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37abb50 0x5569c37b5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37b0cc0 0x5569c37bc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5569c37b3c70 con 0x5569c37abb50 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5569c36cbe20 con 0x5569c37ab160 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5569c37b3df0 con 0x5569c37b0cc0 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ecf22f740 10 client.0 _ll_register_callbacks cb 0x5569c36cb7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ec9200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37abb50 0x5569c37b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6eca600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37b0cc0 0x5569c37bc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ec9200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37abb50 0x5569c37b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53736/0 (socket says 172.21.15.7:53736) 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ec9200640 1 -- 172.21.15.7:0/590685254 learned_addr learned my addr 172.21.15.7:0/590685254 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6eca600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37b0cc0 0x5569c37bc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39802/0 (socket says 172.21.15.7:39802) 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.528+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37b52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54122/0 (socket says 172.21.15.7:54122) 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ec9200640 1 -- 172.21.15.7:0/590685254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37b0cc0 msgr2=0x5569c37bc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ec9200640 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37b0cc0 0x5569c37bc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ec9200640 1 -- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 msgr2=0x5569c37b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ec9200640 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ec9200640 1 -- 172.21.15.7:0/590685254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c34242d0 con 0x5569c37abb50 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6eca600640 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37b0cc0 0x5569c37bc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:15.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ec9200640 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37abb50 0x5569c37b5800 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00b9ab0 tx=0x7f6eb0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6571622fd8066087 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6eab400640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:37:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6eab400640 1 -- 172.21.15.7:0/590685254 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb00bd190 con 0x5569c37abb50 2024-03-18T01:37:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6eab400640 1 -- 172.21.15.7:0/590685254 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6eb00ee7a0 con 0x5569c37abb50 2024-03-18T01:37:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ecf22f740 1 -- 172.21.15.7:0/590685254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb8015070 con 0x5569c37abb50 2024-03-18T01:37:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6eab400640 1 -- 172.21.15.7:0/590685254 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6eb00bcfb0 con 0x5569c37abb50 2024-03-18T01:37:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ecf22f740 10 client.15090 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:15.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.529+0000 7f6ecf22f740 1 -- 172.21.15.7:0/590685254 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5569c34ee6b0 con 0x5569c37abb50 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6eab400640 1 -- 172.21.15.7:0/590685254 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb8015070 con 0x5569c37abb50 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6eab400640 1 -- 172.21.15.7:0/590685254 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6eb00c59f0 con 0x5569c37abb50 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6ecf22f740 10 client.15090 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6ecf22f740 10 client.15090 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6ecf22f740 20 client.15090 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5569c37c2110 0x5569c37c3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6ecf22f740 4 client.15090 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6ecf22f740 1 -- 172.21.15.7:0/590685254 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569c376d2b0 con 0x5569c37c2110 2024-03-18T01:37:15.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.530+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5569c37c2110 0x5569c37c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.531+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5569c37c2110 0x5569c37c3580 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32f444e039db81f6 server_cookie=48ea6d6e89c47ed5 in_seq=0 out_seq=0 2024-03-18T01:37:15.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.531+0000 7f6eab400640 10 client.15090 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:37:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6eab400640 1 -- 172.21.15.7:0/590685254 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5569c376d2b0 con 0x5569c37c2110 2024-03-18T01:37:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6eab400640 10 client.15090 handle_command_reply: tid=0 2024-03-18T01:37:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6ecf22f740 1 client.15090 shutdown 2024-03-18T01:37:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6ecf22f740 1 -- 172.21.15.7:0/590685254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37abb50 msgr2=0x5569c37b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37abb50 0x5569c37b5800 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6eb00b9ab0 tx=0x7f6eb0009560 comp rx=0 tx=0).stop 2024-03-18T01:37:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6ecf22f740 1 -- 172.21.15.7:0/590685254 shutdown_connections 2024-03-18T01:37:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5569c37c2110 0x5569c37c3580 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37b0cc0 0x5569c37bc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37abb50 0x5569c37b5800 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.532+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/590685254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37ab160 0x5569c37b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.533+0000 7f6eab400640 0 client.15090 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:37:15.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.533+0000 7f6ecf22f740 1 -- 172.21.15.7:0/590685254 >> 172.21.15.7:0/590685254 conn(0x5569c37a66d0 msgr2=0x5569c37a5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.533+0000 7f6ecf22f740 1 -- 172.21.15.7:0/590685254 shutdown_connections 2024-03-18T01:37:15.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.533+0000 7f6ecf22f740 1 -- 172.21.15.7:0/590685254 wait complete. 2024-03-18T01:37:15.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.533+0000 7f6ecf22f740 20 client.15090 trim_cache size 0 max 16384 2024-03-18T01:37:15.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.560+0000 7f6ecf22f740 1 Processor -- start 2024-03-18T01:37:15.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.560+0000 7f6ecf22f740 1 -- start start 2024-03-18T01:37:15.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37ab160 0x5569c37b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37abb50 0x5569c37bc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 0x5569c37b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5569c37c18c0 con 0x5569c37b0cc0 2024-03-18T01:37:15.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5569c37c1f90 con 0x5569c37abb50 2024-03-18T01:37:15.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5569c37c2660 con 0x5569c37ab160 2024-03-18T01:37:15.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37ab160 0x5569c37b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ec9200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37abb50 0x5569c37bc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6eca600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 0x5569c37b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37ab160 0x5569c37b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:39818/0 (socket says 172.21.15.7:39818) 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ec9c00640 1 -- 172.21.15.7:0/757775035 learned_addr learned my addr 172.21.15.7:0/757775035 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ec9c00640 1 -- 172.21.15.7:0/757775035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37abb50 msgr2=0x5569c37bc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.561+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/757775035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37abb50 0x5569c37bc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.562+0000 7f6ec9c00640 1 -- 172.21.15.7:0/757775035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 msgr2=0x5569c37b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.562+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/757775035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 0x5569c37b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.562+0000 7f6ec9c00640 1 -- 172.21.15.7:0/757775035 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c37c1410 con 0x5569c37ab160 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.562+0000 7f6ec9200640 1 --2- 172.21.15.7:0/757775035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37abb50 0x5569c37bc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.562+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/757775035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37ab160 0x5569c37b45b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8006010 tx=0x7f6eb800b6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ef00c1307cb6ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.562+0000 7f6eb6000640 1 -- 172.21.15.7:0/757775035 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8005040 con 0x5569c37ab160 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.562+0000 7f6eb6000640 1 -- 172.21.15.7:0/757775035 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb8015070 con 0x5569c37ab160 2024-03-18T01:37:15.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.562+0000 7f6eb6000640 1 -- 172.21.15.7:0/757775035 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8004680 con 0x5569c37ab160 2024-03-18T01:37:15.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 -- 172.21.15.7:0/757775035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37ab160 msgr2=0x5569c37b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/757775035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37ab160 0x5569c37b45b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8006010 tx=0x7f6eb800b6c0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 -- 172.21.15.7:0/757775035 shutdown_connections 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/757775035 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37b0cc0 0x5569c37b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/757775035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37abb50 0x5569c37bc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/757775035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37ab160 0x5569c37b45b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 -- 172.21.15.7:0/757775035 >> 172.21.15.7:0/757775035 conn(0x5569c37a66d0 msgr2=0x5569c37a6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 -- 172.21.15.7:0/757775035 shutdown_connections 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 -- 172.21.15.7:0/757775035 wait complete. 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.563+0000 7f6ecf22f740 1 -- start start 2024-03-18T01:37:15.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.564+0000 7f6ecf22f740 1 Processor -- start 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.564+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37ab160 0x5569c37b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.564+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.564+0000 7f6ecf22f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37b0cc0 0x5569c37bb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.564+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5569c37c3400 con 0x5569c37ab160 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.564+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5569c37c2d30 con 0x5569c37b0cc0 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.564+0000 7f6ecf22f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5569c37c3ad0 con 0x5569c37abb50 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ecf22f740 10 client.0 _ll_register_callbacks cb 0x7f6eb00f44c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37ab160 0x5569c37b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6eca600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37b0cc0 0x5569c37bb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37ab160 0x5569c37b7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53752/0 (socket says 172.21.15.7:53752) 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 -- 172.21.15.7:0/2678962615 learned_addr learned my addr 172.21.15.7:0/2678962615 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9200640 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 -- 172.21.15.7:0/2678962615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 msgr2=0x5569c37b82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 -- 172.21.15.7:0/2678962615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37b0cc0 msgr2=0x5569c37bb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37b0cc0 0x5569c37bb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 -- 172.21.15.7:0/2678962615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c37bb8c0 con 0x5569c37ab160 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6ec9c00640 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37ab160 0x5569c37b7d70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b770 tx=0x7f6eb80041b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f121049b9c3454 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.565+0000 7f6eab400640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.566+0000 7f6eab400640 1 -- 172.21.15.7:0/2678962615 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8033070 con 0x5569c37ab160 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.566+0000 7f6eab400640 1 -- 172.21.15.7:0/2678962615 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6eb801a470 con 0x5569c37ab160 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.566+0000 7f6ecf22f740 1 -- 172.21.15.7:0/2678962615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb8015070 con 0x5569c37ab160 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.566+0000 7f6eab400640 1 -- 172.21.15.7:0/2678962615 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6eb803f050 con 0x5569c37ab160 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.566+0000 7f6ecf22f740 10 client.15096 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.566+0000 7f6ecf22f740 1 -- 172.21.15.7:0/2678962615 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5569c3827e80 con 0x5569c37ab160 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.566+0000 7f6eab400640 1 -- 172.21.15.7:0/2678962615 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb8015070 con 0x5569c37ab160 2024-03-18T01:37:15.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.567+0000 7f6eab400640 1 -- 172.21.15.7:0/2678962615 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6eb8010440 con 0x5569c37ab160 2024-03-18T01:37:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.567+0000 7f6ecf22f740 10 client.15096 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.567+0000 7f6ecf22f740 10 client.15096 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T01:37:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.567+0000 7f6ecf22f740 20 client.15096 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.567+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5569c382c1c0 0x5569c382aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.567+0000 7f6ecf22f740 4 client.15096 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.567+0000 7f6ecf22f740 1 -- 172.21.15.7:0/2678962615 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "status"}) -- 0x5569c371d190 con 0x5569c382c1c0 2024-03-18T01:37:15.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.567+0000 7f6ec9200640 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5569c382c1c0 0x5569c382aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ec9200640 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5569c382c1c0 0x5569c382aed0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2412c9c40aae46f5 server_cookie=72c24e0d1a1cf4b7 in_seq=0 out_seq=0 2024-03-18T01:37:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6eab400640 10 client.15096 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:37:15.571 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6eab400640 1 -- 172.21.15.7:0/2678962615 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5569c371d190 con 0x5569c382c1c0 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6eab400640 10 client.15096 handle_command_reply: tid=0 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 client.15096 shutdown 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 -- 172.21.15.7:0/2678962615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37ab160 msgr2=0x5569c37b7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37ab160 0x5569c37b7d70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800b770 tx=0x7f6eb80041b0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 -- 172.21.15.7:0/2678962615 shutdown_connections 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5569c382c1c0 0x5569c382aed0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5569c37b0cc0 0x5569c37bb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5569c37abb50 0x5569c37b82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6eab400640 0 client.15096 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 --2- 172.21.15.7:0/2678962615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5569c37ab160 0x5569c37b7d70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 -- 172.21.15.7:0/2678962615 >> 172.21.15.7:0/2678962615 conn(0x5569c37a66d0 msgr2=0x5569c37b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 -- 172.21.15.7:0/2678962615 shutdown_connections 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 1 -- 172.21.15.7:0/2678962615 wait complete. 2024-03-18T01:37:15.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.568+0000 7f6ecf22f740 20 client.15096 trim_cache size 0 max 16384 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24596, 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 138.51609440799999, 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:15.573 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:15.574 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:15.574 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 145.33232180799999 2024-03-18T01:37:15.574 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 -- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6eac0a2880 msgr2=0x7f6eac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6eac0a2880 0x7f6eac0a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x5569c37ac3b0 tx=0x7f6ebc009780 comp rx=0 tx=0).stop 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 -- 172.21.15.7:0/1052334251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec4075820 msgr2=0x7f6ec419e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec4075820 0x7f6ec419e6b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6eb002c160 tx=0x7f6eb0004040 comp rx=0 tx=0).stop 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 -- 172.21.15.7:0/1052334251 shutdown_connections 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6eac0a2880 0x7f6eac0a4d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6ec410ddb0 0x7f6ec41a5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ec4075820 0x7f6ec419e6b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 --2- 172.21.15.7:0/1052334251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec4074240 0x7f6ec419e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 -- 172.21.15.7:0/1052334251 >> 172.21.15.7:0/1052334251 conn(0x7f6ec40fcd30 msgr2=0x7f6ec4101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 -- 172.21.15.7:0/1052334251 shutdown_connections 2024-03-18T01:37:15.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:15.569+0000 7f6ecc600640 1 -- 172.21.15.7:0/1052334251 wait complete. 2024-03-18T01:37:15.592 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T01:37:15.592 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:37:15.601 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-03-18T01:37:15.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T01:37:16.051 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:16.529 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:16 smithi007 ceph-mon[31996]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:16 smithi203 ceph-mon[36345]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:16.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:16 smithi046 ceph-mon[36108]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:18.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.340+0000 7efcbec00700 1 -- 172.21.15.7:0/600987772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8104550 msgr2=0x7efcb8104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:18.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.340+0000 7efcbec00700 1 --2- 172.21.15.7:0/600987772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8104550 0x7efcb8104970 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efca8009990 tx=0x7efca8009ca0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.341+0000 7efcbec00700 1 -- 172.21.15.7:0/600987772 shutdown_connections 2024-03-18T01:37:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.341+0000 7efcbec00700 1 --2- 172.21.15.7:0/600987772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb81060f0 0x7efcb810c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.341+0000 7efcbec00700 1 --2- 172.21.15.7:0/600987772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8104550 0x7efcb8104970 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.341+0000 7efcbec00700 1 --2- 172.21.15.7:0/600987772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcb8105750 0x7efcb8105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.341+0000 7efcbec00700 1 -- 172.21.15.7:0/600987772 >> 172.21.15.7:0/600987772 conn(0x7efcb80ffaf0 msgr2=0x7efcb8101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.341+0000 7efcbec00700 1 -- 172.21.15.7:0/600987772 shutdown_connections 2024-03-18T01:37:18.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.341+0000 7efcbec00700 1 -- 172.21.15.7:0/600987772 wait complete. 2024-03-18T01:37:18.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.342+0000 7efcbec00700 1 Processor -- start 2024-03-18T01:37:18.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.342+0000 7efcbec00700 1 -- start start 2024-03-18T01:37:18.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.342+0000 7efcbec00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb8104550 0x7efcb819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:18.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.342+0000 7efcbec00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8105750 0x7efcb819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:18.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.342+0000 7efcbec00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcb81060f0 0x7efcb81a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.342+0000 7efcbec00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efcb8110170 con 0x7efcb81060f0 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.342+0000 7efcbec00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efcb810fff0 con 0x7efcb8104550 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.342+0000 7efcbec00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efcb81102f0 con 0x7efcb8105750 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8105750 0x7efcb819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb8104550 0x7efcb819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb8104550 0x7efcb819ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55928/0 (socket says 172.21.15.7:55928) 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8105750 0x7efcb819f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:50156/0 (socket says 172.21.15.7:50156) 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 -- 172.21.15.7:0/2021641431 learned_addr learned my addr 172.21.15.7:0/2021641431 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcbcc00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcb81060f0 0x7efcb81a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 -- 172.21.15.7:0/2021641431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8105750 msgr2=0x7efcb819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8105750 0x7efcb819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 -- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcb81060f0 msgr2=0x7efcb81a6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcb81060f0 0x7efcb81a6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 -- 172.21.15.7:0/2021641431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb81a6810 con 0x7efcb8104550 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcbcc00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcb81060f0 0x7efcb81a6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.343+0000 7efcb7e00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb8104550 0x7efcb819ebb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efca8000c00 tx=0x7efca8009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.344+0000 7efcb4c00700 1 -- 172.21.15.7:0/2021641431 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8026070 con 0x7efcb8104550 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.344+0000 7efcb4c00700 1 -- 172.21.15.7:0/2021641431 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efca800fde0 con 0x7efcb8104550 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.344+0000 7efcb4c00700 1 -- 172.21.15.7:0/2021641431 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8017650 con 0x7efcb8104550 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.344+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb81a6aa0 con 0x7efcb8104550 2024-03-18T01:37:18.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.344+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb81a6ea0 con 0x7efcb8104550 2024-03-18T01:37:18.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.345+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb8068f90 con 0x7efcb8104550 2024-03-18T01:37:18.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.347+0000 7efcb4c00700 1 -- 172.21.15.7:0/2021641431 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7efca8029070 con 0x7efcb8104550 2024-03-18T01:37:18.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.349+0000 7efcb4c00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efc9c0a2910 0x7efc9c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:18.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.349+0000 7efcb4c00700 1 -- 172.21.15.7:0/2021641431 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efca80cfb60 con 0x7efcb8104550 2024-03-18T01:37:18.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.349+0000 7efcb7400700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efc9c0a2910 0x7efc9c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:18.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.350+0000 7efcb7400700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efc9c0a2910 0x7efc9c0a4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efca4009c60 tx=0x7efca4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:18.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.352+0000 7efcb4c00700 1 -- 172.21.15.7:0/2021641431 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efca8013070 con 0x7efcb8104550 2024-03-18T01:37:18.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:18 smithi007 ceph-mon[31996]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:18 smithi203 ceph-mon[36345]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:18.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.753+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7efcb8079f10 con 0x7efc9c0a2910 2024-03-18T01:37:18.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.755+0000 7efcb4c00700 1 -- 172.21.15.7:0/2021641431 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7efcb8079f10 con 0x7efc9c0a2910 2024-03-18T01:37:18.755 INFO:teuthology.orchestra.run.smithi007.stdout:Backend: cephadm 2024-03-18T01:37:18.756 INFO:teuthology.orchestra.run.smithi007.stdout:Available: Yes 2024-03-18T01:37:18.756 INFO:teuthology.orchestra.run.smithi007.stdout:Paused: No 2024-03-18T01:37:18.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efc9c0a2910 msgr2=0x7efc9c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:18.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efc9c0a2910 0x7efc9c0a4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efca4009c60 tx=0x7efca4009260 comp rx=0 tx=0).stop 2024-03-18T01:37:18.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb8104550 msgr2=0x7efcb819ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb8104550 0x7efcb819ebb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efca8000c00 tx=0x7efca8009580 comp rx=0 tx=0).stop 2024-03-18T01:37:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 shutdown_connections 2024-03-18T01:37:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efc9c0a2910 0x7efc9c0a4dd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efcb81060f0 0x7efcb81a6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb8104550 0x7efcb819ebb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 --2- 172.21.15.7:0/2021641431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcb8105750 0x7efcb819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.759+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 >> 172.21.15.7:0/2021641431 conn(0x7efcb80ffaf0 msgr2=0x7efcb80787f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:18.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.760+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 shutdown_connections 2024-03-18T01:37:18.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:18.760+0000 7efcbec00700 1 -- 172.21.15.7:0/2021641431 wait complete. 2024-03-18T01:37:18.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:18 smithi046 ceph-mon[36108]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:19 smithi007 ceph-mon[31996]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:19.627 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T01:37:19.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:19 smithi203 ceph-mon[36345]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:19.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:19 smithi046 ceph-mon[36108]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:20.125 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:20.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:20 smithi007 ceph-mon[31996]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:20 smithi203 ceph-mon[36345]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:20.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:20 smithi046 ceph-mon[36108]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:22.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 -- 172.21.15.7:0/1458839448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 msgr2=0x7f91ec1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:22.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 --2- 172.21.15.7:0/1458839448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 0x7f91ec1045b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f91d80099c0 tx=0x7f91d8009cd0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 -- 172.21.15.7:0/1458839448 shutdown_connections 2024-03-18T01:37:22.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 --2- 172.21.15.7:0/1458839448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec104af0 0x7f91ec10c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 --2- 172.21.15.7:0/1458839448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91ec105c10 0x7f91ec106030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 --2- 172.21.15.7:0/1458839448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 0x7f91ec1045b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 -- 172.21.15.7:0/1458839448 >> 172.21.15.7:0/1458839448 conn(0x7f91ec0ffaf0 msgr2=0x7f91ec101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:22.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 -- 172.21.15.7:0/1458839448 shutdown_connections 2024-03-18T01:37:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.570+0000 7f91f3200700 1 -- 172.21.15.7:0/1458839448 wait complete. 2024-03-18T01:37:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.571+0000 7f91f3200700 1 Processor -- start 2024-03-18T01:37:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.571+0000 7f91f3200700 1 -- start start 2024-03-18T01:37:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.571+0000 7f91f3200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 0x7f91ec19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91f3200700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec104af0 0x7f91ec19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91f3200700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91ec105c10 0x7f91ec1a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:22.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91f3200700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec110260 con 0x7f91ec105c10 2024-03-18T01:37:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91f3200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec1100e0 con 0x7f91ec104150 2024-03-18T01:37:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91f3200700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec1103e0 con 0x7f91ec104af0 2024-03-18T01:37:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91eb400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec104af0 0x7f91ec19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91ebe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 0x7f91ec19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91eb400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec104af0 0x7f91ec19ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:50182/0 (socket says 172.21.15.7:50182) 2024-03-18T01:37:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91eb400700 1 -- 172.21.15.7:0/2821783278 learned_addr learned my addr 172.21.15.7:0/2821783278 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91ebe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 0x7f91ec19a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55940/0 (socket says 172.21.15.7:55940) 2024-03-18T01:37:22.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91f1200700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91ec105c10 0x7f91ec1a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91eb400700 1 -- 172.21.15.7:0/2821783278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 msgr2=0x7f91ec19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91eb400700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 0x7f91ec19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91eb400700 1 -- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91ec105c10 msgr2=0x7f91ec1a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91eb400700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91ec105c10 0x7f91ec1a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91eb400700 1 -- 172.21.15.7:0/2821783278 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec1a2480 con 0x7f91ec104af0 2024-03-18T01:37:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91f1200700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91ec105c10 0x7f91ec1a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.572+0000 7f91ebe00700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 0x7f91ec19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.573+0000 7f91eb400700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec104af0 0x7f91ec19ad60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f91d8005910 tx=0x7f91d800bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.573+0000 7f91e8c00700 1 -- 172.21.15.7:0/2821783278 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8009e60 con 0x7f91ec104af0 2024-03-18T01:37:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.573+0000 7f91e8c00700 1 -- 172.21.15.7:0/2821783278 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d800f460 con 0x7f91ec104af0 2024-03-18T01:37:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.573+0000 7f91e8c00700 1 -- 172.21.15.7:0/2821783278 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8017650 con 0x7f91ec104af0 2024-03-18T01:37:22.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.573+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ec1a2710 con 0x7f91ec104af0 2024-03-18T01:37:22.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.573+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ec1a2b10 con 0x7f91ec104af0 2024-03-18T01:37:22.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.574+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ec068f90 con 0x7f91ec104af0 2024-03-18T01:37:22.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.576+0000 7f91e8c00700 1 -- 172.21.15.7:0/2821783278 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f91d800f610 con 0x7f91ec104af0 2024-03-18T01:37:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.579+0000 7f91e8c00700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f91d00a2910 0x7f91d00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.579+0000 7f91e8c00700 1 -- 172.21.15.7:0/2821783278 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f91d80cfbe0 con 0x7f91ec104af0 2024-03-18T01:37:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.579+0000 7f91ebe00700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f91d00a2910 0x7f91d00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:22.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.580+0000 7f91ebe00700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f91d00a2910 0x7f91d00a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f91ec105590 tx=0x7f91dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:22.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.582+0000 7f91e8c00700 1 -- 172.21.15.7:0/2821783278 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f91d8013070 con 0x7f91ec104af0 2024-03-18T01:37:22.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:22 smithi203 ceph-mon[36345]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:22.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:22 smithi046 ceph-mon[36108]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:22.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:22 smithi007 ceph-mon[31996]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:22.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.831+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f91ec062710 con 0x7f91d00a2910 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.853+0000 7f91e8c00700 1 -- 172.21.15.7:0/2821783278 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f91ec062710 con 0x7f91d00a2910 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:mds.a smithi007 running (2m) 42s ago 2m 12.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b a8aabe0db2a0 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:mds.b smithi046 running (2m) 83s ago 2m 28.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b efeff0e0d0eb 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:mds.c smithi203 running (2m) 83s ago 2m 16.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d0bd257a2bab 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:mds.d smithi007 running (2m) 42s ago 2m 12.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 0387ab145024 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:mds.e smithi046 running (2m) 83s ago 2m 16.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b f2113119d359 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:mds.f smithi203 running (2m) 83s ago 2m 15.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 66bc46eea89d 2024-03-18T01:37:22.854 INFO:teuthology.orchestra.run.smithi007.stdout:mds.g smithi007 running (2m) 42s ago 2m 16.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 62fd0322f37c 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mds.h smithi046 running (2m) 83s ago 2m 19.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 43371fca8731 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mds.i smithi203 running (2m) 83s ago 2m 19.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b94be326eadc 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mds.j smithi007 running (3m) 42s ago 3m 15.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 6f3aae8dc007 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mds.k smithi046 running (3m) 83s ago 3m 17.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b e12538bc638e 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mds.l smithi203 running (3m) 83s ago 3m 15.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b f292167f2cf1 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi007 *:9283,8765 running (11m) 42s ago 11m 515M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 7d71c73ad82f 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi046 *:8443,8765 running (9m) 83s ago 9m 453M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 532e3a82d13e 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.z smithi203 *:8443,8765 running (9m) 83s ago 9m 453M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 94ad95bb8a37 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (11m) 42s ago 11m 48.8M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b f5ed85e015f2 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi046 running (9m) 83s ago 9m 41.8M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b f6e73844f792 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi203 running (9m) 83s ago 9m 40.3M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 459e01704c44 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (8m) 42s ago 8m 44.6M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 2cd58b430305 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi046 running (8m) 83s ago 8m 57.4M 4271M 19.0.0-2162-g3d78f5ba be86dc8cb22b 4b2ce3d62119 2024-03-18T01:37:22.855 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi203 running (7m) 83s ago 7m 47.4M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 0ce68efba2a1 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (7m) 42s ago 7m 51.1M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 10f50ec8fd5e 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi046 running (6m) 83s ago 6m 46.0M 4271M 19.0.0-2162-g3d78f5ba be86dc8cb22b fae8736942cf 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi203 running (6m) 83s ago 6m 52.4M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 483d7e25c69a 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi007 running (5m) 42s ago 5m 40.9M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b 9076a90d57fc 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi046 running (5m) 83s ago 5m 44.6M 4271M 19.0.0-2162-g3d78f5ba be86dc8cb22b ed03de2b7e76 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 smithi203 running (4m) 83s ago 4m 54.4M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b ce4fc9dce4d9 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 smithi007 running (4m) 42s ago 4m 42.4M 4273M 19.0.0-2162-g3d78f5ba be86dc8cb22b cc5f2dce9df7 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 smithi046 running (3m) 83s ago 3m 44.5M 4271M 19.0.0-2162-g3d78f5ba be86dc8cb22b 3964f92208be 2024-03-18T01:37:22.856 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 smithi203 running (3m) 83s ago 3m 44.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b be1945d11dda 2024-03-18T01:37:22.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f91d00a2910 msgr2=0x7f91d00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:22.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f91d00a2910 0x7f91d00a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f91ec105590 tx=0x7f91dc009260 comp rx=0 tx=0).stop 2024-03-18T01:37:22.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec104af0 msgr2=0x7f91ec19ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec104af0 0x7f91ec19ad60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f91d8005910 tx=0x7f91d800bee0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 shutdown_connections 2024-03-18T01:37:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f91d00a2910 0x7f91d00a4dd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f91ec105c10 0x7f91ec1a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91ec104150 0x7f91ec19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 --2- 172.21.15.7:0/2821783278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91ec104af0 0x7f91ec19ad60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 >> 172.21.15.7:0/2821783278 conn(0x7f91ec0ffaf0 msgr2=0x7f91ec1089d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:22.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 shutdown_connections 2024-03-18T01:37:22.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:22.857+0000 7f91f3200700 1 -- 172.21.15.7:0/2821783278 wait complete. 2024-03-18T01:37:23.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:23 smithi007 ceph-mon[31996]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:23.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:23 smithi203 ceph-mon[36345]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:23.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T01:37:23.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:23 smithi046 ceph-mon[36108]: from='client.25252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:24.159 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:24.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:24 smithi007 ceph-mon[31996]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:24.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:24 smithi203 ceph-mon[36345]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:24.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:24 smithi046 ceph-mon[36108]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 -- 172.21.15.7:0/1905882813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284102350 msgr2=0x7f3284102770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 --2- 172.21.15.7:0/1905882813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284102350 0x7f3284102770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f32740099c0 tx=0x7f3274009cd0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 -- 172.21.15.7:0/1905882813 shutdown_connections 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 --2- 172.21.15.7:0/1905882813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3284103ef0 0x7f328410a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 --2- 172.21.15.7:0/1905882813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284102350 0x7f3284102770 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 --2- 172.21.15.7:0/1905882813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3284103550 0x7f32841039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 -- 172.21.15.7:0/1905882813 >> 172.21.15.7:0/1905882813 conn(0x7f32840fd8f0 msgr2=0x7f32840ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 -- 172.21.15.7:0/1905882813 shutdown_connections 2024-03-18T01:37:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.571+0000 7f3289a00700 1 -- 172.21.15.7:0/1905882813 wait complete. 2024-03-18T01:37:26.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.572+0000 7f3289a00700 1 Processor -- start 2024-03-18T01:37:26.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.572+0000 7f3289a00700 1 -- start start 2024-03-18T01:37:26.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.572+0000 7f3289a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3284102350 0x7f328419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3289a00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3284103550 0x7f328419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3289a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284103ef0 0x7f32841a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3289a00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f328410df60 con 0x7f3284103550 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3289a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f328410dde0 con 0x7f3284103ef0 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3289a00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f328410e0e0 con 0x7f3284102350 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3282a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3284102350 0x7f328419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3282a00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3284102350 0x7f328419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36258/0 (socket says 172.21.15.7:36258) 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3282a00700 1 -- 172.21.15.7:0/3015854646 learned_addr learned my addr 172.21.15.7:0/3015854646 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f327be00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3284103550 0x7f328419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3283400700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284103ef0 0x7f32841a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3282a00700 1 -- 172.21.15.7:0/3015854646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284103ef0 msgr2=0x7f32841a1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3282a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284103ef0 0x7f32841a1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3282a00700 1 -- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3284103550 msgr2=0x7f328419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:26.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3282a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3284103550 0x7f328419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3282a00700 1 -- 172.21.15.7:0/3015854646 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32841a23c0 con 0x7f3284102350 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.573+0000 7f3283400700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284103ef0 0x7f32841a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.574+0000 7f3282a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3284102350 0x7f328419a760 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3274005b40 tx=0x7f3274009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.574+0000 7f327b400700 1 -- 172.21.15.7:0/3015854646 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274027070 con 0x7f3284102350 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.574+0000 7f327b400700 1 -- 172.21.15.7:0/3015854646 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3274022470 con 0x7f3284102350 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.574+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32841a2650 con 0x7f3284102350 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.574+0000 7f327b400700 1 -- 172.21.15.7:0/3015854646 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274009e60 con 0x7f3284102350 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.574+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32841a2b00 con 0x7f3284102350 2024-03-18T01:37:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.575+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3284068f90 con 0x7f3284102350 2024-03-18T01:37:26.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.579+0000 7f327b400700 1 -- 172.21.15.7:0/3015854646 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f327402a070 con 0x7f3284102350 2024-03-18T01:37:26.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.581+0000 7f327b400700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32680a2910 0x7f32680a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:26.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.581+0000 7f327b400700 1 -- 172.21.15.7:0/3015854646 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f32740cfbb0 con 0x7f3284102350 2024-03-18T01:37:26.581 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.581+0000 7f327be00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32680a2910 0x7f32680a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:26.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.582+0000 7f327be00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32680a2910 0x7f32680a4dd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f326c009c60 tx=0x7f326c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:26.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.586+0000 7f327b400700 1 -- 172.21.15.7:0/3015854646 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3274013070 con 0x7f3284102350 2024-03-18T01:37:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:26 smithi203 ceph-mon[36345]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:26 smithi046 ceph-mon[36108]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:26 smithi007 ceph-mon[31996]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:26.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.824+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f3284075c20 con 0x7f32680a2910 2024-03-18T01:37:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.831+0000 7f327b400700 1 -- 172.21.15.7:0/3015854646 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f3284075c20 con 0x7f32680a2910 2024-03-18T01:37:26.831 INFO:teuthology.orchestra.run.smithi007.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T01:37:26.831 INFO:teuthology.orchestra.run.smithi007.stdout:mds.all 12/12 87s ago 3m smithi007=a;smithi007=d;smithi007=g;smithi007=j;smithi046=b;smithi046=e;smithi046=h;smithi046=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-03-18T01:37:26.831 INFO:teuthology.orchestra.run.smithi007.stdout:mgr 3/3 87s ago 9m smithi007=x;smithi046=y;smithi203=z;count:3 2024-03-18T01:37:26.831 INFO:teuthology.orchestra.run.smithi007.stdout:mon 3/3 87s ago 9m smithi007:172.21.15.7=a;smithi046:172.21.15.46=b;smithi203:172.21.15.203=c;count:3 2024-03-18T01:37:26.832 INFO:teuthology.orchestra.run.smithi007.stdout:osd 12 87s ago - 2024-03-18T01:37:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32680a2910 msgr2=0x7f32680a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32680a2910 0x7f32680a4dd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f326c009c60 tx=0x7f326c009260 comp rx=0 tx=0).stop 2024-03-18T01:37:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3284102350 msgr2=0x7f328419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3284102350 0x7f328419a760 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3274005b40 tx=0x7f3274009580 comp rx=0 tx=0).stop 2024-03-18T01:37:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 shutdown_connections 2024-03-18T01:37:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f32680a2910 0x7f32680a4dd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3284103ef0 0x7f32841a1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3284102350 0x7f328419a760 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 --2- 172.21.15.7:0/3015854646 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3284103550 0x7f328419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 >> 172.21.15.7:0/3015854646 conn(0x7f32840fd8f0 msgr2=0x7f3284074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 shutdown_connections 2024-03-18T01:37:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:26.834+0000 7f3289a00700 1 -- 172.21.15.7:0/3015854646 wait complete. 2024-03-18T01:37:27.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:27 smithi007 ceph-mon[31996]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:27.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:27 smithi203 ceph-mon[36345]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:27.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T01:37:27.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:27 smithi046 ceph-mon[36108]: from='client.25258 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:28.183 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:28.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:28 smithi007 ceph-mon[31996]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:28 smithi203 ceph-mon[36345]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:28.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:28 smithi046 ceph-mon[36108]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:30.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:30 smithi203 ceph-mon[36345]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:30.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 -- 172.21.15.7:0/239543536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818103ec0 msgr2=0x7f581810a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:30.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 --2- 172.21.15.7:0/239543536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818103ec0 0x7f581810a770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f581400b0a0 tx=0x7f581400b3b0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.652 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:30 smithi007 ceph-mon[31996]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 -- 172.21.15.7:0/239543536 shutdown_connections 2024-03-18T01:37:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 --2- 172.21.15.7:0/239543536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818103ec0 0x7f581810a770 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 --2- 172.21.15.7:0/239543536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818102320 0x7f5818102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 --2- 172.21.15.7:0/239543536 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5818103520 0x7f5818103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 -- 172.21.15.7:0/239543536 >> 172.21.15.7:0/239543536 conn(0x7f58180fd880 msgr2=0x7f58180ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 -- 172.21.15.7:0/239543536 shutdown_connections 2024-03-18T01:37:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.651+0000 7f5820800700 1 -- 172.21.15.7:0/239543536 wait complete. 2024-03-18T01:37:30.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.653+0000 7f5820800700 1 Processor -- start 2024-03-18T01:37:30.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.653+0000 7f5820800700 1 -- start start 2024-03-18T01:37:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.653+0000 7f5820800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818102320 0x7f5818076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.653+0000 7f5820800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818103520 0x7f58180746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.653+0000 7f5820800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5818103ec0 0x7f5818074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.653+0000 7f5820800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f581810df40 con 0x7f5818103ec0 2024-03-18T01:37:30.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.653+0000 7f5820800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f581810ddc0 con 0x7f5818103520 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.653+0000 7f5820800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f581810e0c0 con 0x7f5818102320 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818103520 0x7f58180746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581de00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818102320 0x7f5818076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818103520 0x7f58180746c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:40208/0 (socket says 172.21.15.7:40208) 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 -- 172.21.15.7:0/2025962315 learned_addr learned my addr 172.21.15.7:0/2025962315 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581de00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818102320 0x7f5818076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:36264/0 (socket says 172.21.15.7:36264) 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581e800700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5818103ec0 0x7f5818074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 -- 172.21.15.7:0/2025962315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818102320 msgr2=0x7f5818076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818102320 0x7f5818076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 -- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5818103ec0 msgr2=0x7f5818074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5818103ec0 0x7f5818074c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 -- 172.21.15.7:0/2025962315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58180754b0 con 0x7f5818103520 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581de00700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818102320 0x7f5818076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:30.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.654+0000 7f581d400700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818103520 0x7f58180746c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f580800e9a0 tx=0x7f580800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.655+0000 7f5812a00700 1 -- 172.21.15.7:0/2025962315 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580800ca10 con 0x7f5818103520 2024-03-18T01:37:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.655+0000 7f5812a00700 1 -- 172.21.15.7:0/2025962315 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f580800cbb0 con 0x7f5818103520 2024-03-18T01:37:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.655+0000 7f5812a00700 1 -- 172.21.15.7:0/2025962315 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808018680 con 0x7f5818103520 2024-03-18T01:37:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.655+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58181a2650 con 0x7f5818103520 2024-03-18T01:37:30.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.655+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58181a2b10 con 0x7f5818103520 2024-03-18T01:37:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.656+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5818068f90 con 0x7f5818103520 2024-03-18T01:37:30.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.658+0000 7f5812a00700 1 -- 172.21.15.7:0/2025962315 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f5808010430 con 0x7f5818103520 2024-03-18T01:37:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.659+0000 7f5812a00700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58000a2910 0x7f58000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.659+0000 7f5812a00700 1 -- 172.21.15.7:0/2025962315 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f58080cefd0 con 0x7f5818103520 2024-03-18T01:37:30.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.659+0000 7f581de00700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58000a2910 0x7f58000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:30.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.660+0000 7f581de00700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58000a2910 0x7f58000a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5818103380 tx=0x7f580c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.663+0000 7f5812a00700 1 -- 172.21.15.7:0/2025962315 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f580809bcb0 con 0x7f5818103520 2024-03-18T01:37:30.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:30 smithi046 ceph-mon[36108]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:30.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.899+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f5818062710 con 0x7f58000a2910 2024-03-18T01:37:30.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.901+0000 7f5812a00700 1 -- 172.21.15.7:0/2025962315 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f5818062710 con 0x7f58000a2910 2024-03-18T01:37:30.902 INFO:teuthology.orchestra.run.smithi007.stdout:HOST ADDR LABELS STATUS 2024-03-18T01:37:30.902 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 172.21.15.7 2024-03-18T01:37:30.902 INFO:teuthology.orchestra.run.smithi007.stdout:smithi046 172.21.15.46 2024-03-18T01:37:30.902 INFO:teuthology.orchestra.run.smithi007.stdout:smithi203 172.21.15.203 2024-03-18T01:37:30.902 INFO:teuthology.orchestra.run.smithi007.stdout:3 hosts in cluster 2024-03-18T01:37:30.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.904+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58000a2910 msgr2=0x7f58000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:30.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.904+0000 7f5820800700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58000a2910 0x7f58000a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5818103380 tx=0x7f580c009260 comp rx=0 tx=0).stop 2024-03-18T01:37:30.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818103520 msgr2=0x7f58180746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:30.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818103520 0x7f58180746c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f580800e9a0 tx=0x7f580800ecb0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 shutdown_connections 2024-03-18T01:37:30.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58000a2910 0x7f58000a4dd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5818103ec0 0x7f5818074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5818102320 0x7f5818076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 --2- 172.21.15.7:0/2025962315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5818103520 0x7f58180746c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:30.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 >> 172.21.15.7:0/2025962315 conn(0x7f58180fd880 msgr2=0x7f5818106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:30.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 shutdown_connections 2024-03-18T01:37:30.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:30.905+0000 7f5820800700 1 -- 172.21.15.7:0/2025962315 wait complete. 2024-03-18T01:37:31.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:31 smithi007 ceph-mon[31996]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:31.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:31 smithi203 ceph-mon[36345]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:31.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:31 smithi046 ceph-mon[36108]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:31.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T01:37:32.428 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:32.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:32 smithi203 ceph-mon[36345]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:32.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:32 smithi046 ceph-mon[36108]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:32.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:32 smithi007 ceph-mon[31996]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:34.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:34 smithi007 ceph-mon[31996]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:34.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:34 smithi203 ceph-mon[36345]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:34.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 -- 172.21.15.7:0/1942470259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4cc076c80 msgr2=0x7fc4cc0770a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 --2- 172.21.15.7:0/1942470259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4cc076c80 0x7fc4cc0770a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009990 tx=0x7fc4c0009ca0 comp rx=0 tx=0).stop 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 -- 172.21.15.7:0/1942470259 shutdown_connections 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 --2- 172.21.15.7:0/1942470259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4cc078890 0x7fc4cc079110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 --2- 172.21.15.7:0/1942470259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4cc076c80 0x7fc4cc0770a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 --2- 172.21.15.7:0/1942470259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4cc077ed0 0x7fc4cc078350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 -- 172.21.15.7:0/1942470259 >> 172.21.15.7:0/1942470259 conn(0x7fc4cc0ffaf0 msgr2=0x7fc4cc101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 -- 172.21.15.7:0/1942470259 shutdown_connections 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.747+0000 7fc4d3600700 1 -- 172.21.15.7:0/1942470259 wait complete. 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.748+0000 7fc4d3600700 1 Processor -- start 2024-03-18T01:37:34.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.748+0000 7fc4d3600700 1 -- start start 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d3600700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4cc076c80 0x7fc4cc19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d3600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4cc077ed0 0x7fc4cc19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d3600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4cc078890 0x7fc4cc1a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d3600700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc114570 con 0x7fc4cc076c80 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d3600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1143f0 con 0x7fc4cc077ed0 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d3600700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1146f0 con 0x7fc4cc078890 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d0c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4cc076c80 0x7fc4cc19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4cbe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4cc077ed0 0x7fc4cc19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d1600700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4cc078890 0x7fc4cc1a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d0c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4cc076c80 0x7fc4cc19eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53608/0 (socket says 172.21.15.7:53608) 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.749+0000 7fc4d0c00700 1 -- 172.21.15.7:0/751429052 learned_addr learned my addr 172.21.15.7:0/751429052 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.750+0000 7fc4d1600700 1 -- 172.21.15.7:0/751429052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4cc077ed0 msgr2=0x7fc4cc19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:34.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.750+0000 7fc4d1600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4cc077ed0 0x7fc4cc19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.750+0000 7fc4d1600700 1 -- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4cc076c80 msgr2=0x7fc4cc19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.750+0000 7fc4d1600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4cc076c80 0x7fc4cc19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.750+0000 7fc4d1600700 1 -- 172.21.15.7:0/751429052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc1a6770 con 0x7fc4cc078890 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.750+0000 7fc4d1600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4cc078890 0x7fc4cc1a60d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009f60 tx=0x7fc4c4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.750+0000 7fc4c9600700 1 -- 172.21.15.7:0/751429052 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c40092e0 con 0x7fc4cc078890 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.751+0000 7fc4c9600700 1 -- 172.21.15.7:0/751429052 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4004bb0 con 0x7fc4cc078890 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.751+0000 7fc4c9600700 1 -- 172.21.15.7:0/751429052 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4004280 con 0x7fc4cc078890 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.751+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4cc1a6a00 con 0x7fc4cc078890 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.751+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4cc1a6ef0 con 0x7fc4cc078890 2024-03-18T01:37:34.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.752+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4cc068f90 con 0x7fc4cc078890 2024-03-18T01:37:34.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.754+0000 7fc4c9600700 1 -- 172.21.15.7:0/751429052 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fc4c401b030 con 0x7fc4cc078890 2024-03-18T01:37:34.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.755+0000 7fc4c9600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc4b40a2890 0x7fc4b40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:34.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.756+0000 7fc4c9600700 1 -- 172.21.15.7:0/751429052 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc4c40ce870 con 0x7fc4cc078890 2024-03-18T01:37:34.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.756+0000 7fc4d0c00700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc4b40a2890 0x7fc4b40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:34.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.756+0000 7fc4d0c00700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc4b40a2890 0x7fc4b40a4d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000b2e0 tx=0x7fc4c000b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:34.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.759+0000 7fc4c9600700 1 -- 172.21.15.7:0/751429052 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc4c4022020 con 0x7fc4cc078890 2024-03-18T01:37:34.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:34 smithi046 ceph-mon[36108]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:34.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:34.996+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fc4cc062710 con 0x7fc4b40a2890 2024-03-18T01:37:35.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.001+0000 7fc4c9600700 1 -- 172.21.15.7:0/751429052 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fc4cc062710 con 0x7fc4b40a2890 2024-03-18T01:37:35.002 INFO:teuthology.orchestra.run.smithi007.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T01:37:35.002 INFO:teuthology.orchestra.run.smithi007.stdout:smithi007 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300GE480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-03-18T01:37:35.002 INFO:teuthology.orchestra.run.smithi007.stdout:smithi046 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009A480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-03-18T01:37:35.002 INFO:teuthology.orchestra.run.smithi007.stdout:smithi203 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc4b40a2890 msgr2=0x7fc4b40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc4b40a2890 0x7fc4b40a4d50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000b2e0 tx=0x7fc4c000b880 comp rx=0 tx=0).stop 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4cc078890 msgr2=0x7fc4cc1a60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4cc078890 0x7fc4cc1a60d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009f60 tx=0x7fc4c4009f90 comp rx=0 tx=0).stop 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 shutdown_connections 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc4b40a2890 0x7fc4b40a4d50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4cc078890 0x7fc4cc1a60d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc4cc076c80 0x7fc4cc19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 --2- 172.21.15.7:0/751429052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4cc077ed0 0x7fc4cc19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 >> 172.21.15.7:0/751429052 conn(0x7fc4cc0ffaf0 msgr2=0x7fc4cc1015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.004+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 shutdown_connections 2024-03-18T01:37:35.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:35.005+0000 7fc4d3600700 1 -- 172.21.15.7:0/751429052 wait complete. 2024-03-18T01:37:35.899 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T01:37:35.909 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-03-18T01:37:35.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T01:37:36.391 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:36 smithi203 ceph-mon[36345]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:36.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:36 smithi203 ceph-mon[36345]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:36 smithi007 ceph-mon[31996]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:36 smithi007 ceph-mon[31996]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:36.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:36 smithi046 ceph-mon[36108]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T01:37:36.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:36 smithi046 ceph-mon[36108]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:38.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:38 smithi007 ceph-mon[31996]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:38.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:38 smithi046 ceph-mon[36108]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:38.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.841+0000 7f8c46c00700 1 -- 172.21.15.7:0/4039942253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c40076280 msgr2=0x7f8c400746e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.841+0000 7f8c46c00700 1 --2- 172.21.15.7:0/4039942253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c40076280 0x7f8c400746e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8c300098f0 tx=0x7f8c30009c00 comp rx=0 tx=0).stop 2024-03-18T01:37:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.841+0000 7f8c46c00700 1 -- 172.21.15.7:0/4039942253 shutdown_connections 2024-03-18T01:37:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.841+0000 7f8c46c00700 1 --2- 172.21.15.7:0/4039942253 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c4010a250 0x7f8c4010c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.841+0000 7f8c46c00700 1 --2- 172.21.15.7:0/4039942253 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c40076280 0x7f8c400746e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.841+0000 7f8c46c00700 1 --2- 172.21.15.7:0/4039942253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40074cb0 0x7f8c40075130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.841+0000 7f8c46c00700 1 -- 172.21.15.7:0/4039942253 >> 172.21.15.7:0/4039942253 conn(0x7f8c400fd8f0 msgr2=0x7f8c400ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.842+0000 7f8c46c00700 1 -- 172.21.15.7:0/4039942253 shutdown_connections 2024-03-18T01:37:38.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.842+0000 7f8c46c00700 1 -- 172.21.15.7:0/4039942253 wait complete. 2024-03-18T01:37:38.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.843+0000 7f8c46c00700 1 Processor -- start 2024-03-18T01:37:38.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.843+0000 7f8c46c00700 1 -- start start 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.843+0000 7f8c46c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40074cb0 0x7f8c4019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.843+0000 7f8c46c00700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c40076280 0x7f8c4019acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.843+0000 7f8c46c00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c4010a250 0x7f8c401a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.843+0000 7f8c46c00700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8c40110150 con 0x7f8c40076280 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.843+0000 7f8c46c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4010ffd0 con 0x7f8c40074cb0 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.843+0000 7f8c46c00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c401102d0 con 0x7f8c4010a250 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40074cb0 0x7f8c4019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40074cb0 0x7f8c4019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41496/0 (socket says 172.21.15.7:41496) 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3fe00700 1 -- 172.21.15.7:0/3945363625 learned_addr learned my addr 172.21.15.7:0/3945363625 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3f400700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c40076280 0x7f8c4019acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c44c00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c4010a250 0x7f8c401a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:38.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3fe00700 1 -- 172.21.15.7:0/3945363625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c4010a250 msgr2=0x7f8c401a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:38.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3fe00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c4010a250 0x7f8c401a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:38.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3fe00700 1 -- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c40076280 msgr2=0x7f8c4019acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:38.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3fe00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c40076280 0x7f8c4019acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:38.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3fe00700 1 -- 172.21.15.7:0/3945363625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c401a23d0 con 0x7f8c40074cb0 2024-03-18T01:37:38.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c44c00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c4010a250 0x7f8c401a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:38.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.844+0000 7f8c3f400700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c40076280 0x7f8c4019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:38.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.845+0000 7f8c3fe00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40074cb0 0x7f8c4019a770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009810 tx=0x7f8c30005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:38.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.845+0000 7f8c3cc00700 1 -- 172.21.15.7:0/3945363625 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c300054f0 con 0x7f8c40074cb0 2024-03-18T01:37:38.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.845+0000 7f8c3cc00700 1 -- 172.21.15.7:0/3945363625 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c30005690 con 0x7f8c40074cb0 2024-03-18T01:37:38.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.845+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c401a2660 con 0x7f8c40074cb0 2024-03-18T01:37:38.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.845+0000 7f8c3cc00700 1 -- 172.21.15.7:0/3945363625 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30003d50 con 0x7f8c40074cb0 2024-03-18T01:37:38.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.845+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c401a2b10 con 0x7f8c40074cb0 2024-03-18T01:37:38.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.846+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c40068f90 con 0x7f8c40074cb0 2024-03-18T01:37:38.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.848+0000 7f8c3cc00700 1 -- 172.21.15.7:0/3945363625 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f8c3000bbb0 con 0x7f8c40074cb0 2024-03-18T01:37:38.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.850+0000 7f8c3cc00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8c240a2910 0x7f8c240a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:38.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.850+0000 7f8c3cc00700 1 -- 172.21.15.7:0/3945363625 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8c300d0120 con 0x7f8c40074cb0 2024-03-18T01:37:38.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.850+0000 7f8c3f400700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8c240a2910 0x7f8c240a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:38.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.851+0000 7f8c3f400700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8c240a2910 0x7f8c240a4dd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c005fd0 tx=0x7f8c2c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:38.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:38.852+0000 7f8c3cc00700 1 -- 172.21.15.7:0/3945363625 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8c3005e740 con 0x7f8c40074cb0 2024-03-18T01:37:38.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:38 smithi203 ceph-mon[36345]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:39.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.131+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f8c4002db70 con 0x7f8c40074cb0 2024-03-18T01:37:39.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.132+0000 7f8c3cc00700 1 -- 172.21.15.7:0/3945363625 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3082 (secure 0 0 0) 0x7f8c30013030 con 0x7f8c40074cb0 2024-03-18T01:37:39.134 INFO:teuthology.orchestra.run.smithi007.stdout:e27 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:legacy client fscid: 1 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:Filesystem 'cephfs' (1) 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:fs_name cephfs 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 27 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-03-18T01:34:56.125856+0000 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:modified 2024-03-18T01:35:30.634148+0000 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:tableserver 0 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:root 0 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:session_timeout 300 2024-03-18T01:37:39.135 INFO:teuthology.orchestra.run.smithi007.stdout:session_autoclose 300 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:max_file_size 1099511627776 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:max_xattr_size 65536 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:required_client_features {} 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:last_failure 0 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:last_failure_osd_epoch 0 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:max_mds 5 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:in 0,1,2,3,4 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:up {0=24596,1=24562,2=24544,3=24542,4=24523} 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:failed 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:damaged 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:stopped 2024-03-18T01:37:39.136 INFO:teuthology.orchestra.run.smithi007.stdout:data_pools [3,4] 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:metadata_pool 2 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:inline_data disabled 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:balancer 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:bal_rank_mask -1 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:standby_count_wanted 0 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:qdb_cluster leader: 24596 members: 24542,24523,24544,24596,24562 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.b{0:24596} state up:active seq 3 addr [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.e{0:24512} state up:standby-replay seq 1 addr [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.g{1:24562} state up:active seq 13 addr [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.f{1:24511} state up:standby-replay seq 1 addr [v2:172.21.15.203:6834/848080613,v1:172.21.15.203:6835/848080613] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.i{2:24544} state up:active seq 14 addr [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.j{2:24475} state up:standby-replay seq 1 addr [v2:172.21.15.7:6834/49989672,v1:172.21.15.7:6835/49989672] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.137 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.h{3:24542} state up:active seq 15 addr [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.k{3:24464} state up:standby-replay seq 1 addr [v2:172.21.15.46:6832/1027889667,v1:172.21.15.46:6833/1027889667] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.c{4:24523} state up:active seq 16 addr [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.l{4:24448} state up:standby-replay seq 1 addr [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout:Standby daemons: 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.d{-1:14403} state up:standby seq 1 addr [v2:172.21.15.7:6836/3527563344,v1:172.21.15.7:6837/3527563344] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.138 INFO:teuthology.orchestra.run.smithi007.stdout:[mds.a{-1:14469} state up:standby seq 1 addr [v2:172.21.15.7:6840/1978448378,v1:172.21.15.7:6841/1978448378] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T01:37:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8c240a2910 msgr2=0x7f8c240a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8c240a2910 0x7f8c240a4dd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c005fd0 tx=0x7f8c2c009310 comp rx=0 tx=0).stop 2024-03-18T01:37:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40074cb0 msgr2=0x7f8c4019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40074cb0 0x7f8c4019a770 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009810 tx=0x7f8c30005250 comp rx=0 tx=0).stop 2024-03-18T01:37:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 shutdown_connections 2024-03-18T01:37:39.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8c240a2910 0x7f8c240a4dd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c4010a250 0x7f8c401a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c40074cb0 0x7f8c4019a770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 --2- 172.21.15.7:0/3945363625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8c40076280 0x7f8c4019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 >> 172.21.15.7:0/3945363625 conn(0x7f8c400fd8f0 msgr2=0x7f8c401061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 shutdown_connections 2024-03-18T01:37:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:39.135+0000 7f8c46c00700 1 -- 172.21.15.7:0/3945363625 wait complete. 2024-03-18T01:37:39.140 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2024-03-18T01:37:39.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:39 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3945363625' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:37:39.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:39 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3945363625' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:37:39.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:39 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3945363625' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T01:37:39.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-18T01:37:40.458 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:40.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:40 smithi007 ceph-mon[31996]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:40.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:40 smithi046 ceph-mon[36108]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:40.792 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:40 smithi203 ceph-mon[36345]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:42.508 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:42 smithi007 ceph-mon[31996]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T01:37:42.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 -- 172.21.15.7:0/3682103665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0078890 msgr2=0x7f41c0079110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:42.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 --2- 172.21.15.7:0/3682103665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0078890 0x7f41c0079110 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f41b800b0a0 tx=0x7f41b800b3b0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 -- 172.21.15.7:0/3682103665 shutdown_connections 2024-03-18T01:37:42.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 --2- 172.21.15.7:0/3682103665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0078890 0x7f41c0079110 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 --2- 172.21.15.7:0/3682103665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c0076c80 0x7f41c00770a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 --2- 172.21.15.7:0/3682103665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c0077ed0 0x7f41c0078350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 -- 172.21.15.7:0/3682103665 >> 172.21.15.7:0/3682103665 conn(0x7f41c00ffaf0 msgr2=0x7f41c0101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:42.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 -- 172.21.15.7:0/3682103665 shutdown_connections 2024-03-18T01:37:42.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.684+0000 7f41c5400700 1 -- 172.21.15.7:0/3682103665 wait complete. 2024-03-18T01:37:42.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.685+0000 7f41c5400700 1 Processor -- start 2024-03-18T01:37:42.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.685+0000 7f41c5400700 1 -- start start 2024-03-18T01:37:42.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.685+0000 7f41c5400700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c0076c80 0x7f41c019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:42.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.685+0000 7f41c5400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0077ed0 0x7f41c019f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:42.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.685+0000 7f41c5400700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c0078890 0x7f41c01a6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:42.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41c5400700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f41c0114570 con 0x7f41c0078890 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41c5400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01143f0 con 0x7f41c0077ed0 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41c5400700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41c01146f0 con 0x7f41c0076c80 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41be800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c0076c80 0x7f41c019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41be800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c0076c80 0x7f41c019eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53470/0 (socket says 172.21.15.7:53470) 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41be800700 1 -- 172.21.15.7:0/3370181772 learned_addr learned my addr 172.21.15.7:0/3370181772 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41b5e00700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0077ed0 0x7f41c019f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41bf200700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c0078890 0x7f41c01a6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41be800700 1 -- 172.21.15.7:0/3370181772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0077ed0 msgr2=0x7f41c019f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41be800700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0077ed0 0x7f41c019f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41be800700 1 -- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c0078890 msgr2=0x7f41c01a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41be800700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c0078890 0x7f41c01a6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41be800700 1 -- 172.21.15.7:0/3370181772 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c01a66f0 con 0x7f41c0076c80 2024-03-18T01:37:42.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.686+0000 7f41b5e00700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0077ed0 0x7f41c019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:42.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.687+0000 7f41be800700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c0076c80 0x7f41c019eae0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f41b000d980 tx=0x7f41b000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:42.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.687+0000 7f41b7e00700 1 -- 172.21.15.7:0/3370181772 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0016070 con 0x7f41c0076c80 2024-03-18T01:37:42.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.687+0000 7f41b7e00700 1 -- 172.21.15.7:0/3370181772 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b000de70 con 0x7f41c0076c80 2024-03-18T01:37:42.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.687+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c01a69e0 con 0x7f41c0076c80 2024-03-18T01:37:42.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.687+0000 7f41b7e00700 1 -- 172.21.15.7:0/3370181772 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b000f3c0 con 0x7f41c0076c80 2024-03-18T01:37:42.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.687+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c01a6f20 con 0x7f41c0076c80 2024-03-18T01:37:42.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.688+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c0068f90 con 0x7f41c0076c80 2024-03-18T01:37:42.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.690+0000 7f41b7e00700 1 -- 172.21.15.7:0/3370181772 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f41b0019070 con 0x7f41c0076c80 2024-03-18T01:37:42.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.692+0000 7f41b7e00700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f41a40a2890 0x7f41a40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:42.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.693+0000 7f41b7e00700 1 -- 172.21.15.7:0/3370181772 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f41b00ce9a0 con 0x7f41c0076c80 2024-03-18T01:37:42.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.693+0000 7f41b5e00700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f41a40a2890 0x7f41a40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:42.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.694+0000 7f41b5e00700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f41a40a2890 0x7f41a40a4d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f41a8011490 tx=0x7f41a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:42.696 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.695+0000 7f41b7e00700 1 -- 172.21.15.7:0/3370181772 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41b009b680 con 0x7f41c0076c80 2024-03-18T01:37:42.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:42 smithi046 ceph-mon[36108]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T01:37:42.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:42 smithi203 ceph-mon[36345]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T01:37:42.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.928+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f41c0062ee0 con 0x7f41c0076c80 2024-03-18T01:37:42.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.929+0000 7f41b7e00700 1 -- 172.21.15.7:0/3370181772 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) ==== 56+0+5562 (secure 0 0 0) 0x7f41b009af80 con 0x7f41c0076c80 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 91 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-03-18T01:25:48.473512+0000 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:modified 2024-03-18T01:35:58.796670+0000 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:crush_version 27 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:full_ratio 0.95 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:backfillfull_ratio 0.9 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:nearfull_ratio 0.85 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:require_min_compat_client luminous 2024-03-18T01:37:42.932 INFO:teuthology.orchestra.run.smithi007.stdout:min_compat_client luminous 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:require_osd_release squid 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:stretch_mode_enabled false 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:max_osd 12 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.7:6802/2740971601,v1:172.21.15.7:6803/2740971601] [v2:172.21.15.7:6804/2740971601,v1:172.21.15.7:6805/2740971601] exists,up ae73eb8b-e60e-49d5-9fb1-9240be07f1d7 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6800/868303069,v1:172.21.15.46:6801/868303069] [v2:172.21.15.46:6802/868303069,v1:172.21.15.46:6803/868303069] exists,up 1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/3828545760,v1:172.21.15.203:6801/3828545760] [v2:172.21.15.203:6802/3828545760,v1:172.21.15.203:6803/3828545760] exists,up 32b52cb8-4fcc-4a3a-9e2a-69a8205f0844 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.7:6810/2660921725,v1:172.21.15.7:6811/2660921725] [v2:172.21.15.7:6812/2660921725,v1:172.21.15.7:6813/2660921725] exists,up b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6808/35412412,v1:172.21.15.46:6809/35412412] [v2:172.21.15.46:6810/35412412,v1:172.21.15.46:6811/35412412] exists,up 175f4003-26aa-4f4c-b0f9-cb83d4d8ab87 2024-03-18T01:37:42.933 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/2080514850,v1:172.21.15.203:6809/2080514850] [v2:172.21.15.203:6810/2080514850,v1:172.21.15.203:6811/2080514850] exists,up 30797882-136d-4d56-8f01-3e02850e5c9c 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.7:6818/3071783622,v1:172.21.15.7:6819/3071783622] [v2:172.21.15.7:6820/3071783622,v1:172.21.15.7:6821/3071783622] exists,up 646180e1-c71d-4288-88f0-20ded1848288 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6816/3445048558,v1:172.21.15.46:6817/3445048558] [v2:172.21.15.46:6818/3445048558,v1:172.21.15.46:6819/3445048558] exists,up aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:osd.8 up in weight 1 up_from 59 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/1251446815,v1:172.21.15.203:6817/1251446815] [v2:172.21.15.203:6818/1251446815,v1:172.21.15.203:6819/1251446815] exists,up db012442-b453-4810-8aba-51d2e9506797 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.7:6826/4123652106,v1:172.21.15.7:6827/4123652106] [v2:172.21.15.7:6828/4123652106,v1:172.21.15.7:6829/4123652106] exists,up 352a39d7-32c4-461f-8c31-61badd6ba568 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6824/511858712,v1:172.21.15.46:6825/511858712] [v2:172.21.15.46:6826/511858712,v1:172.21.15.46:6827/511858712] exists,up d452ca28-0c10-48a8-bd4a-f01b4e4986c0 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/2624070508,v1:172.21.15.203:6825/2624070508] [v2:172.21.15.203:6826/2624070508,v1:172.21.15.203:6827/2624070508] exists,up 7b43ff2d-9d94-4fd3-9a3d-9a2da004016b 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 2.13 [2,11] 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 2.1c [8,11] 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.3 [4,9] 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.10 [5,9] 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.17 [4,8] 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 3.3c [4,9] 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.10 [9,10] 2024-03-18T01:37:42.934 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.15 [5,10] 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.16 [9,8] 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:pg_upmap_items 4.1b [9,8] 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/1169030810 expires 2024-03-19T01:26:57.023116+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3833186711 expires 2024-03-19T01:26:57.023116+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/4234190466 expires 2024-03-19T01:26:57.023116+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6800/1791220024 expires 2024-03-19T01:26:21.850377+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6801/1169030810 expires 2024-03-19T01:26:57.023116+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:6801/1791220024 expires 2024-03-19T01:26:21.850377+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3317891911 expires 2024-03-19T01:26:21.850377+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2698516506 expires 2024-03-19T01:26:21.850377+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/2862047901 expires 2024-03-19T01:26:21.850377+0000 2024-03-18T01:37:42.935 INFO:teuthology.orchestra.run.smithi007.stdout:blocklist 172.21.15.7:0/3780594555 expires 2024-03-19T01:26:57.023116+0000 2024-03-18T01:37:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f41a40a2890 msgr2=0x7f41a40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f41a40a2890 0x7f41a40a4d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f41a8011490 tx=0x7f41a8009260 comp rx=0 tx=0).stop 2024-03-18T01:37:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c0076c80 msgr2=0x7f41c019eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c0076c80 0x7f41c019eae0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f41b000d980 tx=0x7f41b000dc90 comp rx=0 tx=0).stop 2024-03-18T01:37:42.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 shutdown_connections 2024-03-18T01:37:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f41a40a2890 0x7f41a40a4d50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f41c0078890 0x7f41c01a6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c0076c80 0x7f41c019eae0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 --2- 172.21.15.7:0/3370181772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41c0077ed0 0x7f41c019f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.933+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 >> 172.21.15.7:0/3370181772 conn(0x7f41c00ffaf0 msgr2=0x7f41c01015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.934+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 shutdown_connections 2024-03-18T01:37:42.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:42.934+0000 7f41c5400700 1 -- 172.21.15.7:0/3370181772 wait complete. 2024-03-18T01:37:43.752 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:43 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3370181772' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T01:37:43.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:43 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3370181772' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T01:37:43.809 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-18T01:37:43.821 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:37:43.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:43 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3370181772' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 -- 172.21.15.7:0/2432919315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d4075820 msgr2=0x7fc0d4073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 --2- 172.21.15.7:0/2432919315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d4075820 0x7fc0d4073c70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009900 tx=0x7fc0c802bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 -- 172.21.15.7:0/2432919315 shutdown_connections 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 --2- 172.21.15.7:0/2432919315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d41094a0 0x7fc0d410b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 --2- 172.21.15.7:0/2432919315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0d4074240 0x7fc0d40746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 --2- 172.21.15.7:0/2432919315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d4075820 0x7fc0d4073c70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 -- 172.21.15.7:0/2432919315 >> 172.21.15.7:0/2432919315 conn(0x7fc0d40fcb70 msgr2=0x7fc0d40fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 -- 172.21.15.7:0/2432919315 shutdown_connections 2024-03-18T01:37:44.058 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.056+0000 7fc0da200640 1 -- 172.21.15.7:0/2432919315 wait complete. 2024-03-18T01:37:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.058+0000 7fc0da200640 1 Processor -- start 2024-03-18T01:37:44.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.058+0000 7fc0da200640 1 -- start start 2024-03-18T01:37:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0da200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d4074240 0x7fc0d41999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0da200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d4075820 0x7fc0d4199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0da200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0d41094a0 0x7fc0d41a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:44.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0da200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d410f490 con 0x7fc0d41094a0 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0da200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d410f310 con 0x7fc0d4075820 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0da200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc0d410f610 con 0x7fc0d4074240 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d4074240 0x7fc0d41999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d4074240 0x7fc0d41999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53490/0 (socket says 172.21.15.7:53490) 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d4075820 0x7fc0d4199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d8e00640 1 -- 172.21.15.7:0/2902243854 learned_addr learned my addr 172.21.15.7:0/2902243854 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d9800640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0d41094a0 0x7fc0d41a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d8e00640 1 -- 172.21.15.7:0/2902243854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d4075820 msgr2=0x7fc0d4199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d8e00640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d4075820 0x7fc0d4199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d8e00640 1 -- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0d41094a0 msgr2=0x7fc0d41a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d8e00640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0d41094a0 0x7fc0d41a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.059+0000 7fc0d8e00640 1 -- 172.21.15.7:0/2902243854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d41a15d0 con 0x7fc0d4074240 2024-03-18T01:37:44.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.060+0000 7fc0d8e00640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d4074240 0x7fc0d41999b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc0c802c8f0 tx=0x7fc0c802c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:44.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.060+0000 7fc0d1600640 1 -- 172.21.15.7:0/2902243854 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8041070 con 0x7fc0d4074240 2024-03-18T01:37:44.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.060+0000 7fc0d1600640 1 -- 172.21.15.7:0/2902243854 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0c8004450 con 0x7fc0d4074240 2024-03-18T01:37:44.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.060+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0d41a1860 con 0x7fc0d4074240 2024-03-18T01:37:44.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.060+0000 7fc0d1600640 1 -- 172.21.15.7:0/2902243854 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8034440 con 0x7fc0d4074240 2024-03-18T01:37:44.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.060+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0d41a1d50 con 0x7fc0d4074240 2024-03-18T01:37:44.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.061+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0d41081d0 con 0x7fc0d4074240 2024-03-18T01:37:44.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.063+0000 7fc0d1600640 1 -- 172.21.15.7:0/2902243854 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fc0c8044070 con 0x7fc0d4074240 2024-03-18T01:37:44.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.066+0000 7fc0d1600640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc0bc0a28a0 0x7fc0bc0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.066+0000 7fc0d1600640 1 -- 172.21.15.7:0/2902243854 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc0c80f0990 con 0x7fc0d4074240 2024-03-18T01:37:44.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.066+0000 7fc0d3e00640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc0bc0a28a0 0x7fc0bc0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:44.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.067+0000 7fc0d3e00640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc0bc0a28a0 0x7fc0bc0a4d60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc0d419aec0 tx=0x7fc0c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:44.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.071+0000 7fc0d1600640 1 -- 172.21.15.7:0/2902243854 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc0c80bd020 con 0x7fc0d4074240 2024-03-18T01:37:44.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.278+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0d402d3d0 con 0x7fc0d4074240 2024-03-18T01:37:44.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.280+0000 7fc0d1600640 1 -- 172.21.15.7:0/2902243854 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7fc0c80c10c0 con 0x7fc0d4074240 2024-03-18T01:37:44.281 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:44.282 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc0bc0a28a0 msgr2=0x7fc0bc0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc0bc0a28a0 0x7fc0bc0a4d60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc0d419aec0 tx=0x7fc0c0005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d4074240 msgr2=0x7fc0d41999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d4074240 0x7fc0d41999b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc0c802c8f0 tx=0x7fc0c802c660 comp rx=0 tx=0).stop 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 shutdown_connections 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc0bc0a28a0 0x7fc0bc0a4d60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc0d41094a0 0x7fc0d41a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0d4075820 0x7fc0d4199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 --2- 172.21.15.7:0/2902243854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0d4074240 0x7fc0d41999b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 >> 172.21.15.7:0/2902243854 conn(0x7fc0d40fcb70 msgr2=0x7fc0d41053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:44.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 shutdown_connections 2024-03-18T01:37:44.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.283+0000 7fc0da200640 1 -- 172.21.15.7:0/2902243854 wait complete. 2024-03-18T01:37:44.304 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:37:44.304 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:37:44.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 -- 172.21.15.7:0/4135700955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 msgr2=0x7fcf88105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 --2- 172.21.15.7:0/4135700955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 0x7fcf88105090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009ca0 tx=0x7fcf7002c100 comp rx=0 tx=0).stop 2024-03-18T01:37:44.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 -- 172.21.15.7:0/4135700955 shutdown_connections 2024-03-18T01:37:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 --2- 172.21.15.7:0/4135700955 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf881055d0 0x7fcf8810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 --2- 172.21.15.7:0/4135700955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 0x7fcf88105090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 --2- 172.21.15.7:0/4135700955 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88103a20 0x7fcf88103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 -- 172.21.15.7:0/4135700955 >> 172.21.15.7:0/4135700955 conn(0x7fcf880fef90 msgr2=0x7fcf881013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 -- 172.21.15.7:0/4135700955 shutdown_connections 2024-03-18T01:37:44.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.527+0000 7fcf8e800640 1 -- 172.21.15.7:0/4135700955 wait complete. 2024-03-18T01:37:44.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.528+0000 7fcf8e800640 1 Processor -- start 2024-03-18T01:37:44.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.528+0000 7fcf8e800640 1 -- start start 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf8e800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88103a20 0x7fcf88199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf8e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 0x7fcf8819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf8e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf881055d0 0x7fcf881a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf8e800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8810f680 con 0x7fcf88103a20 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf8e800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8810f500 con 0x7fcf88104c10 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf8e800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8810f800 con 0x7fcf881055d0 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf87400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88103a20 0x7fcf88199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf7ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 0x7fcf8819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf87400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88103a20 0x7fcf88199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36752/0 (socket says 172.21.15.7:36752) 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf87400640 1 -- 172.21.15.7:0/3524715094 learned_addr learned my addr 172.21.15.7:0/3524715094 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.529+0000 7fcf87e00640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf881055d0 0x7fcf881a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf7ea00640 1 -- 172.21.15.7:0/3524715094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf881055d0 msgr2=0x7fcf881a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf7ea00640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf881055d0 0x7fcf881a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf7ea00640 1 -- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88103a20 msgr2=0x7fcf88199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf7ea00640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88103a20 0x7fcf88199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.532 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf7ea00640 1 -- 172.21.15.7:0/3524715094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf881a1950 con 0x7fcf88104c10 2024-03-18T01:37:44.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf7ea00640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 0x7fcf8819a280 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009bf0 tx=0x7fcf70035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:44.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf84c00640 1 -- 172.21.15.7:0/3524715094 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70039040 con 0x7fcf88104c10 2024-03-18T01:37:44.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf84c00640 1 -- 172.21.15.7:0/3524715094 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf70003970 con 0x7fcf88104c10 2024-03-18T01:37:44.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf881a1be0 con 0x7fcf88104c10 2024-03-18T01:37:44.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf84c00640 1 -- 172.21.15.7:0/3524715094 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70004280 con 0x7fcf88104c10 2024-03-18T01:37:44.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.530+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf881a20f0 con 0x7fcf88104c10 2024-03-18T01:37:44.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.532+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf88068ab0 con 0x7fcf88104c10 2024-03-18T01:37:44.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.533+0000 7fcf84c00640 1 -- 172.21.15.7:0/3524715094 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fcf70046030 con 0x7fcf88104c10 2024-03-18T01:37:44.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.535+0000 7fcf84c00640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:44.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.535+0000 7fcf84c00640 1 -- 172.21.15.7:0/3524715094 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcf700f09a0 con 0x7fcf88104c10 2024-03-18T01:37:44.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.535+0000 7fcf87400640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:44.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.536+0000 7fcf87400640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcf8819b210 tx=0x7fcf78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:44.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.539+0000 7fcf84c00640 1 -- 172.21.15.7:0/3524715094 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf7004d060 con 0x7fcf88104c10 2024-03-18T01:37:44.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:44 smithi007 ceph-mon[31996]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:44.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:44 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2902243854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:44.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:44 smithi046 ceph-mon[36108]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:44.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:44 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2902243854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:44.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.773+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcf880623e0 con 0x7fcf88104c10 2024-03-18T01:37:44.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.773+0000 7fcf84c00640 1 -- 172.21.15.7:0/3524715094 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fcf700bd680 con 0x7fcf88104c10 2024-03-18T01:37:44.774 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcf6c0a2880 msgr2=0x7fcf6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcf8819b210 tx=0x7fcf78005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 msgr2=0x7fcf8819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 0x7fcf8819a280 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009bf0 tx=0x7fcf70035d10 comp rx=0 tx=0).stop 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 shutdown_connections 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf881055d0 0x7fcf881a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf88104c10 0x7fcf8819a280 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcf6c0a2880 0x7fcf6c0a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 --2- 172.21.15.7:0/3524715094 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcf88103a20 0x7fcf88199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 >> 172.21.15.7:0/3524715094 conn(0x7fcf880fef90 msgr2=0x7fcf88107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:44.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.776+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 shutdown_connections 2024-03-18T01:37:44.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:44.777+0000 7fcf8e800640 1 -- 172.21.15.7:0/3524715094 wait complete. 2024-03-18T01:37:44.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:37:44.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:44 smithi203 ceph-mon[36345]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:44.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:44 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2902243854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:45.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.008+0000 7efd70600640 1 -- 172.21.15.7:0/1463870954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 msgr2=0x7efd68109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.008+0000 7efd70600640 1 --2- 172.21.15.7:0/1463870954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 0x7efd68109930 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efd60009860 tx=0x7efd6002bb70 comp rx=0 tx=0).stop 2024-03-18T01:37:45.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.008+0000 7efd70600640 1 -- 172.21.15.7:0/1463870954 shutdown_connections 2024-03-18T01:37:45.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.008+0000 7efd70600640 1 --2- 172.21.15.7:0/1463870954 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 0x7efd68109930 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.008+0000 7efd70600640 1 --2- 172.21.15.7:0/1463870954 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd68104bb0 0x7efd68106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.008+0000 7efd70600640 1 --2- 172.21.15.7:0/1463870954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd68102280 0x7efd68104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.008+0000 7efd70600640 1 -- 172.21.15.7:0/1463870954 >> 172.21.15.7:0/1463870954 conn(0x7efd680fbc70 msgr2=0x7efd680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:45.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.009+0000 7efd70600640 1 -- 172.21.15.7:0/1463870954 shutdown_connections 2024-03-18T01:37:45.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.009+0000 7efd70600640 1 -- 172.21.15.7:0/1463870954 wait complete. 2024-03-18T01:37:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.010+0000 7efd70600640 1 Processor -- start 2024-03-18T01:37:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.010+0000 7efd70600640 1 -- start start 2024-03-18T01:37:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.010+0000 7efd70600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd68102280 0x7efd68197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:45.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.010+0000 7efd70600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd68104bb0 0x7efd68198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd70600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 0x7efd6819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd70600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efd6810d480 con 0x7efd68102280 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd70600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efd6810d300 con 0x7efd68104bb0 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd70600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efd6810d600 con 0x7efd681074e0 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6dc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd68102280 0x7efd68197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6dc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd68102280 0x7efd68197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36764/0 (socket says 172.21.15.7:36764) 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6dc00640 1 -- 172.21.15.7:0/2252007620 learned_addr learned my addr 172.21.15.7:0/2252007620 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6d200640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd68104bb0 0x7efd68198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6e600640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 0x7efd6819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6dc00640 1 -- 172.21.15.7:0/2252007620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 msgr2=0x7efd6819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6dc00640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 0x7efd6819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6dc00640 1 -- 172.21.15.7:0/2252007620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd68104bb0 msgr2=0x7efd68198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6dc00640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd68104bb0 0x7efd68198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6dc00640 1 -- 172.21.15.7:0/2252007620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6819f6d0 con 0x7efd68102280 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6e600640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 0x7efd6819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.011+0000 7efd6d200640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd68104bb0 0x7efd68198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.012+0000 7efd6dc00640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd68102280 0x7efd68197b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00c420 tx=0x7efd5c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.012+0000 7efd66000640 1 -- 172.21.15.7:0/2252007620 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c003970 con 0x7efd68102280 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.012+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd6819f9c0 con 0x7efd68102280 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.012+0000 7efd66000640 1 -- 172.21.15.7:0/2252007620 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd5c004020 con 0x7efd68102280 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.012+0000 7efd66000640 1 -- 172.21.15.7:0/2252007620 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c00bad0 con 0x7efd68102280 2024-03-18T01:37:45.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.012+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6819ff60 con 0x7efd68102280 2024-03-18T01:37:45.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.013+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd68068ab0 con 0x7efd68102280 2024-03-18T01:37:45.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.016+0000 7efd66000640 1 -- 172.21.15.7:0/2252007620 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7efd5c01d020 con 0x7efd68102280 2024-03-18T01:37:45.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.017+0000 7efd66000640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd500a2880 0x7efd500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:45.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.017+0000 7efd66000640 1 -- 172.21.15.7:0/2252007620 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efd5c0cf1f0 con 0x7efd68102280 2024-03-18T01:37:45.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.017+0000 7efd6d200640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd500a2880 0x7efd500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:45.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.018+0000 7efd6d200640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd500a2880 0x7efd500a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efd68198fc0 tx=0x7efd5400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:45.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.020+0000 7efd66000640 1 -- 172.21.15.7:0/2252007620 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd5c09bed0 con 0x7efd68102280 2024-03-18T01:37:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.242+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efd6802d1b0 con 0x7efd68102280 2024-03-18T01:37:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.243+0000 7efd66000640 1 -- 172.21.15.7:0/2252007620 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7efd6802d1b0 con 0x7efd68102280 2024-03-18T01:37:45.244 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:37:45.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd500a2880 msgr2=0x7efd500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd500a2880 0x7efd500a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efd68198fc0 tx=0x7efd5400b410 comp rx=0 tx=0).stop 2024-03-18T01:37:45.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd68102280 msgr2=0x7efd68197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd68102280 0x7efd68197b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00c420 tx=0x7efd5c009f90 comp rx=0 tx=0).stop 2024-03-18T01:37:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 shutdown_connections 2024-03-18T01:37:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd681074e0 0x7efd6819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd68104bb0 0x7efd68198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd500a2880 0x7efd500a4d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 --2- 172.21.15.7:0/2252007620 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd68102280 0x7efd68197b00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 >> 172.21.15.7:0/2252007620 conn(0x7efd680fbc70 msgr2=0x7efd68102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:45.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.246+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 shutdown_connections 2024-03-18T01:37:45.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.247+0000 7efd70600640 1 -- 172.21.15.7:0/2252007620 wait complete. 2024-03-18T01:37:45.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:37:45.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 -- 172.21.15.7:0/3140360219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8381031f0 msgr2=0x7ff838109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 --2- 172.21.15.7:0/3140360219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8381031f0 0x7ff838109a80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff830009ca0 tx=0x7ff83002c0f0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 -- 172.21.15.7:0/3140360219 shutdown_connections 2024-03-18T01:37:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 --2- 172.21.15.7:0/3140360219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8381031f0 0x7ff838109a80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 --2- 172.21.15.7:0/3140360219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838102830 0x7ff838102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 --2- 172.21.15.7:0/3140360219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff838101640 0x7ff838101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 -- 172.21.15.7:0/3140360219 >> 172.21.15.7:0/3140360219 conn(0x7ff8380fcbf0 msgr2=0x7ff8380ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 -- 172.21.15.7:0/3140360219 shutdown_connections 2024-03-18T01:37:45.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.491+0000 7ff83e000640 1 -- 172.21.15.7:0/3140360219 wait complete. 2024-03-18T01:37:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.493+0000 7ff83e000640 1 Processor -- start 2024-03-18T01:37:45.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.493+0000 7ff83e000640 1 -- start start 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.493+0000 7ff83e000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838101640 0x7ff838199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.493+0000 7ff83e000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff838102830 0x7ff83819a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8381031f0 0x7ff8381a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83e000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff83810d2c0 con 0x7ff838101640 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83e000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff83810d140 con 0x7ff8381031f0 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83e000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff83810d440 con 0x7ff838102830 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83cc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838101640 0x7ff838199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff837e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff838102830 0x7ff83819a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83cc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838101640 0x7ff838199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36780/0 (socket says 172.21.15.7:36780) 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83cc00640 1 -- 172.21.15.7:0/1614650601 learned_addr learned my addr 172.21.15.7:0/1614650601 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:45.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83d600640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8381031f0 0x7ff8381a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83cc00640 1 -- 172.21.15.7:0/1614650601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff838102830 msgr2=0x7ff83819a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83cc00640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff838102830 0x7ff83819a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83cc00640 1 -- 172.21.15.7:0/1614650601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8381031f0 msgr2=0x7ff8381a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83cc00640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8381031f0 0x7ff8381a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83cc00640 1 -- 172.21.15.7:0/1614650601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8381a16f0 con 0x7ff838101640 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.494+0000 7ff83d600640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8381031f0 0x7ff8381a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.495+0000 7ff83cc00640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838101640 0x7ff838199ad0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00e9c0 tx=0x7ff82c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.495+0000 7ff835600640 1 -- 172.21.15.7:0/1614650601 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c00cd80 con 0x7ff838101640 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.495+0000 7ff835600640 1 -- 172.21.15.7:0/1614650601 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff82c0035f0 con 0x7ff838101640 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.495+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8381a19e0 con 0x7ff838101640 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.495+0000 7ff835600640 1 -- 172.21.15.7:0/1614650601 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c010690 con 0x7ff838101640 2024-03-18T01:37:45.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.495+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8381a1f30 con 0x7ff838101640 2024-03-18T01:37:45.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.498+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8381a21c0 con 0x7ff838101640 2024-03-18T01:37:45.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.500+0000 7ff835600640 1 -- 172.21.15.7:0/1614650601 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7ff82c0037a0 con 0x7ff838101640 2024-03-18T01:37:45.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.501+0000 7ff835600640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff8200a2880 0x7ff8200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:45.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.502+0000 7ff835600640 1 -- 172.21.15.7:0/1614650601 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff82c0cf0f0 con 0x7ff838101640 2024-03-18T01:37:45.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.502+0000 7ff837e00640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff8200a2880 0x7ff8200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:45.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.503+0000 7ff837e00640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff8200a2880 0x7ff8200a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff8240095f0 tx=0x7ff824005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:45.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.508+0000 7ff835600640 1 -- 172.21.15.7:0/1614650601 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff82c09bdd0 con 0x7ff838101640 2024-03-18T01:37:45.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:45 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3524715094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:45 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:37:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:45 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2252007620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:45.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:45 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3524715094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:45.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:45 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:37:45.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:45 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2252007620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:45.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.771+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff8381a2390 con 0x7ff838101640 2024-03-18T01:37:45.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.771+0000 7ff835600640 1 -- 172.21.15.7:0/1614650601 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7ff82c09b6d0 con 0x7ff838101640 2024-03-18T01:37:45.772 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:37:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff8200a2880 msgr2=0x7ff8200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff8200a2880 0x7ff8200a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff8240095f0 tx=0x7ff824005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838101640 msgr2=0x7ff838199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838101640 0x7ff838199ad0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00e9c0 tx=0x7ff82c00ee80 comp rx=0 tx=0).stop 2024-03-18T01:37:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 shutdown_connections 2024-03-18T01:37:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8381031f0 0x7ff8381a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff838102830 0x7ff83819a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff8200a2880 0x7ff8200a4d40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 --2- 172.21.15.7:0/1614650601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff838101640 0x7ff838199ad0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:45.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 >> 172.21.15.7:0/1614650601 conn(0x7ff8380fcbf0 msgr2=0x7ff838073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:45.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 shutdown_connections 2024-03-18T01:37:45.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:45.774+0000 7ff83e000640 1 -- 172.21.15.7:0/1614650601 wait complete. 2024-03-18T01:37:45.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:37:45.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:45 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3524715094' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:45.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:45 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:37:45.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:45 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2252007620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:46.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 -- 172.21.15.7:0/4266529383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 msgr2=0x7f1b84105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 --2- 172.21.15.7:0/4266529383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 0x7f1b84105090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009900 tx=0x7f1b7002bc30 comp rx=0 tx=0).stop 2024-03-18T01:37:46.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 -- 172.21.15.7:0/4266529383 shutdown_connections 2024-03-18T01:37:46.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 --2- 172.21.15.7:0/4266529383 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b841055d0 0x7f1b8410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 --2- 172.21.15.7:0/4266529383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 0x7f1b84105090 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 --2- 172.21.15.7:0/4266529383 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b84103a20 0x7f1b84103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 -- 172.21.15.7:0/4266529383 >> 172.21.15.7:0/4266529383 conn(0x7f1b840fef90 msgr2=0x7f1b841013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:46.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 -- 172.21.15.7:0/4266529383 shutdown_connections 2024-03-18T01:37:46.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.008+0000 7f1b8c000640 1 -- 172.21.15.7:0/4266529383 wait complete. 2024-03-18T01:37:46.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.009+0000 7f1b8c000640 1 Processor -- start 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.009+0000 7f1b8c000640 1 -- start start 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.009+0000 7f1b8c000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b84103a20 0x7f1b84199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.009+0000 7f1b8c000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 0x7f1b8419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.009+0000 7f1b8c000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b841055d0 0x7f1b841a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b8c000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8410f6d0 con 0x7f1b84103a20 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b8c000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8410f550 con 0x7f1b841055d0 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b8c000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8410f850 con 0x7f1b84104c10 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b88c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 0x7f1b8419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b89600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b84103a20 0x7f1b84199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b8a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b841055d0 0x7f1b841a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b88c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 0x7f1b8419a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59304/0 (socket says 172.21.15.7:59304) 2024-03-18T01:37:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b88c00640 1 -- 172.21.15.7:0/3784320367 learned_addr learned my addr 172.21.15.7:0/3784320367 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b88c00640 1 -- 172.21.15.7:0/3784320367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b841055d0 msgr2=0x7f1b841a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b88c00640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b841055d0 0x7f1b841a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b88c00640 1 -- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b84103a20 msgr2=0x7f1b84199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b88c00640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b84103a20 0x7f1b84199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b88c00640 1 -- 172.21.15.7:0/3784320367 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b841a1920 con 0x7f1b84104c10 2024-03-18T01:37:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.010+0000 7f1b8a000640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b841055d0 0x7f1b841a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:46.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.011+0000 7f1b88c00640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 0x7f1b8419a250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1b70000c00 tx=0x7f1b7002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:46.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.011+0000 7f1b82000640 1 -- 172.21.15.7:0/3784320367 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70043020 con 0x7f1b84104c10 2024-03-18T01:37:46.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.011+0000 7f1b82000640 1 -- 172.21.15.7:0/3784320367 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b700054e0 con 0x7f1b84104c10 2024-03-18T01:37:46.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.011+0000 7f1b82000640 1 -- 172.21.15.7:0/3784320367 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70003c40 con 0x7f1b84104c10 2024-03-18T01:37:46.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.011+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b841a1bb0 con 0x7f1b84104c10 2024-03-18T01:37:46.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.011+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b841a20b0 con 0x7f1b84104c10 2024-03-18T01:37:46.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.012+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b8410ad40 con 0x7f1b84104c10 2024-03-18T01:37:46.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.014+0000 7f1b82000640 1 -- 172.21.15.7:0/3784320367 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f1b70046020 con 0x7f1b84104c10 2024-03-18T01:37:46.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.017+0000 7f1b82000640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b6c0a2880 0x7f1b6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.017+0000 7f1b82000640 1 -- 172.21.15.7:0/3784320367 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1b700f0e70 con 0x7f1b84104c10 2024-03-18T01:37:46.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.017+0000 7f1b89600640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b6c0a2880 0x7f1b6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.018+0000 7f1b89600640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b6c0a2880 0x7f1b6c0a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b8419b1e0 tx=0x7f1b78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:46.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.020+0000 7f1b82000640 1 -- 172.21.15.7:0/3784320367 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1b7004d020 con 0x7f1b84104c10 2024-03-18T01:37:46.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.220+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1b84062480 con 0x7f1b84104c10 2024-03-18T01:37:46.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.220+0000 7f1b82000640 1 -- 172.21.15.7:0/3784320367 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f1b700bdb50 con 0x7f1b84104c10 2024-03-18T01:37:46.222 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b6c0a2880 msgr2=0x7f1b6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b6c0a2880 0x7f1b6c0a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b8419b1e0 tx=0x7f1b78005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 msgr2=0x7f1b8419a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 0x7f1b8419a250 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1b70000c00 tx=0x7f1b7002c650 comp rx=0 tx=0).stop 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 shutdown_connections 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b841055d0 0x7f1b841a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b84104c10 0x7f1b8419a250 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b6c0a2880 0x7f1b6c0a4d40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 --2- 172.21.15.7:0/3784320367 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1b84103a20 0x7f1b84199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.223+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 >> 172.21.15.7:0/3784320367 conn(0x7f1b840fef90 msgr2=0x7f1b84107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.224+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 shutdown_connections 2024-03-18T01:37:46.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.224+0000 7f1b8c000640 1 -- 172.21.15.7:0/3784320367 wait complete. 2024-03-18T01:37:46.243 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.440+0000 7fd779e00640 1 -- 172.21.15.7:0/1596719615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774102130 msgr2=0x7fd774104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.440+0000 7fd779e00640 1 --2- 172.21.15.7:0/1596719615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774102130 0x7fd774104520 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd764009ca0 tx=0x7fd76402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.440+0000 7fd779e00640 1 -- 172.21.15.7:0/1596719615 shutdown_connections 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.440+0000 7fd779e00640 1 --2- 172.21.15.7:0/1596719615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd774107390 0x7fd774109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.440+0000 7fd779e00640 1 --2- 172.21.15.7:0/1596719615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd774104a60 0x7fd774106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.440+0000 7fd779e00640 1 --2- 172.21.15.7:0/1596719615 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774102130 0x7fd774104520 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.440+0000 7fd779e00640 1 -- 172.21.15.7:0/1596719615 >> 172.21.15.7:0/1596719615 conn(0x7fd7740fbaa0 msgr2=0x7fd7740fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.441+0000 7fd779e00640 1 -- 172.21.15.7:0/1596719615 shutdown_connections 2024-03-18T01:37:46.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.441+0000 7fd779e00640 1 -- 172.21.15.7:0/1596719615 wait complete. 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.442+0000 7fd779e00640 1 Processor -- start 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.442+0000 7fd779e00640 1 -- start start 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.442+0000 7fd779e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd774102130 0x7fd774199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.442+0000 7fd779e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774104a60 0x7fd774199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.442+0000 7fd779e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd774107390 0x7fd7741a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd779e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd77410d270 con 0x7fd774104a60 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd779e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd77410d0f0 con 0x7fd774107390 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd779e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd77410d3f0 con 0x7fd774102130 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd778a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd774102130 0x7fd774199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd76be00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774104a60 0x7fd774199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd778a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd774102130 0x7fd774199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59312/0 (socket says 172.21.15.7:59312) 2024-03-18T01:37:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd778a00640 1 -- 172.21.15.7:0/1221338167 learned_addr learned my addr 172.21.15.7:0/1221338167 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:46.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd779400640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd774107390 0x7fd7741a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd778a00640 1 -- 172.21.15.7:0/1221338167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd774107390 msgr2=0x7fd7741a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd778a00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd774107390 0x7fd7741a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd778a00640 1 -- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774104a60 msgr2=0x7fd774199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd778a00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774104a60 0x7fd774199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd778a00640 1 -- 172.21.15.7:0/1221338167 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7741a1630 con 0x7fd774102130 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.443+0000 7fd76be00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774104a60 0x7fd774199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.444+0000 7fd778a00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd774102130 0x7fd774199a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd7640325f0 tx=0x7fd764032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.444+0000 7fd772000640 1 -- 172.21.15.7:0/1221338167 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd764041020 con 0x7fd774102130 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.444+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7741a18c0 con 0x7fd774102130 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.444+0000 7fd772000640 1 -- 172.21.15.7:0/1221338167 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd764034cd0 con 0x7fd774102130 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.444+0000 7fd772000640 1 -- 172.21.15.7:0/1221338167 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd764039040 con 0x7fd774102130 2024-03-18T01:37:46.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.444+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7741a1dd0 con 0x7fd774102130 2024-03-18T01:37:46.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.446+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd774108e80 con 0x7fd774102130 2024-03-18T01:37:46.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.447+0000 7fd772000640 1 -- 172.21.15.7:0/1221338167 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fd764044020 con 0x7fd774102130 2024-03-18T01:37:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.449+0000 7fd772000640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7580a28a0 0x7fd7580a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.449+0000 7fd772000640 1 -- 172.21.15.7:0/1221338167 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd7640f0c70 con 0x7fd774102130 2024-03-18T01:37:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.449+0000 7fd76be00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7580a28a0 0x7fd7580a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.450+0000 7fd76be00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7580a28a0 0x7fd7580a4d60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd77419af20 tx=0x7fd75c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:46.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.453+0000 7fd772000640 1 -- 172.21.15.7:0/1221338167 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd76404b020 con 0x7fd774102130 2024-03-18T01:37:46.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.707+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd77402d1b0 con 0x7fd774102130 2024-03-18T01:37:46.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.708+0000 7fd772000640 1 -- 172.21.15.7:0/1221338167 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11918 (secure 0 0 0) 0x7fd7640bda40 con 0x7fd774102130 2024-03-18T01:37:46.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:46.709 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:35:30.634148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24542,24523,24544,24596,24562]},"id":1}]} 2024-03-18T01:37:46.709 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2024-03-18T01:37:46.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7580a28a0 msgr2=0x7fd7580a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7580a28a0 0x7fd7580a4d60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd77419af20 tx=0x7fd75c005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:46.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd774102130 msgr2=0x7fd774199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd774102130 0x7fd774199a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd7640325f0 tx=0x7fd764032ec0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 shutdown_connections 2024-03-18T01:37:46.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd774107390 0x7fd7741a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd7580a28a0 0x7fd7580a4d60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd774104a60 0x7fd774199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 --2- 172.21.15.7:0/1221338167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd774102130 0x7fd774199a10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.710+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 >> 172.21.15.7:0/1221338167 conn(0x7fd7740fbaa0 msgr2=0x7fd7741017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:46.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.711+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 shutdown_connections 2024-03-18T01:37:46.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.711+0000 7fd779e00640 1 -- 172.21.15.7:0/1221338167 wait complete. 2024-03-18T01:37:46.732 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1614650601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:46 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3784320367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:46.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T01:37:46.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1614650601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:46.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:46.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:46 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3784320367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.929+0000 7f6256600640 1 -- 172.21.15.7:0/3276646065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6250103a00 msgr2=0x7f6250103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.929+0000 7f6256600640 1 --2- 172.21.15.7:0/3276646065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6250103a00 0x7f6250103e20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6240009900 tx=0x7f624002bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.930+0000 7f6256600640 1 -- 172.21.15.7:0/3276646065 shutdown_connections 2024-03-18T01:37:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.930+0000 7f6256600640 1 --2- 172.21.15.7:0/3276646065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62501055b0 0x7f625010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.930+0000 7f6256600640 1 --2- 172.21.15.7:0/3276646065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6250104bf0 0x7f6250105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.930+0000 7f6256600640 1 --2- 172.21.15.7:0/3276646065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6250103a00 0x7f6250103e20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.930+0000 7f6256600640 1 -- 172.21.15.7:0/3276646065 >> 172.21.15.7:0/3276646065 conn(0x7f62500fef90 msgr2=0x7f62501013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.930+0000 7f6256600640 1 -- 172.21.15.7:0/3276646065 shutdown_connections 2024-03-18T01:37:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.930+0000 7f6256600640 1 -- 172.21.15.7:0/3276646065 wait complete. 2024-03-18T01:37:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.931+0000 7f6256600640 1 Processor -- start 2024-03-18T01:37:46.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.931+0000 7f6256600640 1 -- start start 2024-03-18T01:37:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f6256600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6250103a00 0x7f6250199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f6256600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6250104bf0 0x7f625019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f6256600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62501055b0 0x7f62501a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f6256600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f625010f6a0 con 0x7f6250104bf0 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f6256600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f625010f520 con 0x7f62501055b0 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f6256600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f625010f820 con 0x7f6250103a00 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f624f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6250103a00 0x7f6250199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f624f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6250103a00 0x7f6250199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59322/0 (socket says 172.21.15.7:59322) 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f624f400640 1 -- 172.21.15.7:0/1328369417 learned_addr learned my addr 172.21.15.7:0/1328369417 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f6247e00640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6250104bf0 0x7f625019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.932+0000 7f624fe00640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62501055b0 0x7f62501a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624f400640 1 -- 172.21.15.7:0/1328369417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62501055b0 msgr2=0x7f62501a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624f400640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62501055b0 0x7f62501a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624f400640 1 -- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6250104bf0 msgr2=0x7f625019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624f400640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6250104bf0 0x7f625019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624f400640 1 -- 172.21.15.7:0/1328369417 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62501a1900 con 0x7f6250103a00 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624fe00640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62501055b0 0x7f62501a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624f400640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6250103a00 0x7f6250199cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f624002efd0 tx=0x7f624002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624cc00640 1 -- 172.21.15.7:0/1328369417 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240041070 con 0x7f6250103a00 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624cc00640 1 -- 172.21.15.7:0/1328369417 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6240004450 con 0x7f6250103a00 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62501a1b90 con 0x7f6250103a00 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f624cc00640 1 -- 172.21.15.7:0/1328369417 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240003e50 con 0x7f6250103a00 2024-03-18T01:37:46.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.933+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62501a20a0 con 0x7f6250103a00 2024-03-18T01:37:46.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.935+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f625010acc0 con 0x7f6250103a00 2024-03-18T01:37:46.937 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.937+0000 7f624cc00640 1 -- 172.21.15.7:0/1328369417 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f6240044070 con 0x7f6250103a00 2024-03-18T01:37:46.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.938+0000 7f624cc00640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62340a2a20 0x7f62340a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:46.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.938+0000 7f624cc00640 1 -- 172.21.15.7:0/1328369417 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f62400f09e0 con 0x7f6250103a00 2024-03-18T01:37:46.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.938+0000 7f6247e00640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62340a2a20 0x7f62340a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:46.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.939+0000 7f6247e00640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62340a2a20 0x7f62340a4ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f625019b1c0 tx=0x7f6238005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:46.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:46.942+0000 7f624cc00640 1 -- 172.21.15.7:0/1328369417 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62400bd6c0 con 0x7f6250103a00 2024-03-18T01:37:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T01:37:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1614650601' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:47.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:46 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3784320367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:47.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.179+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62500623e0 con 0x7f6250103a00 2024-03-18T01:37:47.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.181+0000 7f624cc00640 1 -- 172.21.15.7:0/1328369417 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7f62400bcfc0 con 0x7f6250103a00 2024-03-18T01:37:47.182 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:47.183 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62340a2a20 msgr2=0x7f62340a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62340a2a20 0x7f62340a4ee0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f625019b1c0 tx=0x7f6238005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6250103a00 msgr2=0x7f6250199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6250103a00 0x7f6250199cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f624002efd0 tx=0x7f624002c660 comp rx=0 tx=0).stop 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 shutdown_connections 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62501055b0 0x7f62501a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62340a2a20 0x7f62340a4ee0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6250104bf0 0x7f625019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 --2- 172.21.15.7:0/1328369417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6250103a00 0x7f6250199cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 >> 172.21.15.7:0/1328369417 conn(0x7f62500fef90 msgr2=0x7f6250107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:47.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 shutdown_connections 2024-03-18T01:37:47.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.184+0000 7f6256600640 1 -- 172.21.15.7:0/1328369417 wait complete. 2024-03-18T01:37:47.204 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:37:47.205 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:37:47.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 -- 172.21.15.7:0/4230129186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 msgr2=0x7f7b38104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/4230129186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 0x7f7b38104eb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009900 tx=0x7f7b2402bc30 comp rx=0 tx=0).stop 2024-03-18T01:37:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 -- 172.21.15.7:0/4230129186 shutdown_connections 2024-03-18T01:37:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/4230129186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b381053f0 0x7f7b3810bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/4230129186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 0x7f7b38104eb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/4230129186 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b38103840 0x7f7b38103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 -- 172.21.15.7:0/4230129186 >> 172.21.15.7:0/4230129186 conn(0x7f7b380fedb0 msgr2=0x7f7b381011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 -- 172.21.15.7:0/4230129186 shutdown_connections 2024-03-18T01:37:47.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.428+0000 7f7b3ec00640 1 -- 172.21.15.7:0/4230129186 wait complete. 2024-03-18T01:37:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.429+0000 7f7b3ec00640 1 Processor -- start 2024-03-18T01:37:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.429+0000 7f7b3ec00640 1 -- start start 2024-03-18T01:37:47.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ec00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b38103840 0x7f7b38199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ec00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 0x7f7b3819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ec00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b381053f0 0x7f7b381a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ec00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3810f4a0 con 0x7f7b38103840 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ec00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3810f320 con 0x7f7b38104a30 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ec00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3810f620 con 0x7f7b381053f0 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3d800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b38103840 0x7f7b38199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 0x7f7b3819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 0x7f7b3819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34650/0 (socket says 172.21.15.7:34650) 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3d800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b38103840 0x7f7b38199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48192/0 (socket says 172.21.15.7:48192) 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3ce00640 1 -- 172.21.15.7:0/386684088 learned_addr learned my addr 172.21.15.7:0/386684088 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3e200640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b381053f0 0x7f7b381a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3d800640 1 -- 172.21.15.7:0/386684088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b381053f0 msgr2=0x7f7b381a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3d800640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b381053f0 0x7f7b381a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3d800640 1 -- 172.21.15.7:0/386684088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 msgr2=0x7f7b3819a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3d800640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 0x7f7b3819a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.430+0000 7f7b3d800640 1 -- 172.21.15.7:0/386684088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b381a1730 con 0x7f7b38103840 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.431+0000 7f7b3ce00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 0x7f7b3819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.431+0000 7f7b3d800640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b38103840 0x7f7b38199b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00e9c0 tx=0x7f7b2c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.431+0000 7f7b36000640 1 -- 172.21.15.7:0/386684088 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c00ccc0 con 0x7f7b38103840 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.431+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b381a1a20 con 0x7f7b38103840 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.431+0000 7f7b36000640 1 -- 172.21.15.7:0/386684088 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b2c0035f0 con 0x7f7b38103840 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.431+0000 7f7b36000640 1 -- 172.21.15.7:0/386684088 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c00ce60 con 0x7f7b38103840 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.431+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b381a1f40 con 0x7f7b38103840 2024-03-18T01:37:47.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.433+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3804f540 con 0x7f7b38103840 2024-03-18T01:37:47.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.436+0000 7f7b36000640 1 -- 172.21.15.7:0/386684088 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f7b2c0037a0 con 0x7f7b38103840 2024-03-18T01:37:47.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.438+0000 7f7b36000640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7b200a2800 0x7f7b200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:47.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.438+0000 7f7b36000640 1 -- 172.21.15.7:0/386684088 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7b2c0cf240 con 0x7f7b38103840 2024-03-18T01:37:47.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.438+0000 7f7b3ce00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7b200a2800 0x7f7b200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:47.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.439+0000 7f7b3ce00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7b200a2800 0x7f7b200a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7b24000c00 tx=0x7f7b2402c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:47.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.445+0000 7f7b36000640 1 -- 172.21.15.7:0/386684088 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7b2c09bf20 con 0x7f7b38103840 2024-03-18T01:37:47.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.652+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7b380623e0 con 0x7f7b38103840 2024-03-18T01:37:47.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.653+0000 7f7b36000640 1 -- 172.21.15.7:0/386684088 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f7b2c09b820 con 0x7f7b38103840 2024-03-18T01:37:47.654 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7b200a2800 msgr2=0x7f7b200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7b200a2800 0x7f7b200a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7b24000c00 tx=0x7f7b2402c880 comp rx=0 tx=0).stop 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b38103840 msgr2=0x7f7b38199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b38103840 0x7f7b38199b10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c00e9c0 tx=0x7f7b2c00ee80 comp rx=0 tx=0).stop 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 shutdown_connections 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b381053f0 0x7f7b381a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b38104a30 0x7f7b3819a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7b200a2800 0x7f7b200a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 --2- 172.21.15.7:0/386684088 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7b38103840 0x7f7b38199b10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.655+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 >> 172.21.15.7:0/386684088 conn(0x7f7b380fedb0 msgr2=0x7f7b38107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.656+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 shutdown_connections 2024-03-18T01:37:47.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.656+0000 7f7b3ec00640 1 -- 172.21.15.7:0/386684088 wait complete. 2024-03-18T01:37:47.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:37:47.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:47 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1221338167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:47.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:47 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1328369417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:47.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:47 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:47 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:47 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:47.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 -- 172.21.15.7:0/2637564169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 msgr2=0x7fe428103e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 --2- 172.21.15.7:0/2637564169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 0x7fe428103e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe41c009900 tx=0x7fe41c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:47.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 -- 172.21.15.7:0/2637564169 shutdown_connections 2024-03-18T01:37:47.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 --2- 172.21.15.7:0/2637564169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4281055c0 0x7fe42810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 --2- 172.21.15.7:0/2637564169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe428104c00 0x7fe428105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 --2- 172.21.15.7:0/2637564169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 0x7fe428103e30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 -- 172.21.15.7:0/2637564169 >> 172.21.15.7:0/2637564169 conn(0x7fe4280fefa0 msgr2=0x7fe4281013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:47.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 -- 172.21.15.7:0/2637564169 shutdown_connections 2024-03-18T01:37:47.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.890+0000 7fe42f800640 1 -- 172.21.15.7:0/2637564169 wait complete. 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.891+0000 7fe42f800640 1 Processor -- start 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.891+0000 7fe42f800640 1 -- start start 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.891+0000 7fe42f800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 0x7fe428199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.891+0000 7fe42f800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe428104c00 0x7fe42819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42f800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4281055c0 0x7fe4281a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42f800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe42810f6b0 con 0x7fe428104c00 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42f800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe42810f530 con 0x7fe4281055c0 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42f800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe42810f830 con 0x7fe428103a10 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 0x7fe428199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 0x7fe428199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59356/0 (socket says 172.21.15.7:59356) 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42ce00640 1 -- 172.21.15.7:0/3088259342 learned_addr learned my addr 172.21.15.7:0/3088259342 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe427e00640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe428104c00 0x7fe42819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:47.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42d800640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4281055c0 0x7fe4281a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42ce00640 1 -- 172.21.15.7:0/3088259342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4281055c0 msgr2=0x7fe4281a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42ce00640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4281055c0 0x7fe4281a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42ce00640 1 -- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe428104c00 msgr2=0x7fe42819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42ce00640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe428104c00 0x7fe42819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.892+0000 7fe42ce00640 1 -- 172.21.15.7:0/3088259342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4281a1950 con 0x7fe428103a10 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.893+0000 7fe42ce00640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 0x7fe428199d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe41c02efd0 tx=0x7fe41c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.893+0000 7fe425600640 1 -- 172.21.15.7:0/3088259342 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c041070 con 0x7fe428103a10 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.893+0000 7fe425600640 1 -- 172.21.15.7:0/3088259342 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c0044d0 con 0x7fe428103a10 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.893+0000 7fe425600640 1 -- 172.21.15.7:0/3088259342 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c034440 con 0x7fe428103a10 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.893+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4281a1be0 con 0x7fe428103a10 2024-03-18T01:37:47.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.893+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4281a2050 con 0x7fe428103a10 2024-03-18T01:37:47.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.894+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42810ad30 con 0x7fe428103a10 2024-03-18T01:37:47.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.896+0000 7fe425600640 1 -- 172.21.15.7:0/3088259342 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fe41c044070 con 0x7fe428103a10 2024-03-18T01:37:47.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.899+0000 7fe425600640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe4100a2880 0x7fe4100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:47.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.899+0000 7fe425600640 1 -- 172.21.15.7:0/3088259342 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe41c0f0cd0 con 0x7fe428103a10 2024-03-18T01:37:47.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.899+0000 7fe427e00640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe4100a2880 0x7fe4100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:47.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.900+0000 7fe427e00640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe4100a2880 0x7fe4100a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe414009bd0 tx=0x7fe414005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:47.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:47.900+0000 7fe425600640 1 -- 172.21.15.7:0/3088259342 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe41c0bd360 con 0x7fe428103a10 2024-03-18T01:37:48.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:47 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1221338167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:48.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:47 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1328369417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:48.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:47 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:48.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:47 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:48.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:47 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:48.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:47 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1221338167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:48.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:47 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1328369417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:48.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:47 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:48.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:47 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:48.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:47 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.169+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe42802d440 con 0x7fe428103a10 2024-03-18T01:37:48.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.169+0000 7fe425600640 1 -- 172.21.15.7:0/3088259342 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fe41c0c1400 con 0x7fe428103a10 2024-03-18T01:37:48.171 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:37:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe4100a2880 msgr2=0x7fe4100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe4100a2880 0x7fe4100a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe414009bd0 tx=0x7fe414005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 msgr2=0x7fe428199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 0x7fe428199d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe41c02efd0 tx=0x7fe41c005d50 comp rx=0 tx=0).stop 2024-03-18T01:37:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 shutdown_connections 2024-03-18T01:37:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe4281055c0 0x7fe4281a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe4100a2880 0x7fe4100a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe428104c00 0x7fe42819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 --2- 172.21.15.7:0/3088259342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe428103a10 0x7fe428199d40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 >> 172.21.15.7:0/3088259342 conn(0x7fe4280fefa0 msgr2=0x7fe428107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 shutdown_connections 2024-03-18T01:37:48.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.172+0000 7fe42f800640 1 -- 172.21.15.7:0/3088259342 wait complete. 2024-03-18T01:37:48.196 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:37:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.393+0000 7f56ab200640 1 -- 172.21.15.7:0/642399821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ac104a30 msgr2=0x7f56ac104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.393+0000 7f56ab200640 1 --2- 172.21.15.7:0/642399821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ac104a30 0x7f56ac104eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5698009900 tx=0x7f569802bc30 comp rx=0 tx=0).stop 2024-03-18T01:37:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.393+0000 7f56ab200640 1 -- 172.21.15.7:0/642399821 shutdown_connections 2024-03-18T01:37:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.393+0000 7f56ab200640 1 --2- 172.21.15.7:0/642399821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56ac1053f0 0x7f56ac10bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.393+0000 7f56ab200640 1 --2- 172.21.15.7:0/642399821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ac104a30 0x7f56ac104eb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.393+0000 7f56ab200640 1 --2- 172.21.15.7:0/642399821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56ac103840 0x7f56ac103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.393+0000 7f56ab200640 1 -- 172.21.15.7:0/642399821 >> 172.21.15.7:0/642399821 conn(0x7f56ac0fedb0 msgr2=0x7f56ac1011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:48.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.394+0000 7f56ab200640 1 -- 172.21.15.7:0/642399821 shutdown_connections 2024-03-18T01:37:48.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.394+0000 7f56ab200640 1 -- 172.21.15.7:0/642399821 wait complete. 2024-03-18T01:37:48.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.394+0000 7f56ab200640 1 Processor -- start 2024-03-18T01:37:48.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56ab200640 1 -- start start 2024-03-18T01:37:48.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56ab200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56ac103840 0x7f56ac199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56ab200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56ac104a30 0x7f56ac19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56ab200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ac1053f0 0x7f56ac1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56ab200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac10f4c0 con 0x7f56ac104a30 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56ab200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac10f340 con 0x7f56ac1053f0 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56ab200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac10f640 con 0x7f56ac103840 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56ac103840 0x7f56ac199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56a9e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56ac103840 0x7f56ac199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59374/0 (socket says 172.21.15.7:59374) 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56a9e00640 1 -- 172.21.15.7:0/4084229335 learned_addr learned my addr 172.21.15.7:0/4084229335 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56aa800640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ac1053f0 0x7f56ac1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.395+0000 7f56a9400640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56ac104a30 0x7f56ac19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:48.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a9e00640 1 -- 172.21.15.7:0/4084229335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ac1053f0 msgr2=0x7f56ac1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a9e00640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ac1053f0 0x7f56ac1a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a9e00640 1 -- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56ac104a30 msgr2=0x7f56ac19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a9e00640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56ac104a30 0x7f56ac19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a9e00640 1 -- 172.21.15.7:0/4084229335 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ac1a1750 con 0x7f56ac103840 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a9400640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56ac104a30 0x7f56ac19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a9e00640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56ac103840 0x7f56ac199b30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f569c00e9c0 tx=0x7f569c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a2a00640 1 -- 172.21.15.7:0/4084229335 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c00cd80 con 0x7f56ac103840 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56ac1a1a40 con 0x7f56ac103840 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a2a00640 1 -- 172.21.15.7:0/4084229335 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569c0035f0 con 0x7f56ac103840 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56a2a00640 1 -- 172.21.15.7:0/4084229335 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c010690 con 0x7f56ac103840 2024-03-18T01:37:48.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.396+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56ac1a1fb0 con 0x7f56ac103840 2024-03-18T01:37:48.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.398+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56ac10ab60 con 0x7f56ac103840 2024-03-18T01:37:48.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.400+0000 7f56a2a00640 1 -- 172.21.15.7:0/4084229335 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f569c0037a0 con 0x7f56ac103840 2024-03-18T01:37:48.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.401+0000 7f56a2a00640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56900a2800 0x7f56900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:48.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.401+0000 7f56a2a00640 1 -- 172.21.15.7:0/4084229335 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f569c0cf280 con 0x7f56ac103840 2024-03-18T01:37:48.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.401+0000 7f56a9400640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56900a2800 0x7f56900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:48.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.402+0000 7f56a9400640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56900a2800 0x7f56900a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5698006010 tx=0x7f569802c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:48.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.405+0000 7f56a2a00640 1 -- 172.21.15.7:0/4084229335 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f569c09bf60 con 0x7f56ac103840 2024-03-18T01:37:48.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.662+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f56ac0623e0 con 0x7f56ac103840 2024-03-18T01:37:48.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.662+0000 7f56a2a00640 1 -- 172.21.15.7:0/4084229335 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f569c09b860 con 0x7f56ac103840 2024-03-18T01:37:48.667 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:37:48.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56900a2800 msgr2=0x7f56900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56900a2800 0x7f56900a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5698006010 tx=0x7f569802c8a0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56ac103840 msgr2=0x7f56ac199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56ac103840 0x7f56ac199b30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f569c00e9c0 tx=0x7f569c00ee80 comp rx=0 tx=0).stop 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 shutdown_connections 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56ac1053f0 0x7f56ac1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f56900a2800 0x7f56900a4cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f56ac104a30 0x7f56ac19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 --2- 172.21.15.7:0/4084229335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56ac103840 0x7f56ac199b30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.666+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 >> 172.21.15.7:0/4084229335 conn(0x7f56ac0fedb0 msgr2=0x7f56ac107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.667+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 shutdown_connections 2024-03-18T01:37:48.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.667+0000 7f56ab200640 1 -- 172.21.15.7:0/4084229335 wait complete. 2024-03-18T01:37:48.691 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:37:48.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:48 smithi203 ceph-mon[36345]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T01:37:48.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:48 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:48.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:48 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/386684088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:48.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:48 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3088259342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:48.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 -- 172.21.15.7:0/2023925389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c103200 msgr2=0x7f9d9c109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 --2- 172.21.15.7:0/2023925389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c103200 0x7f9d9c109a90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9d94009ca0 tx=0x7f9d9402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 -- 172.21.15.7:0/2023925389 shutdown_connections 2024-03-18T01:37:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 --2- 172.21.15.7:0/2023925389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c103200 0x7f9d9c109a90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 --2- 172.21.15.7:0/2023925389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d9c102840 0x7f9d9c102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 --2- 172.21.15.7:0/2023925389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d9c101650 0x7f9d9c101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 -- 172.21.15.7:0/2023925389 >> 172.21.15.7:0/2023925389 conn(0x7f9d9c0fcbc0 msgr2=0x7f9d9c0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 -- 172.21.15.7:0/2023925389 shutdown_connections 2024-03-18T01:37:48.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.913+0000 7f9d9be00640 1 -- 172.21.15.7:0/2023925389 wait complete. 2024-03-18T01:37:48.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.915+0000 7f9d9be00640 1 Processor -- start 2024-03-18T01:37:48.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.915+0000 7f9d9be00640 1 -- start start 2024-03-18T01:37:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d9c101650 0x7f9d9c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c102840 0x7f9d9c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9be00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d9c103200 0x7f9d9c1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9be00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c10d2b0 con 0x7f9d9c103200 2024-03-18T01:37:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9be00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c10d130 con 0x7f9d9c102840 2024-03-18T01:37:48.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9be00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c10d430 con 0x7f9d9c101650 2024-03-18T01:37:48.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d93e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c102840 0x7f9d9c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:48.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d9c101650 0x7f9d9c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:48.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d93e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c102840 0x7f9d9c19a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34706/0 (socket says 172.21.15.7:34706) 2024-03-18T01:37:48.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d93e00640 1 -- 172.21.15.7:0/306751543 learned_addr learned my addr 172.21.15.7:0/306751543 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:48.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9b400640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d9c103200 0x7f9d9c1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:48.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9aa00640 1 -- 172.21.15.7:0/306751543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c102840 msgr2=0x7f9d9c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9aa00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c102840 0x7f9d9c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9aa00640 1 -- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d9c103200 msgr2=0x7f9d9c1a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9aa00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d9c103200 0x7f9d9c1a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.916+0000 7f9d9aa00640 1 -- 172.21.15.7:0/306751543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9c1a1730 con 0x7f9d9c101650 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.917+0000 7f9d93e00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c102840 0x7f9d9c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.917+0000 7f9d9aa00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d9c101650 0x7f9d9c199b10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00e9c0 tx=0x7f9d8c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.917+0000 7f9d93400640 1 -- 172.21.15.7:0/306751543 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00ccc0 con 0x7f9d9c101650 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.917+0000 7f9d93400640 1 -- 172.21.15.7:0/306751543 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d8c0035f0 con 0x7f9d9c101650 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.917+0000 7f9d93400640 1 -- 172.21.15.7:0/306751543 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c00ce60 con 0x7f9d9c101650 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.917+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d9c1a1a20 con 0x7f9d9c101650 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.917+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d9c1a1f20 con 0x7f9d9c101650 2024-03-18T01:37:48.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.918+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d9c04f540 con 0x7f9d9c101650 2024-03-18T01:37:48.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.920+0000 7f9d93400640 1 -- 172.21.15.7:0/306751543 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9d8c0037a0 con 0x7f9d9c101650 2024-03-18T01:37:48.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.922+0000 7f9d93400640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9d800a2880 0x7f9d800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:48.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.922+0000 7f9d93400640 1 -- 172.21.15.7:0/306751543 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9d8c0cf160 con 0x7f9d9c101650 2024-03-18T01:37:48.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.922+0000 7f9d93e00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9d800a2880 0x7f9d800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:48.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.923+0000 7f9d93e00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9d800a2880 0x7f9d800a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c19b020 tx=0x7f9d84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:48.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:48.924+0000 7f9d93400640 1 -- 172.21.15.7:0/306751543 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9d8c09be40 con 0x7f9d9c101650 2024-03-18T01:37:49.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:48 smithi046 ceph-mon[36108]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T01:37:49.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:48 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:49.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:48 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/386684088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:49.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:48 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3088259342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:48 smithi007 ceph-mon[31996]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T01:37:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:48 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:48 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/386684088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:37:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:48 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3088259342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:37:49.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.170+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9d9c1a2490 con 0x7f9d9c101650 2024-03-18T01:37:49.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.171+0000 7f9d93400640 1 -- 172.21.15.7:0/306751543 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f9d8c09b740 con 0x7f9d9c101650 2024-03-18T01:37:49.172 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:37:49.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9d800a2880 msgr2=0x7f9d800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9d800a2880 0x7f9d800a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c19b020 tx=0x7f9d84005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:49.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d9c101650 msgr2=0x7f9d9c199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d9c101650 0x7f9d9c199b10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c00e9c0 tx=0x7f9d8c00ee80 comp rx=0 tx=0).stop 2024-03-18T01:37:49.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 shutdown_connections 2024-03-18T01:37:49.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9d800a2880 0x7f9d800a4d40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9d9c103200 0x7f9d9c1a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d9c102840 0x7f9d9c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 --2- 172.21.15.7:0/306751543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d9c101650 0x7f9d9c199b10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 >> 172.21.15.7:0/306751543 conn(0x7f9d9c0fcbc0 msgr2=0x7f9d9c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:49.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 shutdown_connections 2024-03-18T01:37:49.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.173+0000 7f9d9be00640 1 -- 172.21.15.7:0/306751543 wait complete. 2024-03-18T01:37:49.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:37:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.381+0000 7fa58fe00640 1 -- 172.21.15.7:0/1558797342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590101620 msgr2=0x7fa590101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.381+0000 7fa58fe00640 1 --2- 172.21.15.7:0/1558797342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590101620 0x7fa590101a40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa580009900 tx=0x7fa58002bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.381+0000 7fa58fe00640 1 -- 172.21.15.7:0/1558797342 shutdown_connections 2024-03-18T01:37:49.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.382+0000 7fa58fe00640 1 --2- 172.21.15.7:0/1558797342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5901031d0 0x7fa590109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.382+0000 7fa58fe00640 1 --2- 172.21.15.7:0/1558797342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa590102810 0x7fa590102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.382+0000 7fa58fe00640 1 --2- 172.21.15.7:0/1558797342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590101620 0x7fa590101a40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.382+0000 7fa58fe00640 1 -- 172.21.15.7:0/1558797342 >> 172.21.15.7:0/1558797342 conn(0x7fa5900fcb90 msgr2=0x7fa5900fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.382+0000 7fa58fe00640 1 -- 172.21.15.7:0/1558797342 shutdown_connections 2024-03-18T01:37:49.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.382+0000 7fa58fe00640 1 -- 172.21.15.7:0/1558797342 wait complete. 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.383+0000 7fa58fe00640 1 Processor -- start 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.383+0000 7fa58fe00640 1 -- start start 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.383+0000 7fa58fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa590101620 0x7fa590197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.383+0000 7fa58fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590102810 0x7fa590197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.383+0000 7fa58fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5901031d0 0x7fa59019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.383+0000 7fa58fe00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa59010d2e0 con 0x7fa5901031d0 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.383+0000 7fa58fe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa59010d160 con 0x7fa590102810 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.383+0000 7fa58fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa59010d460 con 0x7fa590101620 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590102810 0x7fa590197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:49.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa58ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa590101620 0x7fa590197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:49.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590102810 0x7fa590197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34714/0 (socket says 172.21.15.7:34714) 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 -- 172.21.15.7:0/2181418644 learned_addr learned my addr 172.21.15.7:0/2181418644 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa58f400640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5901031d0 0x7fa59019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 -- 172.21.15.7:0/2181418644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa590101620 msgr2=0x7fa590197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa590101620 0x7fa590197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 -- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5901031d0 msgr2=0x7fa59019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5901031d0 0x7fa59019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 -- 172.21.15.7:0/2181418644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59019f480 con 0x7fa590102810 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.384+0000 7fa587e00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590102810 0x7fa590197da0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa57800ea60 tx=0x7fa57800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.385+0000 7fa587400640 1 -- 172.21.15.7:0/2181418644 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa57800cd20 con 0x7fa590102810 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.385+0000 7fa587400640 1 -- 172.21.15.7:0/2181418644 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5780035f0 con 0x7fa590102810 2024-03-18T01:37:49.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.385+0000 7fa587400640 1 -- 172.21.15.7:0/2181418644 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa578010690 con 0x7fa590102810 2024-03-18T01:37:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.385+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59019f770 con 0x7fa590102810 2024-03-18T01:37:49.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.385+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59019fc70 con 0x7fa590102810 2024-03-18T01:37:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.386+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa590108870 con 0x7fa590102810 2024-03-18T01:37:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.388+0000 7fa587400640 1 -- 172.21.15.7:0/2181418644 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fa5780037a0 con 0x7fa590102810 2024-03-18T01:37:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.389+0000 7fa587400640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5740a2920 0x7fa5740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.389+0000 7fa587400640 1 -- 172.21.15.7:0/2181418644 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa5780cf080 con 0x7fa590102810 2024-03-18T01:37:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.389+0000 7fa58ea00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5740a2920 0x7fa5740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:49.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.390+0000 7fa58ea00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5740a2920 0x7fa5740a4de0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa58002c180 tx=0x7fa58002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:49.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.397+0000 7fa587400640 1 -- 172.21.15.7:0/2181418644 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa57809bd60 con 0x7fa590102810 2024-03-18T01:37:49.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.647+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa59002d1b0 con 0x7fa590102810 2024-03-18T01:37:49.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.648+0000 7fa587400640 1 -- 172.21.15.7:0/2181418644 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11918 (secure 0 0 0) 0x7fa57809b660 con 0x7fa590102810 2024-03-18T01:37:49.650 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:49.650 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:35:30.634148+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24542,24523,24544,24596,24562]},"id":1}]} 2024-03-18T01:37:49.650 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 27 2024-03-18T01:37:49.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5740a2920 msgr2=0x7fa5740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5740a2920 0x7fa5740a4de0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa58002c180 tx=0x7fa58002cf90 comp rx=0 tx=0).stop 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590102810 msgr2=0x7fa590197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590102810 0x7fa590197da0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa57800ea60 tx=0x7fa57800ee60 comp rx=0 tx=0).stop 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 shutdown_connections 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa5740a2920 0x7fa5740a4de0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa5901031d0 0x7fa59019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa590102810 0x7fa590197da0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 --2- 172.21.15.7:0/2181418644 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa590101620 0x7fa590197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.651+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 >> 172.21.15.7:0/2181418644 conn(0x7fa5900fcb90 msgr2=0x7fa590105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.652+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 shutdown_connections 2024-03-18T01:37:49.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.652+0000 7fa58fe00640 1 -- 172.21.15.7:0/2181418644 wait complete. 2024-03-18T01:37:49.672 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:37:49.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 -- 172.21.15.7:0/2883255181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00901020a0 msgr2=0x7f0090108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 --2- 172.21.15.7:0/2883255181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00901020a0 0x7f0090108930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0088009ca0 tx=0x7f008802c100 comp rx=0 tx=0).stop 2024-03-18T01:37:49.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 -- 172.21.15.7:0/2883255181 shutdown_connections 2024-03-18T01:37:49.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 --2- 172.21.15.7:0/2883255181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00901020a0 0x7f0090108930 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 --2- 172.21.15.7:0/2883255181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00901016e0 0x7f0090101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 --2- 172.21.15.7:0/2883255181 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00901004f0 0x7f0090100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 -- 172.21.15.7:0/2883255181 >> 172.21.15.7:0/2883255181 conn(0x7f00900fbaa0 msgr2=0x7f00900fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:49.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 -- 172.21.15.7:0/2883255181 shutdown_connections 2024-03-18T01:37:49.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.877+0000 7f0095200640 1 -- 172.21.15.7:0/2883255181 wait complete. 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.878+0000 7f0095200640 1 Processor -- start 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.878+0000 7f0095200640 1 -- start start 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.878+0000 7f0095200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00901004f0 0x7f00901999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.878+0000 7f0095200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00901016e0 0x7f0090199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f0095200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00901020a0 0x7f00901a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f0095200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f009010c150 con 0x7f00901004f0 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f0095200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f009010bfd0 con 0x7f00901020a0 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f0095200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f009010c2d0 con 0x7f00901016e0 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00901004f0 0x7f00901999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:49.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00901016e0 0x7f0090199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00901016e0 0x7f0090199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59434/0 (socket says 172.21.15.7:59434) 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008ea00640 1 -- 172.21.15.7:0/1499685116 learned_addr learned my addr 172.21.15.7:0/1499685116 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008fe00640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00901020a0 0x7f00901a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008ea00640 1 -- 172.21.15.7:0/1499685116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00901020a0 msgr2=0x7f00901a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008ea00640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00901020a0 0x7f00901a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008ea00640 1 -- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00901004f0 msgr2=0x7f00901999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008ea00640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00901004f0 0x7f00901999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.879+0000 7f008ea00640 1 -- 172.21.15.7:0/1499685116 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00901a15c0 con 0x7f00901016e0 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.880+0000 7f008ea00640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00901016e0 0x7f0090199ee0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f007c00e9f0 tx=0x7f007c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.880+0000 7f0087e00640 1 -- 172.21.15.7:0/1499685116 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c00cd80 con 0x7f00901016e0 2024-03-18T01:37:49.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.880+0000 7f0087e00640 1 -- 172.21.15.7:0/1499685116 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c010910 con 0x7f00901016e0 2024-03-18T01:37:49.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.880+0000 7f0087e00640 1 -- 172.21.15.7:0/1499685116 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c018990 con 0x7f00901016e0 2024-03-18T01:37:49.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.880+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00901a18b0 con 0x7f00901016e0 2024-03-18T01:37:49.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.880+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00901a1e50 con 0x7f00901016e0 2024-03-18T01:37:49.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.881+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00901a2080 con 0x7f00901016e0 2024-03-18T01:37:49.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.884+0000 7f0087e00640 1 -- 172.21.15.7:0/1499685116 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f007c010ac0 con 0x7f00901016e0 2024-03-18T01:37:49.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.885+0000 7f0087e00640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f00740a2920 0x7f00740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:49.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.886+0000 7f0087e00640 1 -- 172.21.15.7:0/1499685116 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f007c0cf230 con 0x7f00901016e0 2024-03-18T01:37:49.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.886+0000 7f008f400640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f00740a2920 0x7f00740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:49.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.887+0000 7f0087e00640 1 -- 172.21.15.7:0/1499685116 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f007c09b8c0 con 0x7f00901016e0 2024-03-18T01:37:49.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:49.887+0000 7f008f400640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f00740a2920 0x7f00740a4de0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0090101540 tx=0x7f0080005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:49.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:49 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:49.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:49 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:49.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:49 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4084229335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:49.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:49 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/306751543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:49 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:49 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:49 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/4084229335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:50.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:49 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/306751543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:50.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:49 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:50.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:49 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:50.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:49 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4084229335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:37:50.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:49 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/306751543' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:37:50.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.099+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f00901a2410 con 0x7f00901016e0 2024-03-18T01:37:50.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.101+0000 7f0087e00640 1 -- 172.21.15.7:0/1499685116 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7f007c09f960 con 0x7f00901016e0 2024-03-18T01:37:50.103 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:37:50.103 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:37:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f00740a2920 msgr2=0x7f00740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f00740a2920 0x7f00740a4de0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0090101540 tx=0x7f0080005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00901016e0 msgr2=0x7f0090199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.107 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00901016e0 0x7f0090199ee0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f007c00e9f0 tx=0x7f007c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 shutdown_connections 2024-03-18T01:37:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f00901020a0 0x7f00901a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00901016e0 0x7f0090199ee0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f00740a2920 0x7f00740a4de0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 --2- 172.21.15.7:0/1499685116 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00901004f0 0x7f00901999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 >> 172.21.15.7:0/1499685116 conn(0x7f00900fbaa0 msgr2=0x7f0090068ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 shutdown_connections 2024-03-18T01:37:50.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.104+0000 7f0095200640 1 -- 172.21.15.7:0/1499685116 wait complete. 2024-03-18T01:37:50.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T01:34:56.125856+0000', 'modified': '2024-03-18T01:35:30.634148+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24596, 'mds_1': 24562, 'mds_2': 24544, 'mds_3': 24542, 'mds_4': 24523}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24448': {'gid': 24448, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/2100401567', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 2100401567}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 2100401567}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.46:6833/1027889667', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6832', 'nonce': 1027889667}, {'type': 'v1', 'addr': '172.21.15.46:6833', 'nonce': 1027889667}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24475': {'gid': 24475, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.7:6835/49989672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6834', 'nonce': 49989672}, {'type': 'v1', 'addr': '172.21.15.7:6835', 'nonce': 49989672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24511': {'gid': 24511, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/848080613', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 848080613}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 848080613}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24512': {'gid': 24512, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.46:6835/2878820671', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6834', 'nonce': 2878820671}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 2878820671}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.203:6837/2344597644', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 2344597644}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 2344597644}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24542': {'gid': 24542, 'name': 'h', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.46:6837/1509246906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6836', 'nonce': 1509246906}, {'type': 'v1', 'addr': '172.21.15.46:6837', 'nonce': 1509246906}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24544': {'gid': 24544, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.203:6839/3801010143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 3801010143}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 3801010143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24562': {'gid': 24562, 'name': 'g', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.7:6839/3166631604', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.7:6838', 'nonce': 3166631604}, {'type': 'v1', 'addr': '172.21.15.7:6839', 'nonce': 3166631604}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24596': {'gid': 24596, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6839/1986652812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6838', 'nonce': 1986652812}, {'type': 'v1', 'addr': '172.21.15.46:6839', 'nonce': 1986652812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24596, 'qdb_cluster': [24542, 24523, 24544, 24596, 24562]} 2024-03-18T01:37:50.127 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T01:37:50.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T01:37:50.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 -- 172.21.15.7:0/1274891639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981049e0 msgr2=0x7f0098106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 --2- 172.21.15.7:0/1274891639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981049e0 0x7f0098106dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0084009860 tx=0x7f008402bb70 comp rx=0 tx=0).stop 2024-03-18T01:37:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 -- 172.21.15.7:0/1274891639 shutdown_connections 2024-03-18T01:37:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 --2- 172.21.15.7:0/1274891639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0098107310 0x7f0098109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 --2- 172.21.15.7:0/1274891639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981049e0 0x7f0098106dd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 --2- 172.21.15.7:0/1274891639 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00981020b0 0x7f00981044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 -- 172.21.15.7:0/1274891639 >> 172.21.15.7:0/1274891639 conn(0x7f00980fbac0 msgr2=0x7f00980fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 -- 172.21.15.7:0/1274891639 shutdown_connections 2024-03-18T01:37:50.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.337+0000 7f009cc00640 1 -- 172.21.15.7:0/1274891639 wait complete. 2024-03-18T01:37:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.338+0000 7f009cc00640 1 Processor -- start 2024-03-18T01:37:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f009cc00640 1 -- start start 2024-03-18T01:37:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f009cc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981020b0 0x7f0098197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f009cc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00981049e0 0x7f0098197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f009cc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0098107310 0x7f009819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f009cc00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f009810d2a0 con 0x7f00981049e0 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f009cc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f009810d120 con 0x7f0098107310 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f009cc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f009810d420 con 0x7f00981020b0 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f0097400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981020b0 0x7f0098197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f0097400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981020b0 0x7f0098197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59456/0 (socket says 172.21.15.7:59456) 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f0097400640 1 -- 172.21.15.7:0/3413520821 learned_addr learned my addr 172.21.15.7:0/3413520821 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f0096a00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00981049e0 0x7f0098197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.339+0000 7f0097e00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0098107310 0x7f009819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f0097400640 1 -- 172.21.15.7:0/3413520821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0098107310 msgr2=0x7f009819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f0097400640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0098107310 0x7f009819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f0097400640 1 -- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00981049e0 msgr2=0x7f0098197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f0097400640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00981049e0 0x7f0098197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f0097400640 1 -- 172.21.15.7:0/3413520821 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009819f560 con 0x7f00981020b0 2024-03-18T01:37:50.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f0097400640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981020b0 0x7f0098197940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f008800e9c0 tx=0x7f008800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f008fe00640 1 -- 172.21.15.7:0/3413520821 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008800ccc0 con 0x7f00981020b0 2024-03-18T01:37:50.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f008fe00640 1 -- 172.21.15.7:0/3413520821 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0088005500 con 0x7f00981020b0 2024-03-18T01:37:50.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f008fe00640 1 -- 172.21.15.7:0/3413520821 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088010430 con 0x7f00981020b0 2024-03-18T01:37:50.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f009cc00640 1 -- 172.21.15.7:0/3413520821 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009819f850 con 0x7f00981020b0 2024-03-18T01:37:50.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.340+0000 7f009cc00640 1 -- 172.21.15.7:0/3413520821 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009819fd20 con 0x7f00981020b0 2024-03-18T01:37:50.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f00a1009740 1 Processor -- start 2024-03-18T01:37:50.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f008fe00640 1 -- 172.21.15.7:0/3413520821 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f00880046c0 con 0x7f00981020b0 2024-03-18T01:37:50.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f00a1009740 1 -- start start 2024-03-18T01:37:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f00a1009740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83340 0x55ee3ac83760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f00a1009740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac88830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f00a1009740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac8b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f00a1009740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ee3abe7c50 con 0x55ee3ac88ea0 2024-03-18T01:37:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f00a1009740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ee3ab909d0 con 0x55ee3ac83d30 2024-03-18T01:37:50.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.344+0000 7f00a1009740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee3abe3cf0 con 0x55ee3ac83340 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac88830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83340 0x55ee3ac83760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac88830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34764/0 (socket says 172.21.15.7:34764) 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 -- 172.21.15.7:0/3825355591 learned_addr learned my addr 172.21.15.7:0/3825355591 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83340 0x55ee3ac83760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59472/0 (socket says 172.21.15.7:59472) 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0096a00640 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac8b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 -- 172.21.15.7:0/3825355591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83340 msgr2=0x55ee3ac83760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83340 0x55ee3ac83760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 -- 172.21.15.7:0/3825355591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac88ea0 msgr2=0x55ee3ac8b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac8b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 -- 172.21.15.7:0/3825355591 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee3abaaf10 con 0x55ee3ac83d30 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097e00640 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83340 0x55ee3ac83760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.345+0000 7f0097400640 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac88830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f00880d2f90 tx=0x7f00880d71a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4856d1f6568ae3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.346+0000 7f008fe00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f007c0a2880 0x7f007c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.346+0000 7f0083e00640 1 -- 172.21.15.7:0/3825355591 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00880e2070 con 0x55ee3ac83d30 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.346+0000 7f0083e00640 1 -- 172.21.15.7:0/3825355591 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00880d12e0 con 0x55ee3ac83d30 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.346+0000 7f0083e00640 1 -- 172.21.15.7:0/3825355591 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088016330 con 0x55ee3ac83d30 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.346+0000 7f0097e00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f007c0a2880 0x7f007c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.346+0000 7f008fe00640 1 -- 172.21.15.7:0/3413520821 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f00880cf080 con 0x7f00981020b0 2024-03-18T01:37:50.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 -- 172.21.15.7:0/3825355591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 msgr2=0x55ee3ac88830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac88830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f00880d2f90 tx=0x7f00880d71a0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f0097e00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f007c0a2880 0x7f007c0a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55ee3ac84590 tx=0x7f0090009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 -- 172.21.15.7:0/3825355591 shutdown_connections 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac8b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac88830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 --2- 172.21.15.7:0/3825355591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83340 0x55ee3ac83760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 -- 172.21.15.7:0/3825355591 >> 172.21.15.7:0/3825355591 conn(0x55ee3ac7e8b0 msgr2=0x55ee3ac7ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 -- 172.21.15.7:0/3825355591 shutdown_connections 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 -- 172.21.15.7:0/3825355591 wait complete. 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.347+0000 7f00a1009740 1 -- start start 2024-03-18T01:37:50.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f00a1009740 1 Processor -- start 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f00a1009740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac918a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f00a1009740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ac91de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f00a1009740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac98e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f00a1009740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ee3ac8be50 con 0x55ee3ac83340 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f00a1009740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ee3aba3e20 con 0x55ee3ac88ea0 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f00a1009740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee3ac8bfd0 con 0x55ee3ac83d30 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f00a1009740 10 client.0 _ll_register_callbacks cb 0x55ee3aba37f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f0097400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac918a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f0097e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac98e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f0096a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ac91de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f0097e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac98e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34770/0 (socket says 172.21.15.7:34770) 2024-03-18T01:37:50.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f0097400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac918a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48308/0 (socket says 172.21.15.7:48308) 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.348+0000 7f0097e00640 1 -- 172.21.15.7:0/723521832 learned_addr learned my addr 172.21.15.7:0/723521832 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0096a00640 1 -- 172.21.15.7:0/723521832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 msgr2=0x55ee3ac98e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0096a00640 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac98e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0096a00640 1 -- 172.21.15.7:0/723521832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 msgr2=0x55ee3ac918a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0096a00640 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac918a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0096a00640 1 -- 172.21.15.7:0/723521832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee3a8fc2d0 con 0x55ee3ac83d30 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0097400640 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac918a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0096a00640 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ac91de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f008402c0b0 tx=0x7f0084009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db2dbd7a4331e639 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0080c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.349+0000 7f0080c00640 1 -- 172.21.15.7:0/723521832 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084043070 con 0x55ee3ac83d30 2024-03-18T01:37:50.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f0080c00640 1 -- 172.21.15.7:0/723521832 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0084004040 con 0x55ee3ac83d30 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 1 -- 172.21.15.7:0/723521832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00880d12e0 con 0x55ee3ac83d30 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f0080c00640 1 -- 172.21.15.7:0/723521832 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f008404f050 con 0x55ee3ac83d30 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 10 client.25372 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 1 -- 172.21.15.7:0/723521832 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee3a9c66b0 con 0x55ee3ac83d30 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f0080c00640 1 -- 172.21.15.7:0/723521832 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00880d12e0 con 0x55ee3ac83d30 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f0080c00640 1 -- 172.21.15.7:0/723521832 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0084032640 con 0x55ee3ac83d30 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 10 client.25372 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 10 client.25372 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 20 client.25372 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55ee3ac9e690 0x55ee3ac9fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 4 client.25372 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:50.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.350+0000 7f00a1009740 1 -- 172.21.15.7:0/723521832 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee3ac452b0 con 0x55ee3ac9e690 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.351+0000 7f0097400640 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55ee3ac9e690 0x55ee3ac9fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.351+0000 7f0097400640 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55ee3ac9e690 0x55ee3ac9fb00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e48176484cf80da server_cookie=f1a16e947443be6a in_seq=0 out_seq=0 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.351+0000 7f0080c00640 10 client.25372 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f0080c00640 1 -- 172.21.15.7:0/723521832 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee3ac452b0 con 0x55ee3ac9e690 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f0080c00640 10 client.25372 handle_command_reply: tid=0 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 client.25372 shutdown 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 -- 172.21.15.7:0/723521832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 msgr2=0x55ee3ac91de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ac91de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f008402c0b0 tx=0x7f0084009540 comp rx=0 tx=0).stop 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 -- 172.21.15.7:0/723521832 shutdown_connections 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55ee3ac9e690 0x55ee3ac9fb00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac98e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ac91de0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 --2- 172.21.15.7:0/723521832 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac918a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f0080c00640 0 client.25372 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:37:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 -- 172.21.15.7:0/723521832 >> 172.21.15.7:0/723521832 conn(0x55ee3ac7e8b0 msgr2=0x55ee3ac7da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 -- 172.21.15.7:0/723521832 shutdown_connections 2024-03-18T01:37:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.352+0000 7f00a1009740 1 -- 172.21.15.7:0/723521832 wait complete. 2024-03-18T01:37:50.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.353+0000 7f00a1009740 20 client.25372 trim_cache size 0 max 16384 2024-03-18T01:37:50.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f00a1009740 1 Processor -- start 2024-03-18T01:37:50.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f00a1009740 1 -- start start 2024-03-18T01:37:50.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f00a1009740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac8bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f00a1009740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac98de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f00a1009740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac914b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f00a1009740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ee3ac9de40 con 0x55ee3ac83340 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f00a1009740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ee3ac9e510 con 0x55ee3ac83d30 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f00a1009740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee3ac9ebe0 con 0x55ee3ac88ea0 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f0096a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac98de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f0097400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac8bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f0096a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac98de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34784/0 (socket says 172.21.15.7:34784) 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f0096a00640 1 -- 172.21.15.7:0/2540586556 learned_addr learned my addr 172.21.15.7:0/2540586556 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.373+0000 7f0097e00640 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac914b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0096a00640 1 -- 172.21.15.7:0/2540586556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac88ea0 msgr2=0x55ee3ac914b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0096a00640 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac914b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0096a00640 1 -- 172.21.15.7:0/2540586556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 msgr2=0x55ee3ac8bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0096a00640 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac8bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0096a00640 1 -- 172.21.15.7:0/2540586556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee3ac92740 con 0x55ee3ac83d30 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0097400640 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac8bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0096a00640 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac98de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0084009780 tx=0x7f00840095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42155cd308ae1860 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0083400640 1 -- 172.21.15.7:0/2540586556 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084058070 con 0x55ee3ac83d30 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0083400640 1 -- 172.21.15.7:0/2540586556 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0084032640 con 0x55ee3ac83d30 2024-03-18T01:37:50.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.374+0000 7f0083400640 1 -- 172.21.15.7:0/2540586556 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084043040 con 0x55ee3ac83d30 2024-03-18T01:37:50.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 -- 172.21.15.7:0/2540586556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 msgr2=0x55ee3ac98de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac98de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0084009780 tx=0x7f00840095b0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 -- 172.21.15.7:0/2540586556 shutdown_connections 2024-03-18T01:37:50.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac88ea0 0x55ee3ac914b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac83d30 0x55ee3ac98de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 --2- 172.21.15.7:0/2540586556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac8bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 -- 172.21.15.7:0/2540586556 >> 172.21.15.7:0/2540586556 conn(0x55ee3ac7e8b0 msgr2=0x55ee3ac7ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 -- 172.21.15.7:0/2540586556 shutdown_connections 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 -- 172.21.15.7:0/2540586556 wait complete. 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.375+0000 7f00a1009740 1 -- start start 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f00a1009740 1 Processor -- start 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f00a1009740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac96d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f00a1009740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ad003e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f00a1009740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ad02930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f00a1009740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ee3ac9f980 con 0x55ee3ac83340 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f00a1009740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ee3ac9f2b0 con 0x55ee3ac88ea0 2024-03-18T01:37:50.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f00a1009740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee3aca0050 con 0x55ee3ac83d30 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f00a1009740 10 client.0 _ll_register_callbacks cb 0x7f00840096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f0097400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac96d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f0096a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ad003e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f0096a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ad003e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59496/0 (socket says 172.21.15.7:59496) 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f0096a00640 1 -- 172.21.15.7:0/3553433914 learned_addr learned my addr 172.21.15.7:0/3553433914 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.376+0000 7f0097e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ad02930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0096a00640 1 -- 172.21.15.7:0/3553433914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 msgr2=0x55ee3ad02930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0096a00640 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ad02930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0096a00640 1 -- 172.21.15.7:0/3553433914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 msgr2=0x55ee3ac96d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0096a00640 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac96d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0096a00640 1 -- 172.21.15.7:0/3553433914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee3ad03140 con 0x55ee3ac83d30 2024-03-18T01:37:50.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0096a00640 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ad003e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f008402c0b0 tx=0x7f008402cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6db320f45fdd3dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0080c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0080c00640 1 -- 172.21.15.7:0/3553433914 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084058070 con 0x55ee3ac83d30 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0080c00640 1 -- 172.21.15.7:0/3553433914 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f008402c510 con 0x55ee3ac83d30 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f00a1009740 1 -- 172.21.15.7:0/3553433914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0084032640 con 0x55ee3ac83d30 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f0080c00640 1 -- 172.21.15.7:0/3553433914 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f008404c040 con 0x55ee3ac83d30 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f00a1009740 10 client.25378 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.377+0000 7f00a1009740 1 -- 172.21.15.7:0/3553433914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee3ad04440 con 0x55ee3ac83d30 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f0080c00640 1 -- 172.21.15.7:0/3553433914 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0084032640 con 0x55ee3ac83d30 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f0080c00640 1 -- 172.21.15.7:0/3553433914 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f00840038f0 con 0x55ee3ac83d30 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f00a1009740 10 client.25378 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f00a1009740 10 client.25378 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f00a1009740 20 client.25378 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:50.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f00a1009740 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55ee3ad08cd0 0x55ee3ad079e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f00a1009740 4 client.25378 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f00a1009740 1 -- 172.21.15.7:0/3553433914 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "status"}) -- 0x55ee3abf5190 con 0x55ee3ad08cd0 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.378+0000 7f0097400640 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55ee3ad08cd0 0x55ee3ad079e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f0097400640 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55ee3ad08cd0 0x55ee3ad079e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=65bc3c463b42eed5 server_cookie=ed006953b8076ea in_seq=0 out_seq=0 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f0080c00640 10 client.25378 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f0080c00640 1 -- 172.21.15.7:0/3553433914 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ee3abf5190 con 0x55ee3ad08cd0 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f0080c00640 10 client.25378 handle_command_reply: tid=0 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 client.25378 shutdown 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 -- 172.21.15.7:0/3553433914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 msgr2=0x55ee3ad003e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ad003e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f008402c0b0 tx=0x7f008402cd20 comp rx=0 tx=0).stop 2024-03-18T01:37:50.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 -- 172.21.15.7:0/3553433914 shutdown_connections 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55ee3ad08cd0 0x55ee3ad079e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ee3ac88ea0 0x55ee3ad02930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee3ac83d30 0x55ee3ad003e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 --2- 172.21.15.7:0/3553433914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ee3ac83340 0x55ee3ac96d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f0080c00640 0 client.25378 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.379+0000 7f00a1009740 1 -- 172.21.15.7:0/3553433914 >> 172.21.15.7:0/3553433914 conn(0x55ee3ac7e8b0 msgr2=0x55ee3ac914b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f00a1009740 1 -- 172.21.15.7:0/3553433914 shutdown_connections 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f00a1009740 1 -- 172.21.15.7:0/3553433914 wait complete. 2024-03-18T01:37:50.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f00a1009740 20 client.25378 trim_cache size 0 max 16384 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 4, 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24523, 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 140.020658674, 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:50.388 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:50.389 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 197.52967571299999 2024-03-18T01:37:50.389 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 -- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f007c0a2880 msgr2=0x7f007c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f007c0a2880 0x7f007c0a4d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55ee3ac84590 tx=0x7f0090009500 comp rx=0 tx=0).stop 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 -- 172.21.15.7:0/3413520821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981020b0 msgr2=0x7f0098197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981020b0 0x7f0098197940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f008800e9c0 tx=0x7f008800ee80 comp rx=0 tx=0).stop 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 -- 172.21.15.7:0/3413520821 shutdown_connections 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0098107310 0x7f009819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f007c0a2880 0x7f007c0a4d40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f00981049e0 0x7f0098197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 --2- 172.21.15.7:0/3413520821 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00981020b0 0x7f0098197940 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.380+0000 7f009cc00640 1 -- 172.21.15.7:0/3413520821 >> 172.21.15.7:0/3413520821 conn(0x7f00980fbac0 msgr2=0x7f00981029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.381+0000 7f009cc00640 1 -- 172.21.15.7:0/3413520821 shutdown_connections 2024-03-18T01:37:50.391 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.381+0000 7f009cc00640 1 -- 172.21.15.7:0/3413520821 wait complete. 2024-03-18T01:37:50.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T01:37:50.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 -- 172.21.15.7:0/1270895266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d0102340 msgr2=0x7f74d0104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 --2- 172.21.15.7:0/1270895266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d0102340 0x7f74d0104730 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f74c0009900 tx=0x7f74c002bc40 comp rx=0 tx=0).stop 2024-03-18T01:37:50.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 -- 172.21.15.7:0/1270895266 shutdown_connections 2024-03-18T01:37:50.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 --2- 172.21.15.7:0/1270895266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74d01075a0 0x7f74d0109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 --2- 172.21.15.7:0/1270895266 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d0104c70 0x7f74d0107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 --2- 172.21.15.7:0/1270895266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d0102340 0x7f74d0104730 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 -- 172.21.15.7:0/1270895266 >> 172.21.15.7:0/1270895266 conn(0x7f74d00fbc90 msgr2=0x7f74d00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 -- 172.21.15.7:0/1270895266 shutdown_connections 2024-03-18T01:37:50.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.609+0000 7f74d5800640 1 -- 172.21.15.7:0/1270895266 wait complete. 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.610+0000 7f74d5800640 1 Processor -- start 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.610+0000 7f74d5800640 1 -- start start 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.610+0000 7f74d5800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d0102340 0x7f74d0199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.610+0000 7f74d5800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74d0104c70 0x7f74d019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.610+0000 7f74d5800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d01075a0 0x7f74d01a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.610+0000 7f74d5800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f74d0101490 con 0x7f74d0102340 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.610+0000 7f74d5800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f74d0101310 con 0x7f74d0104c70 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.610+0000 7f74d5800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74d0101610 con 0x7f74d01075a0 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74d0104c70 0x7f74d019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74cf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d01075a0 0x7f74d01a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74cea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d0102340 0x7f74d0199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74d0104c70 0x7f74d019a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34826/0 (socket says 172.21.15.7:34826) 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 -- 172.21.15.7:0/2403470727 learned_addr learned my addr 172.21.15.7:0/2403470727 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 -- 172.21.15.7:0/2403470727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d01075a0 msgr2=0x7f74d01a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d01075a0 0x7f74d01a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 -- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d0102340 msgr2=0x7f74d0199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d0102340 0x7f74d0199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 -- 172.21.15.7:0/2403470727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d01a1980 con 0x7f74d0104c70 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74cf400640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d01075a0 0x7f74d01a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74cea00640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d0102340 0x7f74d0199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.611+0000 7f74c7e00640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74d0104c70 0x7f74d019a2a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f74b800e9c0 tx=0x7f74b800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.612+0000 7f74c7400640 1 -- 172.21.15.7:0/2403470727 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b800cd80 con 0x7f74d0104c70 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.612+0000 7f74c7400640 1 -- 172.21.15.7:0/2403470727 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b8005500 con 0x7f74d0104c70 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.612+0000 7f74d5800640 1 -- 172.21.15.7:0/2403470727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d01a1c70 con 0x7f74d0104c70 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.612+0000 7f74c7400640 1 -- 172.21.15.7:0/2403470727 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b8010690 con 0x7f74d0104c70 2024-03-18T01:37:50.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.612+0000 7f74d5800640 1 -- 172.21.15.7:0/2403470727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d01a21c0 con 0x7f74d0104c70 2024-03-18T01:37:50.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.615+0000 7f74c7400640 1 -- 172.21.15.7:0/2403470727 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f74b80046c0 con 0x7f74d0104c70 2024-03-18T01:37:50.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74c7400640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74b40a2880 0x7f74b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74d83f5740 1 Processor -- start 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74c7400640 1 -- 172.21.15.7:0/2403470727 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74b80cf0d0 con 0x7f74d0104c70 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74d83f5740 1 -- start start 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74cea00640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74b40a2880 0x7f74b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adc3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adc8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9adcb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74d83f5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556b9ad27c50 con 0x556b9adc3160 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74d83f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b9acd09d0 con 0x556b9adc8cc0 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.617+0000 7f74d83f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b9ad23cf0 con 0x556b9adc3b50 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adc3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adc8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cf400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9adcb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adc8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59534/0 (socket says 172.21.15.7:59534) 2024-03-18T01:37:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adc3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48354/0 (socket says 172.21.15.7:48354) 2024-03-18T01:37:50.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74c7e00640 1 -- 172.21.15.7:0/2503106144 learned_addr learned my addr 172.21.15.7:0/2503106144 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cf400640 1 -- 172.21.15.7:0/2503106144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 msgr2=0x556b9adc8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cf400640 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adc8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cf400640 1 -- 172.21.15.7:0/2503106144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 msgr2=0x556b9adc3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cf400640 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adc3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cf400640 1 -- 172.21.15.7:0/2503106144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b9aceaf10 con 0x556b9adc8cc0 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74c7e00640 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adc8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.618+0000 7f74cea00640 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adc3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.619+0000 7f74cf400640 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9adcb150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f74c8005fd0 tx=0x7f74c8011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bedd93988eafe7c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.619+0000 7f74cea00640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74b40a2880 0x7f74b40a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f74c002c250 tx=0x7f74c0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.621+0000 7f74bfe00640 1 -- 172.21.15.7:0/2503106144 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c8004d20 con 0x556b9adc8cc0 2024-03-18T01:37:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.621+0000 7f74bfe00640 1 -- 172.21.15.7:0/2503106144 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c800f9a0 con 0x556b9adc8cc0 2024-03-18T01:37:50.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.621+0000 7f74bfe00640 1 -- 172.21.15.7:0/2503106144 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c8004550 con 0x556b9adc8cc0 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.621+0000 7f74d83f5740 1 -- 172.21.15.7:0/2503106144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 msgr2=0x556b9adcb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.621+0000 7f74d83f5740 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9adcb150 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f74c8005fd0 tx=0x7f74c8011bf0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.622+0000 7f74d83f5740 1 -- 172.21.15.7:0/2503106144 shutdown_connections 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.622+0000 7f74d83f5740 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9adcb150 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.622+0000 7f74d83f5740 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adc8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.622+0000 7f74d83f5740 1 --2- 172.21.15.7:0/2503106144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adc3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.622+0000 7f74d83f5740 1 -- 172.21.15.7:0/2503106144 >> 172.21.15.7:0/2503106144 conn(0x556b9adbe6d0 msgr2=0x556b9adbead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.622+0000 7f74d83f5740 1 -- 172.21.15.7:0/2503106144 shutdown_connections 2024-03-18T01:37:50.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.622+0000 7f74d83f5740 1 -- 172.21.15.7:0/2503106144 wait complete. 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.622+0000 7f74d83f5740 1 -- start start 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74d83f5740 1 Processor -- start 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9add4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74d83f5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556b9adcbc70 con 0x556b9adc3160 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74d83f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b9ace3e20 con 0x556b9adc8cc0 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74d83f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b9adcbdf0 con 0x556b9adc3b50 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74d83f5740 10 client.0 _ll_register_callbacks cb 0x556b9ace37f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.627 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74cf400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9add4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59540/0 (socket says 172.21.15.7:59540) 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74cf400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9add4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34844/0 (socket says 172.21.15.7:34844) 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.623+0000 7f74c7e00640 1 -- 172.21.15.7:0/1867095332 learned_addr learned my addr 172.21.15.7:0/1867095332 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74cea00640 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74c7e00640 1 -- 172.21.15.7:0/1867095332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 msgr2=0x556b9add4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74c7e00640 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9add4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74c7e00640 1 -- 172.21.15.7:0/1867095332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 msgr2=0x556b9adcd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74c7e00640 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74c7e00640 1 -- 172.21.15.7:0/1867095332 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b9aa3c2d0 con 0x556b9adc3b50 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74cea00640 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74c7e00640 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f74b800c6a0 tx=0x7f74b809b2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b70246d710eb480f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.624+0000 7f74bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.625+0000 7f74bcc00640 1 -- 172.21.15.7:0/1867095332 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b80b3020 con 0x556b9adc3b50 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.625+0000 7f74bcc00640 1 -- 172.21.15.7:0/1867095332 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74b80a3bc0 con 0x556b9adc3b50 2024-03-18T01:37:50.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.625+0000 7f74d83f5740 1 -- 172.21.15.7:0/1867095332 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74c800f9a0 con 0x556b9adc3b50 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.625+0000 7f74bcc00640 1 -- 172.21.15.7:0/1867095332 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74b809f800 con 0x556b9adc3b50 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.625+0000 7f74d83f5740 10 client.25390 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.625+0000 7f74d83f5740 1 -- 172.21.15.7:0/1867095332 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556b9ab066b0 con 0x556b9adc3b50 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74bcc00640 1 -- 172.21.15.7:0/1867095332 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74c800f9a0 con 0x556b9adc3b50 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74bcc00640 1 -- 172.21.15.7:0/1867095332 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f74b8097400 con 0x556b9adc3b50 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74d83f5740 10 client.25390 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74d83f5740 10 client.25390 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74d83f5740 20 client.25390 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74d83f5740 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x556b9adda0f0 0x556b9addb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74d83f5740 4 client.25390 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:50.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74d83f5740 1 -- 172.21.15.7:0/1867095332 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b9ad852b0 con 0x556b9adda0f0 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.626+0000 7f74cea00640 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x556b9adda0f0 0x556b9addb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.627+0000 7f74cea00640 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x556b9adda0f0 0x556b9addb560 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cc2c96b8d6079e66 server_cookie=e9855379ba450f4a in_seq=0 out_seq=0 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.627+0000 7f74bcc00640 10 client.25390 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74bcc00640 1 -- 172.21.15.7:0/1867095332 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556b9ad852b0 con 0x556b9adda0f0 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74bcc00640 10 client.25390 handle_command_reply: tid=0 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 client.25390 shutdown 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 -- 172.21.15.7:0/1867095332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 msgr2=0x556b9adcd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f74b800c6a0 tx=0x7f74b809b2b0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 -- 172.21.15.7:0/1867095332 shutdown_connections 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x556b9adda0f0 0x556b9addb560 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc8cc0 0x556b9add4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 --2- 172.21.15.7:0/1867095332 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74d83f5740 1 -- 172.21.15.7:0/1867095332 >> 172.21.15.7:0/1867095332 conn(0x556b9adbe6d0 msgr2=0x556b9adbda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.628+0000 7f74bcc00640 0 client.25390 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.629+0000 7f74d83f5740 1 -- 172.21.15.7:0/1867095332 shutdown_connections 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.629+0000 7f74d83f5740 1 -- 172.21.15.7:0/1867095332 wait complete. 2024-03-18T01:37:50.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.629+0000 7f74d83f5740 20 client.25390 trim_cache size 0 max 16384 2024-03-18T01:37:50.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74d83f5740 1 Processor -- start 2024-03-18T01:37:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74d83f5740 1 -- start start 2024-03-18T01:37:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc8cc0 0x556b9add47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74d83f5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556b9add98a0 con 0x556b9adc3160 2024-03-18T01:37:50.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74d83f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b9add9f70 con 0x556b9adc3b50 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74d83f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b9adda640 con 0x556b9adc8cc0 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34852/0 (socket says 172.21.15.7:34852) 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74c7e00640 1 -- 172.21.15.7:0/845017400 learned_addr learned my addr 172.21.15.7:0/845017400 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.649+0000 7f74cea00640 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74cf400640 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc8cc0 0x556b9add47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74c7e00640 1 -- 172.21.15.7:0/845017400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc8cc0 msgr2=0x556b9add47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74c7e00640 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc8cc0 0x556b9add47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74c7e00640 1 -- 172.21.15.7:0/845017400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 msgr2=0x556b9adcd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74c7e00640 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74c7e00640 1 -- 172.21.15.7:0/845017400 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b9ad94750 con 0x556b9adc3b50 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74cf400640 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc8cc0 0x556b9add47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:37:50.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74c7e00640 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f74b8098080 tx=0x7f74b8099bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef52257a36f00ae2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74bf400640 1 -- 172.21.15.7:0/845017400 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b80b3070 con 0x556b9adc3b50 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74bf400640 1 -- 172.21.15.7:0/845017400 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b809ae20 con 0x556b9adc3b50 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.650+0000 7f74bf400640 1 -- 172.21.15.7:0/845017400 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b809a570 con 0x556b9adc3b50 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 -- 172.21.15.7:0/845017400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3b50 msgr2=0x556b9adcd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f74b8098080 tx=0x7f74b8099bc0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 -- 172.21.15.7:0/845017400 shutdown_connections 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc8cc0 0x556b9add47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3b50 0x556b9adcd7c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 --2- 172.21.15.7:0/845017400 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc3160 0x556b9adcd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 -- 172.21.15.7:0/845017400 >> 172.21.15.7:0/845017400 conn(0x556b9adbe6d0 msgr2=0x556b9add8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 -- 172.21.15.7:0/845017400 shutdown_connections 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 -- 172.21.15.7:0/845017400 wait complete. 2024-03-18T01:37:50.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.651+0000 7f74d83f5740 1 -- start start 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74d83f5740 1 Processor -- start 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3160 0x556b9add27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9ae3be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74d83f5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc8cc0 0x556b9ae3e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74d83f5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556b9addb3e0 con 0x556b9adc8cc0 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74d83f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b9addad10 con 0x556b9adc3160 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74d83f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b9addbab0 con 0x556b9adc3b50 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74d83f5740 10 client.0 _ll_register_callbacks cb 0x7f74b80979e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9ae3be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74cea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3160 0x556b9add27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74cf400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc8cc0 0x556b9ae3e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74c7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9ae3be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59560/0 (socket says 172.21.15.7:59560) 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74cea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3160 0x556b9add27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34854/0 (socket says 172.21.15.7:34854) 2024-03-18T01:37:50.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.652+0000 7f74c7e00640 1 -- 172.21.15.7:0/3364270523 learned_addr learned my addr 172.21.15.7:0/3364270523 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74c7e00640 1 -- 172.21.15.7:0/3364270523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3160 msgr2=0x556b9add27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74c7e00640 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3160 0x556b9add27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74c7e00640 1 -- 172.21.15.7:0/3364270523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc8cc0 msgr2=0x556b9ae3e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74c7e00640 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc8cc0 0x556b9ae3e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74c7e00640 1 -- 172.21.15.7:0/3364270523 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b9ae3ea30 con 0x556b9adc3b50 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74cea00640 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3160 0x556b9add27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74c7e00640 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9ae3be40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f74b8099960 tx=0x7f74b8004b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=337fa7b635b99661 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74bcc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74bcc00640 1 -- 172.21.15.7:0/3364270523 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b80b3070 con 0x556b9adc3b50 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74bcc00640 1 -- 172.21.15.7:0/3364270523 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74b80b4220 con 0x556b9adc3b50 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74d83f5740 1 -- 172.21.15.7:0/3364270523 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74b809ae20 con 0x556b9adc3b50 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74bcc00640 1 -- 172.21.15.7:0/3364270523 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74b80a3c30 con 0x556b9adc3b50 2024-03-18T01:37:50.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74d83f5740 10 client.25393 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.653+0000 7f74d83f5740 1 -- 172.21.15.7:0/3364270523 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556b9ae3ea30 con 0x556b9adc3b50 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74bcc00640 1 -- 172.21.15.7:0/3364270523 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b809ae20 con 0x556b9adc3b50 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74bcc00640 1 -- 172.21.15.7:0/3364270523 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f74b80a0720 con 0x556b9adc3b50 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74d83f5740 10 client.25393 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74d83f5740 10 client.25393 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74d83f5740 20 client.25393 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74d83f5740 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x556b9ae442a0 0x556b9ae40db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74d83f5740 4 client.25393 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74d83f5740 1 -- 172.21.15.7:0/3364270523 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "status"}) -- 0x556b9ad35190 con 0x556b9ae442a0 2024-03-18T01:37:50.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.654+0000 7f74cea00640 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x556b9ae442a0 0x556b9ae40db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74cea00640 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x556b9ae442a0 0x556b9ae40db0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=785c40787faa1009 server_cookie=8659313779f9f5e5 in_seq=0 out_seq=0 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74bcc00640 10 client.25393 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74bcc00640 1 -- 172.21.15.7:0/3364270523 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556b9ad35190 con 0x556b9ae442a0 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74bcc00640 10 client.25393 handle_command_reply: tid=0 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 client.25393 shutdown 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 -- 172.21.15.7:0/3364270523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 msgr2=0x556b9ae3be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9ae3be40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f74b8099960 tx=0x7f74b8004b30 comp rx=0 tx=0).stop 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 -- 172.21.15.7:0/3364270523 shutdown_connections 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x556b9ae442a0 0x556b9ae40db0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b9adc8cc0 0x556b9ae3e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b9adc3b50 0x556b9ae3be40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74bcc00640 0 client.25393 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:37:50.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 --2- 172.21.15.7:0/3364270523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b9adc3160 0x556b9add27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 -- 172.21.15.7:0/3364270523 >> 172.21.15.7:0/3364270523 conn(0x556b9adbe6d0 msgr2=0x556b9adcd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.655+0000 7f74d83f5740 1 -- 172.21.15.7:0/3364270523 shutdown_connections 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d83f5740 1 -- 172.21.15.7:0/3364270523 wait complete. 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d83f5740 20 client.25393 trim_cache size 0 max 16384 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 3, 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24542, 2024-03-18T01:37:50.667 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 141.304952273, 2024-03-18T01:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 194.86783872000001 2024-03-18T01:37:50.668 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 -- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74b40a2880 msgr2=0x7f74b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74b40a2880 0x7f74b40a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f74c002c250 tx=0x7f74c0000bc0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 -- 172.21.15.7:0/2403470727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74d0104c70 msgr2=0x7f74d019a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74d0104c70 0x7f74d019a2a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f74b800e9c0 tx=0x7f74b800ee80 comp rx=0 tx=0).stop 2024-03-18T01:37:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 -- 172.21.15.7:0/2403470727 shutdown_connections 2024-03-18T01:37:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74d01075a0 0x7f74d01a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74d0104c70 0x7f74d019a2a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74b40a2880 0x7f74b40a4d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 --2- 172.21.15.7:0/2403470727 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74d0102340 0x7f74d0199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.656+0000 7f74d5800640 1 -- 172.21.15.7:0/2403470727 >> 172.21.15.7:0/2403470727 conn(0x7f74d00fbc90 msgr2=0x7f74d0102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.670 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.657+0000 7f74d5800640 1 -- 172.21.15.7:0/2403470727 shutdown_connections 2024-03-18T01:37:50.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.657+0000 7f74d5800640 1 -- 172.21.15.7:0/2403470727 wait complete. 2024-03-18T01:37:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:50 smithi007 ceph-mon[31996]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:50 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2181418644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:50.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:37:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:37:50.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:50.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:50 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1499685116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:50.677 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T01:37:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.875+0000 7f7786600640 1 -- 172.21.15.7:0/3921841373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 msgr2=0x7f7780109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.875+0000 7f7786600640 1 --2- 172.21.15.7:0/3921841373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 0x7f7780109780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7778009ca0 tx=0x7f777802c100 comp rx=0 tx=0).stop 2024-03-18T01:37:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.876+0000 7f7786600640 1 -- 172.21.15.7:0/3921841373 shutdown_connections 2024-03-18T01:37:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.876+0000 7f7786600640 1 --2- 172.21.15.7:0/3921841373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 0x7f7780109780 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.876+0000 7f7786600640 1 --2- 172.21.15.7:0/3921841373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780104a60 0x7f7780106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.876+0000 7f7786600640 1 --2- 172.21.15.7:0/3921841373 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7780102130 0x7f7780104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.876+0000 7f7786600640 1 -- 172.21.15.7:0/3921841373 >> 172.21.15.7:0/3921841373 conn(0x7f77800fbac0 msgr2=0x7f77800fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.876+0000 7f7786600640 1 -- 172.21.15.7:0/3921841373 shutdown_connections 2024-03-18T01:37:50.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.876+0000 7f7786600640 1 -- 172.21.15.7:0/3921841373 wait complete. 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7786600640 1 Processor -- start 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7786600640 1 -- start start 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7786600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7780102130 0x7f7780101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7786600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780104a60 0x7f77801000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7786600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 0x7f7780100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7786600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f778010d2c0 con 0x7f7780102130 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7786600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f778010d140 con 0x7f7780107390 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7786600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f778010d440 con 0x7f7780104a60 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7785200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7780102130 0x7f7780101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.877+0000 7f7785c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 0x7f7780100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7780102130 0x7f7780101aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48376/0 (socket says 172.21.15.7:48376) 2024-03-18T01:37:50.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7777e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780104a60 0x7f77801000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 0x7f7780100620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34876/0 (socket says 172.21.15.7:34876) 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785200640 1 -- 172.21.15.7:0/2574046732 learned_addr learned my addr 172.21.15.7:0/2574046732 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785c00640 1 -- 172.21.15.7:0/2574046732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780104a60 msgr2=0x7f77801000e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785c00640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780104a60 0x7f77801000e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785c00640 1 -- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7780102130 msgr2=0x7f7780101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785c00640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7780102130 0x7f7780101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785c00640 1 -- 172.21.15.7:0/2574046732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7780100eb0 con 0x7f7780107390 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.878+0000 7f7785c00640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 0x7f7780100620 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f777802c640 tx=0x7f7778035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.879+0000 7f777e000640 1 -- 172.21.15.7:0/2574046732 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778039040 con 0x7f7780107390 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.879+0000 7f777e000640 1 -- 172.21.15.7:0/2574046732 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7778044070 con 0x7f7780107390 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.879+0000 7f777e000640 1 -- 172.21.15.7:0/2574046732 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77780043e0 con 0x7f7780107390 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.879+0000 7f7786600640 1 -- 172.21.15.7:0/2574046732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77801a1960 con 0x7f7780107390 2024-03-18T01:37:50.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.879+0000 7f7786600640 1 -- 172.21.15.7:0/2574046732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77801a1e30 con 0x7f7780107390 2024-03-18T01:37:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f778a994740 1 Processor -- start 2024-03-18T01:37:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f777e000640 1 -- 172.21.15.7:0/2574046732 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f7778004580 con 0x7f7780107390 2024-03-18T01:37:50.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f778a994740 1 -- start start 2024-03-18T01:37:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f778a994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e2394a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f778a994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2394f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f778a994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23952330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f778a994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564e238aec50 con 0x564e2394a340 2024-03-18T01:37:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f778a994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564e238579d0 con 0x564e2394ad30 2024-03-18T01:37:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.882+0000 7f778a994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564e238aacf0 con 0x564e2394fea0 2024-03-18T01:37:50.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7785200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2394f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7785200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2394f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34882/0 (socket says 172.21.15.7:34882) 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7785200640 1 -- 172.21.15.7:0/4173690968 learned_addr learned my addr 172.21.15.7:0/4173690968 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7777e00640 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e2394a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7785200640 1 -- 172.21.15.7:0/4173690968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 msgr2=0x564e23952330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7777e00640 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23952330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7785200640 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23952330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7785200640 1 -- 172.21.15.7:0/4173690968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 msgr2=0x564e2394a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7785200640 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e2394a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7785200640 1 -- 172.21.15.7:0/4173690968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e23871f10 con 0x564e2394ad30 2024-03-18T01:37:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.883+0000 7f7777e00640 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23952330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:37:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.884+0000 7f7785200640 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2394f830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7770009dc0 tx=0x7f7770011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2f255a97be5e85d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.884+0000 7f777e000640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77640a2880 0x7f77640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.884+0000 7f777e000640 1 -- 172.21.15.7:0/2574046732 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f77780f1330 con 0x7f7780107390 2024-03-18T01:37:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.884+0000 7f7776000640 1 -- 172.21.15.7:0/4173690968 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777001e070 con 0x564e2394ad30 2024-03-18T01:37:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.884+0000 7f7776000640 1 -- 172.21.15.7:0/4173690968 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7770019b10 con 0x564e2394ad30 2024-03-18T01:37:50.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.884+0000 7f7776000640 1 -- 172.21.15.7:0/4173690968 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777000eb00 con 0x564e2394ad30 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.884+0000 7f7777e00640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77640a2880 0x7f77640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 -- 172.21.15.7:0/4173690968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 msgr2=0x564e2394f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2394f830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7770009dc0 tx=0x7f7770011810 comp rx=0 tx=0).stop 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f7777e00640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77640a2880 0x7f77640a4d40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7768009710 tx=0x7f7768005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 -- 172.21.15.7:0/4173690968 shutdown_connections 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23952330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2394f830 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 --2- 172.21.15.7:0/4173690968 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e2394a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 -- 172.21.15.7:0/4173690968 >> 172.21.15.7:0/4173690968 conn(0x564e239458b0 msgr2=0x564e23945cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 -- 172.21.15.7:0/4173690968 shutdown_connections 2024-03-18T01:37:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.885+0000 7f778a994740 1 -- 172.21.15.7:0/4173690968 wait complete. 2024-03-18T01:37:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.886+0000 7f778a994740 1 -- start start 2024-03-18T01:37:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.886+0000 7f778a994740 1 Processor -- start 2024-03-18T01:37:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f778a994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e23954490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f778a994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394ad30 0x564e239549d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f778a994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f778a994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564e23952e50 con 0x564e2394a340 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f778a994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564e2386ae20 con 0x564e2394fea0 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f778a994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564e23952fd0 con 0x564e2394ad30 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f778a994740 10 client.0 _ll_register_callbacks cb 0x564e2386a7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e23954490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e23954490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48394/0 (socket says 172.21.15.7:48394) 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785200640 1 -- 172.21.15.7:0/505046625 learned_addr learned my addr 172.21.15.7:0/505046625 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7777e00640 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394ad30 0x564e239549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785c00640 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785200640 1 -- 172.21.15.7:0/505046625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394ad30 msgr2=0x564e239549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785200640 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394ad30 0x564e239549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785200640 1 -- 172.21.15.7:0/505046625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 msgr2=0x564e2395ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785200640 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.896 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785200640 1 -- 172.21.15.7:0/505046625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e235c32d0 con 0x564e2394a340 2024-03-18T01:37:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.887+0000 7f7785c00640 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.888+0000 7f7785200640 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e23954490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7770000c00 tx=0x7f7770019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3090d7fba864d650 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.888+0000 7f776ea00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:37:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.888+0000 7f776ea00640 1 -- 172.21.15.7:0/505046625 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7770027070 con 0x564e2394a340 2024-03-18T01:37:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.888+0000 7f776ea00640 1 -- 172.21.15.7:0/505046625 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7770004020 con 0x564e2394a340 2024-03-18T01:37:50.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.888+0000 7f778a994740 1 -- 172.21.15.7:0/505046625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7770019b10 con 0x564e2394a340 2024-03-18T01:37:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.888+0000 7f776ea00640 1 -- 172.21.15.7:0/505046625 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7770032050 con 0x564e2394a340 2024-03-18T01:37:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.888+0000 7f778a994740 10 client.15261 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.888+0000 7f778a994740 1 -- 172.21.15.7:0/505046625 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e2368d6b0 con 0x564e2394a340 2024-03-18T01:37:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.889+0000 7f776ea00640 1 -- 172.21.15.7:0/505046625 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7770019b10 con 0x564e2394a340 2024-03-18T01:37:50.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.889+0000 7f776ea00640 1 -- 172.21.15.7:0/505046625 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f777000e880 con 0x564e2394a340 2024-03-18T01:37:50.899 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:50 smithi203 ceph-mon[36345]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:50.899 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:50 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2181418644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:50.899 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:37:50.899 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:37:50.899 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:50.899 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:50 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1499685116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.889+0000 7f778a994740 10 client.15261 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.889+0000 7f778a994740 10 client.15261 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T01:37:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.889+0000 7f778a994740 20 client.15261 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:50.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.889+0000 7f778a994740 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x564e23960f00 0x564e23962370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.889+0000 7f778a994740 4 client.15261 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.889+0000 7f778a994740 1 -- 172.21.15.7:0/505046625 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e2390c2b0 con 0x564e23960f00 2024-03-18T01:37:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.890+0000 7f7777e00640 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x564e23960f00 0x564e23962370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.890+0000 7f7777e00640 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x564e23960f00 0x564e23962370 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28d8f9bcd684532f server_cookie=da94548a1626c03c in_seq=0 out_seq=0 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.890+0000 7f776ea00640 10 client.15261 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.891+0000 7f776ea00640 1 -- 172.21.15.7:0/505046625 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e2390c2b0 con 0x564e23960f00 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.891+0000 7f776ea00640 10 client.15261 handle_command_reply: tid=0 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.891+0000 7f778a994740 1 client.15261 shutdown 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.891+0000 7f778a994740 1 -- 172.21.15.7:0/505046625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 msgr2=0x564e23954490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.891+0000 7f778a994740 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e23954490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7770000c00 tx=0x7f7770019660 comp rx=0 tx=0).stop 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 1 -- 172.21.15.7:0/505046625 shutdown_connections 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x564e23960f00 0x564e23962370 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394ad30 0x564e239549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 1 --2- 172.21.15.7:0/505046625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e23954490 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f776ea00640 0 client.15261 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:37:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 1 -- 172.21.15.7:0/505046625 >> 172.21.15.7:0/505046625 conn(0x564e239458b0 msgr2=0x564e23944a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 1 -- 172.21.15.7:0/505046625 shutdown_connections 2024-03-18T01:37:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 1 -- 172.21.15.7:0/505046625 wait complete. 2024-03-18T01:37:50.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.892+0000 7f778a994740 20 client.15261 trim_cache size 0 max 16384 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f778a994740 1 Processor -- start 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f778a994740 1 -- start start 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f778a994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e239441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f778a994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2395b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f778a994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23944720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f778a994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564e239606b0 con 0x564e2394a340 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f778a994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564e23960d80 con 0x564e2394ad30 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f778a994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564e23961450 con 0x564e2394fea0 2024-03-18T01:37:50.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f7777e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2395b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f7785c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23944720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f7777e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2395b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34890/0 (socket says 172.21.15.7:34890) 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f7777e00640 1 -- 172.21.15.7:0/2685612775 learned_addr learned my addr 172.21.15.7:0/2685612775 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.913+0000 7f7785200640 1 --2- 172.21.15.7:0/2685612775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e239441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7777e00640 1 -- 172.21.15.7:0/2685612775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 msgr2=0x564e23944720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7777e00640 1 --2- 172.21.15.7:0/2685612775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23944720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7777e00640 1 -- 172.21.15.7:0/2685612775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 msgr2=0x564e239441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7777e00640 1 --2- 172.21.15.7:0/2685612775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e239441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7777e00640 1 -- 172.21.15.7:0/2685612775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e239555c0 con 0x564e2394ad30 2024-03-18T01:37:50.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7777e00640 1 --2- 172.21.15.7:0/2685612775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2395b930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f776801f720 tx=0x7f776801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a73e23882427960d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7775600640 1 -- 172.21.15.7:0/2685612775 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776802b3b0 con 0x564e2394ad30 2024-03-18T01:37:50.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7775600640 1 -- 172.21.15.7:0/2685612775 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7768005500 con 0x564e2394ad30 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.914+0000 7f7775600640 1 -- 172.21.15.7:0/2685612775 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7768004b40 con 0x564e2394ad30 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 -- 172.21.15.7:0/2685612775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 msgr2=0x564e2395b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 --2- 172.21.15.7:0/2685612775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2395b930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f776801f720 tx=0x7f776801b000 comp rx=0 tx=0).stop 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 -- 172.21.15.7:0/2685612775 shutdown_connections 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 --2- 172.21.15.7:0/2685612775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394fea0 0x564e23944720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 --2- 172.21.15.7:0/2685612775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394ad30 0x564e2395b930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 --2- 172.21.15.7:0/2685612775 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394a340 0x564e239441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 -- 172.21.15.7:0/2685612775 >> 172.21.15.7:0/2685612775 conn(0x564e239458b0 msgr2=0x564e23945c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 -- 172.21.15.7:0/2685612775 shutdown_connections 2024-03-18T01:37:50.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.915+0000 7f778a994740 1 -- 172.21.15.7:0/2685612775 wait complete. 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 1 -- start start 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 1 Processor -- start 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394a340 0x564e23956d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394ad30 0x564e23957290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564e239621f0 con 0x564e2394ad30 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564e23961b20 con 0x564e2394fea0 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564e239628c0 con 0x564e2394a340 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f778a994740 10 client.0 _ll_register_callbacks cb 0x7f777000e4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f7785200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394a340 0x564e23956d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f7785c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.916+0000 7f7785200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394a340 0x564e23956d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59626/0 (socket says 172.21.15.7:59626) 2024-03-18T01:37:50.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34900/0 (socket says 172.21.15.7:34900) 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785200640 1 -- 172.21.15.7:0/4033309974 learned_addr learned my addr 172.21.15.7:0/4033309974 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7777e00640 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394ad30 0x564e23957290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785200640 1 -- 172.21.15.7:0/4033309974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 msgr2=0x564e2395a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785200640 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785200640 1 -- 172.21.15.7:0/4033309974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394ad30 msgr2=0x564e23957290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785200640 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394ad30 0x564e23957290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785200640 1 -- 172.21.15.7:0/4033309974 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e2395a8a0 con 0x564e2394a340 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785c00640 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f7785200640 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394a340 0x564e23956d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f777000ed70 tx=0x7f7770019980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa7ba2348bc07cbd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f776ea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.917+0000 7f776ea00640 1 -- 172.21.15.7:0/4033309974 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777002f070 con 0x564e2394a340 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 1 -- 172.21.15.7:0/4033309974 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7768005500 con 0x564e2394a340 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f776ea00640 1 -- 172.21.15.7:0/4033309974 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f777000edf0 con 0x564e2394a340 2024-03-18T01:37:50.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f776ea00640 1 -- 172.21.15.7:0/4033309974 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7770027040 con 0x564e2394a340 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 10 client.25408 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 1 -- 172.21.15.7:0/4033309974 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e2395bba0 con 0x564e2394a340 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f776ea00640 1 -- 172.21.15.7:0/4033309974 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7768005500 con 0x564e2394a340 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f776ea00640 1 -- 172.21.15.7:0/4033309974 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f77700035f0 con 0x564e2394a340 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 10 client.25408 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 10 client.25408 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 20 client.25408 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x564e239cb400 0x564e239ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 4 client.25408 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.918+0000 7f778a994740 1 -- 172.21.15.7:0/4033309974 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "status"}) -- 0x564e238bc190 con 0x564e239cb400 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.919+0000 7f7777e00640 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x564e239cb400 0x564e239ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.919+0000 7f7777e00640 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x564e239cb400 0x564e239ca110 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dcb789e4b8a17e16 server_cookie=98497e4620defba2 in_seq=0 out_seq=0 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.919+0000 7f776ea00640 10 client.25408 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:37:50.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.919+0000 7f776ea00640 1 -- 172.21.15.7:0/4033309974 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564e238bc190 con 0x564e239cb400 2024-03-18T01:37:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.919+0000 7f776ea00640 10 client.25408 handle_command_reply: tid=0 2024-03-18T01:37:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.919+0000 7f778a994740 1 client.25408 shutdown 2024-03-18T01:37:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 -- 172.21.15.7:0/4033309974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394a340 msgr2=0x564e23956d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394a340 0x564e23956d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f777000ed70 tx=0x7f7770019980 comp rx=0 tx=0).stop 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 -- 172.21.15.7:0/4033309974 shutdown_connections 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564e2394fea0 0x564e2395a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564e2394ad30 0x564e23957290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x564e239cb400 0x564e239ca110 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 --2- 172.21.15.7:0/4033309974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564e2394a340 0x564e23956d50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f776ea00640 0 client.25408 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 -- 172.21.15.7:0/4033309974 >> 172.21.15.7:0/4033309974 conn(0x564e239458b0 msgr2=0x564e239441e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 -- 172.21.15.7:0/4033309974 shutdown_connections 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 1 -- 172.21.15.7:0/4033309974 wait complete. 2024-03-18T01:37:50.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.920+0000 7f778a994740 20 client.25408 trim_cache size 0 max 16384 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 2, 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24544, 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 142.580469647, 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:50.931 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 192.33175150899999 2024-03-18T01:37:50.932 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 -- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77640a2880 msgr2=0x7f77640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77640a2880 0x7f77640a4d40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7768009710 tx=0x7f7768005f90 comp rx=0 tx=0).stop 2024-03-18T01:37:50.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 -- 172.21.15.7:0/2574046732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 msgr2=0x7f7780100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 0x7f7780100620 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f777802c640 tx=0x7f7778035d90 comp rx=0 tx=0).stop 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 -- 172.21.15.7:0/2574046732 shutdown_connections 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107390 0x7f7780100620 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7780104a60 0x7f77801000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77640a2880 0x7f77640a4d40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 --2- 172.21.15.7:0/2574046732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7780102130 0x7f7780101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 -- 172.21.15.7:0/2574046732 >> 172.21.15.7:0/2574046732 conn(0x7f77800fbac0 msgr2=0x7f77801029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 -- 172.21.15.7:0/2574046732 shutdown_connections 2024-03-18T01:37:50.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:50.921+0000 7f7786600640 1 -- 172.21.15.7:0/2574046732 wait complete. 2024-03-18T01:37:50.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-03-18T01:37:51.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:50 smithi046 ceph-mon[36108]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T01:37:51.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:50 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2181418644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:37:51.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:37:51.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:37:51.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:37:51.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:50 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1499685116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:37:51.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 -- 172.21.15.7:0/1492786756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7490103820 msgr2=0x7f7490103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 --2- 172.21.15.7:0/1492786756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7490103820 0x7f7490103c40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7480009900 tx=0x7f748002bc30 comp rx=0 tx=0).stop 2024-03-18T01:37:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 -- 172.21.15.7:0/1492786756 shutdown_connections 2024-03-18T01:37:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 --2- 172.21.15.7:0/1492786756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f74901053d0 0x7f749010bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 --2- 172.21.15.7:0/1492786756 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7490104a10 0x7f7490104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 --2- 172.21.15.7:0/1492786756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7490103820 0x7f7490103c40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 -- 172.21.15.7:0/1492786756 >> 172.21.15.7:0/1492786756 conn(0x7f74900fedb0 msgr2=0x7f74901011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 -- 172.21.15.7:0/1492786756 shutdown_connections 2024-03-18T01:37:51.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.170+0000 7f748fe00640 1 -- 172.21.15.7:0/1492786756 wait complete. 2024-03-18T01:37:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.171+0000 7f748fe00640 1 Processor -- start 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748fe00640 1 -- start start 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490103820 0x7f7490199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7490104a10 0x7f749019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74901053d0 0x7f74901a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748fe00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f749010f4c0 con 0x7f74901053d0 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748fe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f749010f340 con 0x7f7490103820 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f749010f640 con 0x7f7490104a10 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490103820 0x7f7490199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490103820 0x7f7490199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34916/0 (socket says 172.21.15.7:34916) 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748ea00640 1 -- 172.21.15.7:0/2581761789 learned_addr learned my addr 172.21.15.7:0/2581761789 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.172+0000 7f748e000640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7490104a10 0x7f749019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f748f400640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74901053d0 0x7f74901a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f748ea00640 1 -- 172.21.15.7:0/2581761789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7490104a10 msgr2=0x7f749019a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f748ea00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7490104a10 0x7f749019a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f748ea00640 1 -- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74901053d0 msgr2=0x7f74901a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f748ea00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74901053d0 0x7f74901a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f748ea00640 1 -- 172.21.15.7:0/2581761789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74901a1760 con 0x7f7490103820 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f748f400640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74901053d0 0x7f74901a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f748ea00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490103820 0x7f7490199b40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7480000c00 tx=0x7f748002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.173+0000 7f7494a00640 1 -- 172.21.15.7:0/2581761789 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480043020 con 0x7f7490103820 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.174+0000 7f748fe00640 1 -- 172.21.15.7:0/2581761789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74901a19f0 con 0x7f7490103820 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.174+0000 7f7494a00640 1 -- 172.21.15.7:0/2581761789 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74800054e0 con 0x7f7490103820 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.174+0000 7f7494a00640 1 -- 172.21.15.7:0/2581761789 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480003ce0 con 0x7f7490103820 2024-03-18T01:37:51.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.174+0000 7f748fe00640 1 -- 172.21.15.7:0/2581761789 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74901a1f60 con 0x7f7490103820 2024-03-18T01:37:51.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.178+0000 7f7498b90740 1 Processor -- start 2024-03-18T01:37:51.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.178+0000 7f7498b90740 1 -- start start 2024-03-18T01:37:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.178+0000 7f7498b90740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902e340 0x55d33902e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.178+0000 7f7498b90740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902ed30 0x55d339033830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.178+0000 7f7498b90740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d339036330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.178+0000 7f7498b90740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d338f92c50 con 0x55d339033ea0 2024-03-18T01:37:51.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.178+0000 7f7498b90740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d338f3b9d0 con 0x55d33902e340 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.178+0000 7f7498b90740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d338f8ecf0 con 0x55d33902ed30 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902e340 0x55d33902e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f7494a00640 1 -- 172.21.15.7:0/2581761789 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f7480046020 con 0x7f7490103820 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902e340 0x55d33902e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34928/0 (socket says 172.21.15.7:34928) 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748f400640 1 -- 172.21.15.7:0/649621592 learned_addr learned my addr 172.21.15.7:0/649621592 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748e000640 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d339036330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748f400640 1 -- 172.21.15.7:0/649621592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902ed30 msgr2=0x55d339033830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748ea00640 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902ed30 0x55d339033830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748f400640 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902ed30 0x55d339033830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748f400640 1 -- 172.21.15.7:0/649621592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 msgr2=0x55d339036330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748f400640 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d339036330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748f400640 1 -- 172.21.15.7:0/649621592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d338f55f10 con 0x55d33902e340 2024-03-18T01:37:51.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748e000640 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d339036330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:51.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.179+0000 7f748ea00640 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902ed30 0x55d339033830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:37:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.180+0000 7f748f400640 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902e340 0x55d33902e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7488009fd0 tx=0x7f74880108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6494c287b28d4115 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.180+0000 7f747be00640 1 -- 172.21.15.7:0/649621592 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7488011760 con 0x55d33902e340 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.180+0000 7f747be00640 1 -- 172.21.15.7:0/649621592 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7488011de0 con 0x55d33902e340 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.180+0000 7f747be00640 1 -- 172.21.15.7:0/649621592 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748800e3c0 con 0x55d33902e340 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.180+0000 7f7494a00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74740a2880 0x7f74740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 -- 172.21.15.7:0/649621592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902e340 msgr2=0x55d33902e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902e340 0x55d33902e760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7488009fd0 tx=0x7f74880108e0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f748ea00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74740a2880 0x7f74740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7494a00640 1 -- 172.21.15.7:0/2581761789 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74800f0ec0 con 0x7f7490103820 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 -- 172.21.15.7:0/649621592 shutdown_connections 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d339036330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902ed30 0x55d339033830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 --2- 172.21.15.7:0/649621592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902e340 0x55d33902e760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 -- 172.21.15.7:0/649621592 >> 172.21.15.7:0/649621592 conn(0x55d3390298b0 msgr2=0x55d339029cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 -- 172.21.15.7:0/649621592 shutdown_connections 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 -- 172.21.15.7:0/649621592 wait complete. 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f7498b90740 1 -- start start 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.181+0000 7f748ea00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74740a2880 0x7f74740a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f74800ee9d0 tx=0x7f74800dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f7498b90740 1 Processor -- start 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f7498b90740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d33902e340 0x55d3390384d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f7498b90740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d339038a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f7498b90740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d339033ea0 0x55d33903fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f7498b90740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d339036e50 con 0x55d33902e340 2024-03-18T01:37:51.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f7498b90740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d338f4ee20 con 0x55d33902ed30 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f7498b90740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d339036fd0 con 0x55d339033ea0 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f7498b90740 10 client.0 _ll_register_callbacks cb 0x55d338f4e7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f748e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d339038a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f748f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d33902e340 0x55d3390384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f748e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d339038a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34944/0 (socket says 172.21.15.7:34944) 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.182+0000 7f748e000640 1 -- 172.21.15.7:0/1657765460 learned_addr learned my addr 172.21.15.7:0/1657765460 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f748f400640 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d339033ea0 0x55d33903fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f748e000640 1 -- 172.21.15.7:0/1657765460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d339033ea0 msgr2=0x55d33903fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f748e000640 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d339033ea0 0x55d33903fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f748e000640 1 -- 172.21.15.7:0/1657765460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d33902e340 msgr2=0x55d3390384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f748e000640 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d33902e340 0x55d3390384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f748e000640 1 -- 172.21.15.7:0/1657765460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d338ca72d0 con 0x55d33902ed30 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f748e000640 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d339038a10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f747c00bad0 tx=0x7f747c00bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24d3a1bb70f05bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f7478c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.183+0000 7f7478c00640 1 -- 172.21.15.7:0/1657765460 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c003e20 con 0x55d33902ed30 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 1 -- 172.21.15.7:0/1657765460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7488011de0 con 0x55d33902ed30 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7478c00640 1 -- 172.21.15.7:0/1657765460 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f747c011440 con 0x55d33902ed30 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7478c00640 1 -- 172.21.15.7:0/1657765460 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f747c011640 con 0x55d33902ed30 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 10 client.25424 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 1 -- 172.21.15.7:0/1657765460 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d338d716b0 con 0x55d33902ed30 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7478c00640 1 -- 172.21.15.7:0/1657765460 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7488011de0 con 0x55d33902ed30 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7478c00640 1 -- 172.21.15.7:0/1657765460 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f747c01ba20 con 0x55d33902ed30 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 10 client.25424 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 10 client.25424 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 20 client.25424 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:51.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d339045240 0x55d3390466b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 4 client.25424 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.184+0000 7f7498b90740 1 -- 172.21.15.7:0/1657765460 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d338ff02b0 con 0x55d339045240 2024-03-18T01:37:51.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.185+0000 7f748f400640 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d339045240 0x55d3390466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.185+0000 7f7478c00640 10 client.25424 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.185+0000 7f748f400640 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d339045240 0x55d3390466b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f236d4061265a35 server_cookie=a9815ba7314dae55 in_seq=0 out_seq=0 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7478c00640 1 -- 172.21.15.7:0/1657765460 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d338ff02b0 con 0x55d339045240 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7478c00640 10 client.25424 handle_command_reply: tid=0 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 client.25424 shutdown 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 -- 172.21.15.7:0/1657765460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 msgr2=0x55d339038a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d339038a10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f747c00bad0 tx=0x7f747c00bf90 comp rx=0 tx=0).stop 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 -- 172.21.15.7:0/1657765460 shutdown_connections 2024-03-18T01:37:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d339045240 0x55d3390466b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d339033ea0 0x55d33903fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d339038a10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 --2- 172.21.15.7:0/1657765460 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d33902e340 0x55d3390384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 -- 172.21.15.7:0/1657765460 >> 172.21.15.7:0/1657765460 conn(0x55d3390298b0 msgr2=0x55d339028a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7478c00640 0 client.25424 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:37:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 -- 172.21.15.7:0/1657765460 shutdown_connections 2024-03-18T01:37:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 1 -- 172.21.15.7:0/1657765460 wait complete. 2024-03-18T01:37:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.187+0000 7f7498b90740 20 client.25424 trim_cache size 0 max 16384 2024-03-18T01:37:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.207+0000 7f7498b90740 1 Processor -- start 2024-03-18T01:37:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.207+0000 7f7498b90740 1 -- start start 2024-03-18T01:37:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.207+0000 7f7498b90740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 0x55d3390377c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.207+0000 7f7498b90740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f7498b90740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d339029c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f7498b90740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d3390449f0 con 0x55d339033ea0 2024-03-18T01:37:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f7498b90740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d3390450c0 con 0x55d33902ed30 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f7498b90740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d339045790 con 0x55d33902e340 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34950/0 (socket says 172.21.15.7:34950) 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 -- 172.21.15.7:0/34116703 learned_addr learned my addr 172.21.15.7:0/34116703 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 -- 172.21.15.7:0/34116703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 msgr2=0x55d3390377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 --2- 172.21.15.7:0/34116703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 0x55d3390377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 -- 172.21.15.7:0/34116703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 msgr2=0x55d339029c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 --2- 172.21.15.7:0/34116703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d339029c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 -- 172.21.15.7:0/34116703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d339039600 con 0x55d33902ed30 2024-03-18T01:37:51.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.208+0000 7f748e000640 1 --2- 172.21.15.7:0/34116703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903fa10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f747c01be60 tx=0x7f747c003a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d8c247f613183f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f747b400640 1 -- 172.21.15.7:0/34116703 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c003ea0 con 0x55d33902ed30 2024-03-18T01:37:51.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f747b400640 1 -- 172.21.15.7:0/34116703 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f747c010df0 con 0x55d33902ed30 2024-03-18T01:37:51.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f747b400640 1 -- 172.21.15.7:0/34116703 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c0117e0 con 0x55d33902ed30 2024-03-18T01:37:51.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f7498b90740 1 -- 172.21.15.7:0/34116703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 msgr2=0x55d33903fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f7498b90740 1 --2- 172.21.15.7:0/34116703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903fa10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f747c01be60 tx=0x7f747c003a80 comp rx=0 tx=0).stop 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f7498b90740 1 -- 172.21.15.7:0/34116703 shutdown_connections 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f7498b90740 1 --2- 172.21.15.7:0/34116703 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d339029c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f7498b90740 1 --2- 172.21.15.7:0/34116703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903fa10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.209+0000 7f7498b90740 1 --2- 172.21.15.7:0/34116703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 0x55d3390377c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.210+0000 7f7498b90740 1 -- 172.21.15.7:0/34116703 >> 172.21.15.7:0/34116703 conn(0x55d3390298b0 msgr2=0x55d3390380e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.210+0000 7f7498b90740 1 -- 172.21.15.7:0/34116703 shutdown_connections 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.210+0000 7f7498b90740 1 -- 172.21.15.7:0/34116703 wait complete. 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.210+0000 7f7498b90740 1 -- start start 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.210+0000 7f7498b90740 1 Processor -- start 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f7498b90740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 0x55d33903b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f7498b90740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f7498b90740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d33903ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f7498b90740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55d339046530 con 0x55d339033ea0 2024-03-18T01:37:51.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f7498b90740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55d339045e60 con 0x55d33902ed30 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f7498b90740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d339046c00 con 0x55d33902e340 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f7498b90740 10 client.0 _ll_register_callbacks cb 0x7f747c0043c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34956/0 (socket says 172.21.15.7:34956) 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748e000640 1 -- 172.21.15.7:0/1731932754 learned_addr learned my addr 172.21.15.7:0/1731932754 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748f400640 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 0x55d33903b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748f400640 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d33903ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748e000640 1 -- 172.21.15.7:0/1731932754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 msgr2=0x55d33903b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748e000640 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 0x55d33903b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748e000640 1 -- 172.21.15.7:0/1731932754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 msgr2=0x55d33903ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748e000640 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d33903ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748e000640 1 -- 172.21.15.7:0/1731932754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d33903f400 con 0x55d33902ed30 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.211+0000 7f748f400640 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 0x55d33903b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:51.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.212+0000 7f748e000640 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903be40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f747c010fe0 tx=0x7f747c00b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=579bbf5985f17cd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.212+0000 7f7478c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.212+0000 7f7478c00640 1 -- 172.21.15.7:0/1731932754 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f747c012bc0 con 0x55d33902ed30 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.212+0000 7f7478c00640 1 -- 172.21.15.7:0/1731932754 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f747c012d60 con 0x55d33902ed30 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.212+0000 7f7498b90740 1 -- 172.21.15.7:0/1731932754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f747c010df0 con 0x55d33902ed30 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.212+0000 7f7478c00640 1 -- 172.21.15.7:0/1731932754 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f747c0104e0 con 0x55d33902ed30 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.212+0000 7f7498b90740 10 client.25430 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.212+0000 7f7498b90740 1 -- 172.21.15.7:0/1731932754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d3390ab710 con 0x55d33902ed30 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f7478c00640 1 -- 172.21.15.7:0/1731932754 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f747c010df0 con 0x55d33902ed30 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f7478c00640 1 -- 172.21.15.7:0/1731932754 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f747c01b9d0 con 0x55d33902ed30 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f7498b90740 10 client.25430 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f7498b90740 10 client.25430 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f7498b90740 20 client.25430 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f7498b90740 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d3390afdf0 0x55d3390aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f7498b90740 4 client.25430 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f7498b90740 1 -- 172.21.15.7:0/1731932754 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "status"}) -- 0x55d338fa0190 con 0x55d3390afdf0 2024-03-18T01:37:51.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.213+0000 7f748f400640 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d3390afdf0 0x55d3390aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f748f400640 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d3390afdf0 0x55d3390aeb00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ebf74a58c08a8b42 server_cookie=3f8ed888f8197718 in_seq=0 out_seq=0 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7478c00640 10 client.25430 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7478c00640 1 -- 172.21.15.7:0/1731932754 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55d338fa0190 con 0x55d3390afdf0 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7478c00640 10 client.25430 handle_command_reply: tid=0 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 client.25430 shutdown 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 -- 172.21.15.7:0/1731932754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 msgr2=0x55d33903be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903be40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f747c010fe0 tx=0x7f747c00b5b0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 -- 172.21.15.7:0/1731932754 shutdown_connections 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x55d3390afdf0 0x55d3390aeb00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55d339033ea0 0x55d33903ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d33902ed30 0x55d33903be40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 --2- 172.21.15.7:0/1731932754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d33902e340 0x55d33903b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7478c00640 0 client.25430 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.214+0000 7f7498b90740 1 -- 172.21.15.7:0/1731932754 >> 172.21.15.7:0/1731932754 conn(0x55d3390298b0 msgr2=0x55d3390281e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.215+0000 7f7498b90740 1 -- 172.21.15.7:0/1731932754 shutdown_connections 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.215+0000 7f7498b90740 1 -- 172.21.15.7:0/1731932754 wait complete. 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.215+0000 7f7498b90740 20 client.25430 trim_cache size 0 max 16384 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:51.220 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 1, 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24562, 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 143.888332101, 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 188.257985099 2024-03-18T01:37:51.221 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:51.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.215+0000 7f748fe00640 1 -- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74740a2880 msgr2=0x7f74740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.215+0000 7f748fe00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74740a2880 0x7f74740a4d40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f74800ee9d0 tx=0x7f74800dd040 comp rx=0 tx=0).stop 2024-03-18T01:37:51.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.215+0000 7f748fe00640 1 -- 172.21.15.7:0/2581761789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490103820 msgr2=0x7f7490199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.215+0000 7f748fe00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490103820 0x7f7490199b40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7480000c00 tx=0x7f748002c6c0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.215+0000 7f748fe00640 1 -- 172.21.15.7:0/2581761789 shutdown_connections 2024-03-18T01:37:51.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.216+0000 7f748fe00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f74740a2880 0x7f74740a4d40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.216+0000 7f748fe00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f74901053d0 0x7f74901a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.216+0000 7f748fe00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7490104a10 0x7f749019a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.216+0000 7f748fe00640 1 --2- 172.21.15.7:0/2581761789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7490103820 0x7f7490199b40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.216+0000 7f748fe00640 1 -- 172.21.15.7:0/2581761789 >> 172.21.15.7:0/2581761789 conn(0x7f74900fedb0 msgr2=0x7f7490107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.216+0000 7f748fe00640 1 -- 172.21.15.7:0/2581761789 shutdown_connections 2024-03-18T01:37:51.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.216+0000 7f748fe00640 1 -- 172.21.15.7:0/2581761789 wait complete. 2024-03-18T01:37:51.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T01:37:51.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 -- 172.21.15.7:0/104680654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac103200 msgr2=0x7f37ac109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 --2- 172.21.15.7:0/104680654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac103200 0x7f37ac109a90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009ca0 tx=0x7f37a802c100 comp rx=0 tx=0).stop 2024-03-18T01:37:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 -- 172.21.15.7:0/104680654 shutdown_connections 2024-03-18T01:37:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 --2- 172.21.15.7:0/104680654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac103200 0x7f37ac109a90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 --2- 172.21.15.7:0/104680654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ac102840 0x7f37ac102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 --2- 172.21.15.7:0/104680654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ac101650 0x7f37ac101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 -- 172.21.15.7:0/104680654 >> 172.21.15.7:0/104680654 conn(0x7f37ac0fcbc0 msgr2=0x7f37ac0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 -- 172.21.15.7:0/104680654 shutdown_connections 2024-03-18T01:37:51.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.478+0000 7f37b3200640 1 -- 172.21.15.7:0/104680654 wait complete. 2024-03-18T01:37:51.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.479+0000 7f37b3200640 1 Processor -- start 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.479+0000 7f37b3200640 1 -- start start 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b3200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ac101650 0x7f37ac107440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b3200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac102840 0x7f37ac107980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b3200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ac103200 0x7f37ac103fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b3200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac10d2b0 con 0x7f37ac101650 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b3200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac10d130 con 0x7f37ac103200 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b3200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac10d430 con 0x7f37ac102840 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac102840 0x7f37ac107980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac102840 0x7f37ac107980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59682/0 (socket says 172.21.15.7:59682) 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1400640 1 -- 172.21.15.7:0/3252163778 learned_addr learned my addr 172.21.15.7:0/3252163778 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1e00640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ac101650 0x7f37ac107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b2800640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ac103200 0x7f37ac103fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1400640 1 -- 172.21.15.7:0/3252163778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ac103200 msgr2=0x7f37ac103fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1400640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ac103200 0x7f37ac103fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1400640 1 -- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ac101650 msgr2=0x7f37ac107440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1400640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ac101650 0x7f37ac107440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b1400640 1 -- 172.21.15.7:0/3252163778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ac104870 con 0x7f37ac102840 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.480+0000 7f37b2800640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ac103200 0x7f37ac103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.481+0000 7f37b1400640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac102840 0x7f37ac107980 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f379c00e9f0 tx=0x7f379c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.481+0000 7f37a6a00640 1 -- 172.21.15.7:0/3252163778 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c00cd80 con 0x7f37ac102840 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.481+0000 7f37a6a00640 1 -- 172.21.15.7:0/3252163778 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f379c010910 con 0x7f37ac102840 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.481+0000 7f37b3200640 1 -- 172.21.15.7:0/3252163778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ac1a5f00 con 0x7f37ac102840 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.481+0000 7f37a6a00640 1 -- 172.21.15.7:0/3252163778 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c018990 con 0x7f37ac102840 2024-03-18T01:37:51.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.481+0000 7f37b3200640 1 -- 172.21.15.7:0/3252163778 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ac1a62b0 con 0x7f37ac102840 2024-03-18T01:37:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.484+0000 7f37b75f4740 1 Processor -- start 2024-03-18T01:37:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.484+0000 7f37b75f4740 1 -- start start 2024-03-18T01:37:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.484+0000 7f37a6a00640 1 -- 172.21.15.7:0/3252163778 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f379c010ac0 con 0x7f37ac102840 2024-03-18T01:37:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196340 0x564cd4196760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd419b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd419e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b75f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564cd40fac50 con 0x564cd419bea0 2024-03-18T01:37:51.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b75f4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564cd40a39d0 con 0x564cd4196340 2024-03-18T01:37:51.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b75f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564cd40f6cf0 con 0x564cd4196d30 2024-03-18T01:37:51.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196340 0x564cd4196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196340 0x564cd4196760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34988/0 (socket says 172.21.15.7:34988) 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b2800640 1 -- 172.21.15.7:0/2159824656 learned_addr learned my addr 172.21.15.7:0/2159824656 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd419b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.485+0000 7f37b1400640 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd419e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37b1e00640 1 -- 172.21.15.7:0/2159824656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196340 msgr2=0x564cd4196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196340 0x564cd4196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37b1e00640 1 -- 172.21.15.7:0/2159824656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 msgr2=0x564cd419e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd419e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37b1e00640 1 -- 172.21.15.7:0/2159824656 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cd40bdf10 con 0x564cd4196d30 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37b1400640 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd419e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd419b830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009bf0 tx=0x7f37a0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42236dc13f68c73a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f379b000640 1 -- 172.21.15.7:0/2159824656 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0009da0 con 0x564cd4196d30 2024-03-18T01:37:51.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37a6a00640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f37940a2800 0x7f37940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f379b000640 1 -- 172.21.15.7:0/2159824656 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a0005020 con 0x564cd4196d30 2024-03-18T01:37:51.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f379b000640 1 -- 172.21.15.7:0/2159824656 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0004b60 con 0x564cd4196d30 2024-03-18T01:37:51.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37a6a00640 1 -- 172.21.15.7:0/3252163778 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f379c0cf150 con 0x7f37ac102840 2024-03-18T01:37:51.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.486+0000 7f37b2800640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f37940a2800 0x7f37940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 -- 172.21.15.7:0/2159824656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 msgr2=0x564cd419b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd419b830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009bf0 tx=0x7f37a0011810 comp rx=0 tx=0).stop 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 -- 172.21.15.7:0/2159824656 shutdown_connections 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd419e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd419b830 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2159824656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196340 0x564cd4196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 -- 172.21.15.7:0/2159824656 >> 172.21.15.7:0/2159824656 conn(0x564cd41918b0 msgr2=0x564cd4191cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b2800640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f37940a2800 0x7f37940a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f37a8032420 tx=0x7f37a8032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 -- 172.21.15.7:0/2159824656 shutdown_connections 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.487+0000 7f37b75f4740 1 -- 172.21.15.7:0/2159824656 wait complete. 2024-03-18T01:37:51.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.488+0000 7f37b75f4740 1 -- start start 2024-03-18T01:37:51.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.488+0000 7f37b75f4740 1 Processor -- start 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd41a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196d30 0x564cd41a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd41a79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b75f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564cd419ee50 con 0x564cd419bea0 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b75f4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564cd40b6e20 con 0x564cd4196d30 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b75f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564cd419efd0 con 0x564cd4196340 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b75f4740 10 client.0 _ll_register_callbacks cb 0x564cd40b67f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd41a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196d30 0x564cd41a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b2800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd41a79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196d30 0x564cd41a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:35000/0 (socket says 172.21.15.7:35000) 2024-03-18T01:37:51.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd41a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59700/0 (socket says 172.21.15.7:59700) 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.489+0000 7f37b1400640 1 -- 172.21.15.7:0/3324358100 learned_addr learned my addr 172.21.15.7:0/3324358100 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b1400640 1 -- 172.21.15.7:0/3324358100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 msgr2=0x564cd41a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b1400640 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd41a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b1400640 1 -- 172.21.15.7:0/3324358100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 msgr2=0x564cd41a79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b1400640 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd41a79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b1400640 1 -- 172.21.15.7:0/3324358100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cd3e0f2d0 con 0x564cd4196d30 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b1e00640 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd41a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b1400640 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196d30 0x564cd41a09a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f379c097c40 tx=0x7f379c09b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=471ace66455f2b04 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f378f400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f378f400640 1 -- 172.21.15.7:0/3324358100 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c09b7e0 con 0x564cd4196d30 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f378f400640 1 -- 172.21.15.7:0/3324358100 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f379c09f400 con 0x564cd4196d30 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b75f4740 1 -- 172.21.15.7:0/3324358100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37a0005020 con 0x564cd4196d30 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f378f400640 1 -- 172.21.15.7:0/3324358100 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f379c09f880 con 0x564cd4196d30 2024-03-18T01:37:51.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b75f4740 10 client.25442 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.490+0000 7f37b75f4740 1 -- 172.21.15.7:0/3324358100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564cd3ed96b0 con 0x564cd4196d30 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f378f400640 1 -- 172.21.15.7:0/3324358100 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a0005020 con 0x564cd4196d30 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f378f400640 1 -- 172.21.15.7:0/3324358100 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f379c018bc0 con 0x564cd4196d30 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f37b75f4740 10 client.25442 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f37b75f4740 10 client.25442 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f37b75f4740 20 client.25442 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f37b75f4740 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564cd41ad240 0x564cd41ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f37b75f4740 4 client.25442 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f37b75f4740 1 -- 172.21.15.7:0/3324358100 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cd41582b0 con 0x564cd41ad240 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.491+0000 7f37b1e00640 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564cd41ad240 0x564cd41ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.492+0000 7f37b1e00640 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564cd41ad240 0x564cd41ae6b0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c33190b2c983988 server_cookie=46f17c70947cd963 in_seq=0 out_seq=0 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.492+0000 7f378f400640 10 client.25442 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f378f400640 1 -- 172.21.15.7:0/3324358100 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564cd41582b0 con 0x564cd41ad240 2024-03-18T01:37:51.497 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f378f400640 10 client.25442 handle_command_reply: tid=0 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 client.25442 shutdown 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 -- 172.21.15.7:0/3324358100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196d30 msgr2=0x564cd41a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196d30 0x564cd41a09a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f379c097c40 tx=0x7f379c09b5b0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 -- 172.21.15.7:0/3324358100 shutdown_connections 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564cd41ad240 0x564cd41ae6b0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd419bea0 0x564cd41a79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd4196d30 0x564cd41a09a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 --2- 172.21.15.7:0/3324358100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd41a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f378f400640 0 client.25442 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.493+0000 7f37b75f4740 1 -- 172.21.15.7:0/3324358100 >> 172.21.15.7:0/3324358100 conn(0x564cd41918b0 msgr2=0x564cd4190a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.494+0000 7f37b75f4740 1 -- 172.21.15.7:0/3324358100 shutdown_connections 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.494+0000 7f37b75f4740 1 -- 172.21.15.7:0/3324358100 wait complete. 2024-03-18T01:37:51.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.494+0000 7f37b75f4740 20 client.25442 trim_cache size 0 max 16384 2024-03-18T01:37:51.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.517+0000 7f37b75f4740 1 Processor -- start 2024-03-18T01:37:51.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.517+0000 7f37b75f4740 1 -- start start 2024-03-18T01:37:51.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.517+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd419e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.517+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196d30 0x564cd41a79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.517+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 0x564cd418c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.517+0000 7f37b75f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564cd41ac9f0 con 0x564cd4196d30 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.517+0000 7f37b75f4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564cd41ad0c0 con 0x564cd419bea0 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.517+0000 7f37b75f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564cd41ad790 con 0x564cd4196340 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd419e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd419e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59710/0 (socket says 172.21.15.7:59710) 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 -- 172.21.15.7:0/2094553593 learned_addr learned my addr 172.21.15.7:0/2094553593 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b2800640 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 0x564cd418c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1400640 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196d30 0x564cd41a79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 -- 172.21.15.7:0/2094553593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 msgr2=0x564cd418c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 0x564cd418c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 -- 172.21.15.7:0/2094553593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196d30 msgr2=0x564cd41a79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196d30 0x564cd41a79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 -- 172.21.15.7:0/2094553593 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cd4192620 con 0x564cd4196340 2024-03-18T01:37:51.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1400640 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196d30 0x564cd41a79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:51.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd419e290 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f37a0014040 tx=0x7f37a0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8597ead2db15ecec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.518+0000 7f379a600640 1 -- 172.21.15.7:0/2094553593 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0005560 con 0x564cd4196340 2024-03-18T01:37:51.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f379a600640 1 -- 172.21.15.7:0/2094553593 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a0003ad0 con 0x564cd4196340 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f379a600640 1 -- 172.21.15.7:0/2094553593 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0019a90 con 0x564cd4196340 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 -- 172.21.15.7:0/2094553593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 msgr2=0x564cd419e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd419e290 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f37a0014040 tx=0x7f37a0009b60 comp rx=0 tx=0).stop 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 -- 172.21.15.7:0/2094553593 shutdown_connections 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 0x564cd418c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196d30 0x564cd41a79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2094553593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196340 0x564cd419e290 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 -- 172.21.15.7:0/2094553593 >> 172.21.15.7:0/2094553593 conn(0x564cd41918b0 msgr2=0x564cd419fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 -- 172.21.15.7:0/2094553593 shutdown_connections 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.519+0000 7f37b75f4740 1 -- 172.21.15.7:0/2094553593 wait complete. 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.520+0000 7f37b75f4740 1 -- start start 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.520+0000 7f37b75f4740 1 Processor -- start 2024-03-18T01:37:51.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196340 0x564cd41a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd41a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b75f4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 0x564cd4210fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b75f4740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564cd41ae530 con 0x564cd4196340 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b75f4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564cd41ade60 con 0x564cd419bea0 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b75f4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564cd41aec00 con 0x564cd4196d30 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b75f4740 10 client.0 _ll_register_callbacks cb 0x7f379c097a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196340 0x564cd41a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196340 0x564cd41a2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48486/0 (socket says 172.21.15.7:48486) 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1e00640 1 -- 172.21.15.7:0/2109513670 learned_addr learned my addr 172.21.15.7:0/2109513670 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1400640 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd41a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b2800640 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 0x564cd4210fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1e00640 1 -- 172.21.15.7:0/2109513670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 msgr2=0x564cd41a2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd41a2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1e00640 1 -- 172.21.15.7:0/2109513670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 msgr2=0x564cd4210fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 0x564cd4210fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1e00640 1 -- 172.21.15.7:0/2109513670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cd4211590 con 0x564cd4196340 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.521+0000 7f37b1400640 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd41a2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f37b1e00640 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196340 0x564cd41a2a50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009ee0 tx=0x7f37a0019500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee16052e17311a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f378f400640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f378f400640 1 -- 172.21.15.7:0/2109513670 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0003c80 con 0x564cd4196340 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f378f400640 1 -- 172.21.15.7:0/2109513670 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f37a00035f0 con 0x564cd4196340 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f37b75f4740 1 -- 172.21.15.7:0/2109513670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37a0003ad0 con 0x564cd4196340 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f378f400640 1 -- 172.21.15.7:0/2109513670 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f37a0003810 con 0x564cd4196340 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f37b75f4740 10 client.15300 fetch_fsmap learned FSMap version 27 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f37b75f4740 1 -- 172.21.15.7:0/2109513670 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564cd4212890 con 0x564cd4196340 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.522+0000 7f378f400640 1 -- 172.21.15.7:0/2109513670 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37a0003ad0 con 0x564cd4196340 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.523+0000 7f378f400640 1 -- 172.21.15.7:0/2109513670 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f37a003b030 con 0x564cd4196340 2024-03-18T01:37:51.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.523+0000 7f37b75f4740 10 client.15300 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.523+0000 7f37b75f4740 10 client.15300 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.523+0000 7f37b75f4740 20 client.15300 populate_metadata read hostname 'smithi007' 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.523+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564cd4216f40 0x564cd4215c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.523+0000 7f37b75f4740 4 client.15300 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.523+0000 7f37b75f4740 1 -- 172.21.15.7:0/2109513670 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "status"}) -- 0x564cd4108190 con 0x564cd4216f40 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.523+0000 7f37b1400640 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564cd4216f40 0x564cd4215c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b1400640 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564cd4216f40 0x564cd4215c50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66b77d2e161a6ae7 server_cookie=84d63e8807f24e0 in_seq=0 out_seq=0 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f378f400640 10 client.15300 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f378f400640 1 -- 172.21.15.7:0/2109513670 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x564cd4108190 con 0x564cd4216f40 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f378f400640 10 client.15300 handle_command_reply: tid=0 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 client.15300 shutdown 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 -- 172.21.15.7:0/2109513670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196340 msgr2=0x564cd41a2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196340 0x564cd41a2a50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f37a0009ee0 tx=0x7f37a0019500 comp rx=0 tx=0).stop 2024-03-18T01:37:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 -- 172.21.15.7:0/2109513670 shutdown_connections 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564cd4216f40 0x564cd4215c50 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cd419bea0 0x564cd4210fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564cd4196d30 0x564cd41a2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 --2- 172.21.15.7:0/2109513670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564cd4196340 0x564cd41a2a50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 -- 172.21.15.7:0/2109513670 >> 172.21.15.7:0/2109513670 conn(0x564cd41918b0 msgr2=0x564cd418c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f378f400640 0 client.15300 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 -- 172.21.15.7:0/2109513670 shutdown_connections 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.524+0000 7f37b75f4740 1 -- 172.21.15.7:0/2109513670 wait complete. 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b75f4740 20 client.15300 trim_cache size 0 max 16384 2024-03-18T01:37:51.530 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "cluster_fsid": "3fdfdc7c-e4c6-11ee-95c9-87774f69a715", 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "whoami": 0, 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "id": 24596, 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "want_state": "up:active", 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "state": "up:active", 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "fs_name": "cephfs", 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "rank_uptime": 174.47310575, 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "mdsmap_epoch": 27, 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch": 91, 2024-03-18T01:37:51.531 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T01:37:51.533 INFO:teuthology.orchestra.run.smithi007.stdout: "uptime": 181.28932814999999 2024-03-18T01:37:51.533 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:37:51.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 -- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f37940a2800 msgr2=0x7f37940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f37940a2800 0x7f37940a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f37a8032420 tx=0x7f37a8032590 comp rx=0 tx=0).stop 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 -- 172.21.15.7:0/3252163778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac102840 msgr2=0x7f37ac107980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac102840 0x7f37ac107980 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f379c00e9f0 tx=0x7f379c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 -- 172.21.15.7:0/3252163778 shutdown_connections 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ac103200 0x7f37ac103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37ac102840 0x7f37ac107980 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f37940a2800 0x7f37940a4cc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 --2- 172.21.15.7:0/3252163778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37ac101650 0x7f37ac107440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 -- 172.21.15.7:0/3252163778 >> 172.21.15.7:0/3252163778 conn(0x7f37ac0fcbc0 msgr2=0x7f37ac073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 -- 172.21.15.7:0/3252163778 shutdown_connections 2024-03-18T01:37:51.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:51.525+0000 7f37b3200640 1 -- 172.21.15.7:0/3252163778 wait complete. 2024-03-18T01:37:51.544 INFO:teuthology.run_tasks:Running task kclient... 2024-03-18T01:37:51.558 INFO:tasks.kclient:Mounting kernel clients... 2024-03-18T01:37:51.558 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-18T01:37:51.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:51.558 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-03-18T01:37:51.605 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-18T01:37:51.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:51.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T01:37:51.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:51.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2024-03-18T01:37:51.866 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-18T01:37:51.866 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T01:37:51.866 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T01:37:51.866 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T01:37:52.301 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:37:52.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:52 smithi007 ceph-mon[31996]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:53.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:52 smithi046 ceph-mon[36108]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:53.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:52 smithi203 ceph-mon[36345]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:37:54.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.704+0000 7f5bd1800700 1 -- 172.21.15.7:0/4155621452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc107630 msgr2=0x7f5bcc10f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:54.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.704+0000 7f5bd1800700 1 --2- 172.21.15.7:0/4155621452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc107630 0x7f5bcc10f200 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4009fc0 tx=0x7f5bc400b330 comp rx=0 tx=0).stop 2024-03-18T01:37:54.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.705+0000 7f5bd1800700 1 -- 172.21.15.7:0/4155621452 shutdown_connections 2024-03-18T01:37:54.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.705+0000 7f5bd1800700 1 --2- 172.21.15.7:0/4155621452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc107630 0x7f5bcc10f200 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:54.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.705+0000 7f5bd1800700 1 --2- 172.21.15.7:0/4155621452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bcc102550 0x7f5bcc102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:54.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.705+0000 7f5bd1800700 1 --2- 172.21.15.7:0/4155621452 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc102f00 0x7f5bcc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:54.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.705+0000 7f5bd1800700 1 -- 172.21.15.7:0/4155621452 >> 172.21.15.7:0/4155621452 conn(0x7f5bcc0fdda0 msgr2=0x7f5bcc1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:54.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.705+0000 7f5bd1800700 1 -- 172.21.15.7:0/4155621452 shutdown_connections 2024-03-18T01:37:54.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.705+0000 7f5bd1800700 1 -- 172.21.15.7:0/4155621452 wait complete. 2024-03-18T01:37:54.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.707+0000 7f5bd1800700 1 Processor -- start 2024-03-18T01:37:54.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.707+0000 7f5bd1800700 1 -- start start 2024-03-18T01:37:54.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.707+0000 7f5bd1800700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc102550 0x7f5bcc0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.707+0000 7f5bd1800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bcc102f00 0x7f5bcc074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.707+0000 7f5bd1800700 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc107630 0x7f5bcc075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bd1800700 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc112670 con 0x7f5bcc107630 2024-03-18T01:37:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bd1800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc1124f0 con 0x7f5bcc102f00 2024-03-18T01:37:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bd1800700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc1127f0 con 0x7f5bcc102550 2024-03-18T01:37:54.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcaa00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc102550 0x7f5bcc0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcaa00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc102550 0x7f5bcc0745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59742/0 (socket says 172.21.15.7:59742) 2024-03-18T01:37:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcaa00700 1 -- 172.21.15.7:0/2392746106 learned_addr learned my addr 172.21.15.7:0/2392746106 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:54.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bca000700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bcc102f00 0x7f5bcc074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:54.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcb400700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc107630 0x7f5bcc075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:54.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcaa00700 1 -- 172.21.15.7:0/2392746106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bcc102f00 msgr2=0x7f5bcc074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:54.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcaa00700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bcc102f00 0x7f5bcc074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:54.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcaa00700 1 -- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc107630 msgr2=0x7f5bcc075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:54.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcaa00700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc107630 0x7f5bcc075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:54.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcaa00700 1 -- 172.21.15.7:0/2392746106 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bcc1a7000 con 0x7f5bcc102550 2024-03-18T01:37:54.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.708+0000 7f5bcb400700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc107630 0x7f5bcc075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:37:54.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.709+0000 7f5bcaa00700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc102550 0x7f5bcc0745d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00e9a0 tx=0x7f5bbc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:54.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.709+0000 7f5bc3400700 1 -- 172.21.15.7:0/2392746106 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc00cbd0 con 0x7f5bcc102550 2024-03-18T01:37:54.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.709+0000 7f5bc3400700 1 -- 172.21.15.7:0/2392746106 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bbc00cd70 con 0x7f5bcc102550 2024-03-18T01:37:54.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.709+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bcc1a72f0 con 0x7f5bcc102550 2024-03-18T01:37:54.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.709+0000 7f5bc3400700 1 -- 172.21.15.7:0/2392746106 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc018750 con 0x7f5bcc102550 2024-03-18T01:37:54.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.709+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bcc1a7880 con 0x7f5bcc102550 2024-03-18T01:37:54.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.711+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bcc068f90 con 0x7f5bcc102550 2024-03-18T01:37:54.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.714+0000 7f5bc3400700 1 -- 172.21.15.7:0/2392746106 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f5bbc010430 con 0x7f5bcc102550 2024-03-18T01:37:54.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.715+0000 7f5bc3400700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb00a2910 0x7f5bb00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:54.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.715+0000 7f5bc3400700 1 -- 172.21.15.7:0/2392746106 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5bbc0cf0b0 con 0x7f5bcc102550 2024-03-18T01:37:54.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.716+0000 7f5bca000700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb00a2910 0x7f5bb00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:54.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.716+0000 7f5bca000700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb00a2910 0x7f5bb00a4dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009750 tx=0x7f5bb8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:54.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:54.719+0000 7f5bc3400700 1 -- 172.21.15.7:0/2392746106 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5bbc09bd90 con 0x7f5bcc102550 2024-03-18T01:37:55.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:54 smithi046 ceph-mon[36108]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:55.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:54 smithi007 ceph-mon[31996]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:55.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.090+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5bcc02db70 con 0x7f5bcc102550 2024-03-18T01:37:55.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.091+0000 7f5bc3400700 1 -- 172.21.15.7:0/2392746106 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7f5bbc09b690 con 0x7f5bcc102550 2024-03-18T01:37:55.092 INFO:teuthology.orchestra.run.smithi007.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T01:37:55.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.094+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb00a2910 msgr2=0x7f5bb00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:55.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.094+0000 7f5bd1800700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb00a2910 0x7f5bb00a4dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009750 tx=0x7f5bb8009310 comp rx=0 tx=0).stop 2024-03-18T01:37:55.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.094+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc102550 msgr2=0x7f5bcc0745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:55.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.094+0000 7f5bd1800700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc102550 0x7f5bcc0745d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00e9a0 tx=0x7f5bbc00ecb0 comp rx=0 tx=0).stop 2024-03-18T01:37:55.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.095+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 shutdown_connections 2024-03-18T01:37:55.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.095+0000 7f5bd1800700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5bb00a2910 0x7f5bb00a4dd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:55.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.095+0000 7f5bd1800700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5bcc107630 0x7f5bcc075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:55.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.095+0000 7f5bd1800700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5bcc102550 0x7f5bcc0745d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:55.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.095+0000 7f5bd1800700 1 --2- 172.21.15.7:0/2392746106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bcc102f00 0x7f5bcc074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:55.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.095+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 >> 172.21.15.7:0/2392746106 conn(0x7f5bcc0fdda0 msgr2=0x7f5bcc0fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:55.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.095+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 shutdown_connections 2024-03-18T01:37:55.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:55.095+0000 7f5bd1800700 1 -- 172.21.15.7:0/2392746106 wait complete. 2024-03-18T01:37:55.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:54 smithi203 ceph-mon[36345]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:37:55.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:55 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2392746106' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:37:55.999 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T01:37:55.999 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T01:37:55.999 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi007.front.sepia.ceph.com 2024-03-18T01:37:56.000 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T01:37:56.000 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T01:37:56.000 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T01:37:56.000 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T01:37:56.000 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T01:37:56.000 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T01:37:56.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:56.000 DEBUG:teuthology.orchestra.run.smithi007:> ip addr 2024-03-18T01:37:56.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:55 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2392746106' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:37:56.060 INFO:teuthology.orchestra.run.smithi007.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 ::1/128 scope host 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dc brd ff:ff:ff:ff:ff:ff 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f0 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:6b:fd:dd brd ff:ff:ff:ff:ff:ff 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp7s0f1 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:ea brd ff:ff:ff:ff:ff:ff 2024-03-18T01:37:56.061 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f0 2024-03-18T01:37:56.062 INFO:teuthology.orchestra.run.smithi007.stdout: inet 172.21.15.7/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T01:37:56.062 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft 41799sec preferred_lft 41799sec 2024-03-18T01:37:56.062 INFO:teuthology.orchestra.run.smithi007.stdout: inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-03-18T01:37:56.062 INFO:teuthology.orchestra.run.smithi007.stdout: valid_lft forever preferred_lft forever 2024-03-18T01:37:56.062 INFO:teuthology.orchestra.run.smithi007.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T01:37:56.062 INFO:teuthology.orchestra.run.smithi007.stdout: link/ether 0c:c4:7a:bd:15:eb brd ff:ff:ff:ff:ff:ff 2024-03-18T01:37:56.062 INFO:teuthology.orchestra.run.smithi007.stdout: altname enp1s0f1 2024-03-18T01:37:56.063 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T01:37:56.063 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:37:56.063 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:37:56.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add name ceph-brx type bridge 2024-03-18T01:37:56.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr flush dev ceph-brx 2024-03-18T01:37:56.063 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set ceph-brx up 2024-03-18T01:37:56.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T01:37:56.064 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:37:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:55 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2392746106' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T01:37:56.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:37:56.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:56.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:56.372 DEBUG:teuthology.orchestra.run.smithi007:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T01:37:56.447 INFO:teuthology.orchestra.run.smithi007.stdout:1 2024-03-18T01:37:56.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:56.451 DEBUG:teuthology.orchestra.run.smithi007:> route 2024-03-18T01:37:56.497 INFO:teuthology.orchestra.run.smithi007.stdout:Kernel IP routing table 2024-03-18T01:37:56.497 INFO:teuthology.orchestra.run.smithi007.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T01:37:56.497 INFO:teuthology.orchestra.run.smithi007.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T01:37:56.497 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T01:37:56.497 INFO:teuthology.orchestra.run.smithi007.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T01:37:56.498 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:37:56.498 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:37:56.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T01:37:56.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T01:37:56.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T01:37:56.498 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:37:56.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:37:56.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:56.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:56.796 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-03-18T01:37:56.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:56.845 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list-id 2024-03-18T01:37:56.893 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:37:56.893 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:37:56.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T01:37:56.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T01:37:56.893 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:37:56.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:37:57.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:56 smithi046 ceph-mon[36108]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:57.025 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:56 smithi007 ceph-mon[31996]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:57.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:57.088 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T01:37:57.088 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:37:57.088 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:37:57.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T01:37:57.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T01:37:57.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T01:37:57.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T01:37:57.088 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T01:37:57.088 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:37:57.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:56 smithi203 ceph-mon[36345]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:37:57.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:37:57.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:57.438 DEBUG:teuthology.orchestra.run.smithi007:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T01:37:57.438 DEBUG:teuthology.orchestra.run.smithi007:> set -e 2024-03-18T01:37:57.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set brx.0 up 2024-03-18T01:37:57.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T01:37:57.438 DEBUG:teuthology.orchestra.run.smithi007:> ') 2024-03-18T01:37:57.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:37:57.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:57.671 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T01:37:57.671 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T01:37:57.719 INFO:teuthology.orchestra.run.smithi007.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T01:37:57.719 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T01:37:57.719 DEBUG:teuthology.orchestra.run.smithi007:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T01:37:57.766 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-18T01:37:57.766 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-18T01:37:57.766 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/e5720a72-fcfe-434a-81cc-b7152c6f2925 2024-03-18T01:37:57.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:57.766 DEBUG:teuthology.orchestra.run.smithi007:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/e5720a72-fcfe-434a-81cc-b7152c6f2925 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-18T01:37:57.945 INFO:teuthology.orchestra.run.smithi007.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-18T01:37:57.945 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-18T01:37:57.945 INFO:teuthology.orchestra.run.smithi007.stdout:invalid new device string format 2024-03-18T01:37:58.675 INFO:teuthology.orchestra.run.smithi007.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-18T01:37:58.676 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-18T01:37:58.676 INFO:teuthology.orchestra.run.smithi007.stdout:invalid new device string format 2024-03-18T01:37:58.676 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: resolved to: "172.21.15.7:3300,172.21.15.46:3300,172.21.15.203:3300" 2024-03-18T01:37:58.676 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.7:3300,172.21.15.46:3300,172.21.15.203:3300:/volumes/qa/sv_1/e5720a72-fcfe-434a-81cc-b7152c6f2925 2024-03-18T01:37:58.676 INFO:teuthology.orchestra.run.smithi007.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=3fdfdc7c-e4c6-11ee-95c9-87774f69a715" will pass to kernel 2024-03-18T01:37:58.679 INFO:teuthology.orchestra.run.smithi007.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-18T01:37:58.679 INFO:teuthology.orchestra.run.smithi007.stdout: You just mounted a file system that supports labels which does not 2024-03-18T01:37:58.679 INFO:teuthology.orchestra.run.smithi007.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-18T01:37:58.679 INFO:teuthology.orchestra.run.smithi007.stdout: applications will generate AVC messages and not be allowed access to 2024-03-18T01:37:58.679 INFO:teuthology.orchestra.run.smithi007.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-18T01:37:58.683 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-18T01:37:58.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:37:58.684 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T01:37:58.781 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:37:58.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:37:58.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:58.902 INFO:teuthology.orchestra.run.smithi007.stdout:client25447 2024-03-18T01:37:58.903 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:37:58.903 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:37:58.960 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:37:59.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:37:58 smithi046 ceph-mon[36108]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T01:37:59.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:37:58 smithi007 ceph-mon[31996]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T01:37:59.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:37:59.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:59.086 INFO:teuthology.orchestra.run.smithi007.stdout:client25447 2024-03-18T01:37:59.086 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T01:37:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:37:58 smithi203 ceph-mon[36345]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T01:37:59.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T01:37:59.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:59.210 INFO:teuthology.orchestra.run.smithi007.stdout:client25447 2024-03-18T01:37:59.210 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715.client25447/status') 2024-03-18T01:37:59.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T01:37:59.361 INFO:teuthology.orchestra.run.smithi007.stdout:instance: client.25447 (3)172.21.15.7:0/685562138 2024-03-18T01:37:59.361 INFO:teuthology.orchestra.run.smithi007.stdout:blocklisted: false 2024-03-18T01:37:59.361 INFO:teuthology.orchestra.run.smithi007.stderr:0+1 records in 2024-03-18T01:37:59.362 INFO:teuthology.orchestra.run.smithi007.stderr:0+1 records out 2024-03-18T01:37:59.362 INFO:teuthology.orchestra.run.smithi007.stderr:69 bytes copied, 8.2121e-05 s, 840 kB/s 2024-03-18T01:37:59.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T01:37:59.375 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T01:37:59.386 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T01:37:59.386 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-03-18T01:37:59.387 DEBUG:teuthology.orchestra.run.smithi007:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-18T01:37:59.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.589+0000 7fe752400640 1 -- 172.21.15.7:0/1957985404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c104c30 msgr2=0x7fe74c1050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:59.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.589+0000 7fe752400640 1 --2- 172.21.15.7:0/1957985404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c104c30 0x7fe74c1050b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe738009900 tx=0x7fe73802bc30 comp rx=0 tx=0).stop 2024-03-18T01:37:59.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.590+0000 7fe752400640 1 -- 172.21.15.7:0/1957985404 shutdown_connections 2024-03-18T01:37:59.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.590+0000 7fe752400640 1 --2- 172.21.15.7:0/1957985404 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe74c1055f0 0x7fe74c10be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:59.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.590+0000 7fe752400640 1 --2- 172.21.15.7:0/1957985404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c104c30 0x7fe74c1050b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:59.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.590+0000 7fe752400640 1 --2- 172.21.15.7:0/1957985404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74c103a40 0x7fe74c103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:59.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.590+0000 7fe752400640 1 -- 172.21.15.7:0/1957985404 >> 172.21.15.7:0/1957985404 conn(0x7fe74c0fefd0 msgr2=0x7fe74c1013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:37:59.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.590+0000 7fe752400640 1 -- 172.21.15.7:0/1957985404 shutdown_connections 2024-03-18T01:37:59.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.590+0000 7fe752400640 1 -- 172.21.15.7:0/1957985404 wait complete. 2024-03-18T01:37:59.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.591+0000 7fe752400640 1 Processor -- start 2024-03-18T01:37:59.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.591+0000 7fe752400640 1 -- start start 2024-03-18T01:37:59.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.591+0000 7fe752400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c103a40 0x7fe74c199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:59.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.591+0000 7fe752400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe74c104c30 0x7fe74c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:59.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.591+0000 7fe752400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74c1055f0 0x7fe74c1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:59.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.591+0000 7fe752400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c10f6f0 con 0x7fe74c104c30 2024-03-18T01:37:59.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.591+0000 7fe752400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c10f570 con 0x7fe74c1055f0 2024-03-18T01:37:59.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.591+0000 7fe752400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c10f870 con 0x7fe74c103a40 2024-03-18T01:37:59.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c103a40 0x7fe74c199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:59.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c103a40 0x7fe74c199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52224/0 (socket says 172.21.15.7:52224) 2024-03-18T01:37:59.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 -- 172.21.15.7:0/1569668299 learned_addr learned my addr 172.21.15.7:0/1569668299 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:37:59.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74aa00640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe74c104c30 0x7fe74c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:59.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74be00640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74c1055f0 0x7fe74c1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:59.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 -- 172.21.15.7:0/1569668299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74c1055f0 msgr2=0x7fe74c1a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:59.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74c1055f0 0x7fe74c1a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:59.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 -- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe74c104c30 msgr2=0x7fe74c19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:37:59.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe74c104c30 0x7fe74c19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:37:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 -- 172.21.15.7:0/1569668299 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c1a1930 con 0x7fe74c103a40 2024-03-18T01:37:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.592+0000 7fe74b400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c103a40 0x7fe74c199d20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00e9c0 tx=0x7fe73c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.593+0000 7fe743e00640 1 -- 172.21.15.7:0/1569668299 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c00cd80 con 0x7fe74c103a40 2024-03-18T01:37:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.593+0000 7fe743e00640 1 -- 172.21.15.7:0/1569668299 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe73c0035f0 con 0x7fe74c103a40 2024-03-18T01:37:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.593+0000 7fe743e00640 1 -- 172.21.15.7:0/1569668299 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c010690 con 0x7fe74c103a40 2024-03-18T01:37:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.593+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74c1a1c20 con 0x7fe74c103a40 2024-03-18T01:37:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.593+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74c1a2120 con 0x7fe74c103a40 2024-03-18T01:37:59.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.594+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe74c10ad00 con 0x7fe74c103a40 2024-03-18T01:37:59.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.596+0000 7fe743e00640 1 -- 172.21.15.7:0/1569668299 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fe73c0037a0 con 0x7fe74c103a40 2024-03-18T01:37:59.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.597+0000 7fe743e00640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7300a2870 0x7fe7300a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:37:59.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.597+0000 7fe743e00640 1 -- 172.21.15.7:0/1569668299 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe73c0cf1c0 con 0x7fe74c103a40 2024-03-18T01:37:59.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.598+0000 7fe74aa00640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7300a2870 0x7fe7300a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:37:59.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.598+0000 7fe74aa00640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7300a2870 0x7fe7300a4d30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe73802c240 tx=0x7fe73802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:37:59.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.601+0000 7fe743e00640 1 -- 172.21.15.7:0/1569668299 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe73c09bea0 con 0x7fe74c103a40 2024-03-18T01:37:59.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:37:59.947+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fe74c062520 con 0x7fe74c103a40 2024-03-18T01:38:00.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.730+0000 7fe743e00640 1 -- 172.21.15.7:0/1569668299 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v28) ==== 117+0+0 (secure 0 0 0) 0x7fe73c09b7a0 con 0x7fe74c103a40 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.735+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7300a2870 msgr2=0x7fe7300a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.735+0000 7fe752400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7300a2870 0x7fe7300a4d30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe73802c240 tx=0x7fe73802df90 comp rx=0 tx=0).stop 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.735+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c103a40 msgr2=0x7fe74c199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.735+0000 7fe752400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c103a40 0x7fe74c199d20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe73c00e9c0 tx=0x7fe73c00ee80 comp rx=0 tx=0).stop 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.736+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 shutdown_connections 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.736+0000 7fe752400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe74c1055f0 0x7fe74c1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.736+0000 7fe752400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7300a2870 0x7fe7300a4d30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.736+0000 7fe752400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe74c104c30 0x7fe74c19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:00.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.736+0000 7fe752400640 1 --2- 172.21.15.7:0/1569668299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe74c103a40 0x7fe74c199d20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:00.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.736+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 >> 172.21.15.7:0/1569668299 conn(0x7fe74c0fefd0 msgr2=0x7fe74c107ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:00.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.736+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 shutdown_connections 2024-03-18T01:38:00.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:00.736+0000 7fe752400640 1 -- 172.21.15.7:0/1569668299 wait complete. 2024-03-18T01:38:00.765 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-18T01:38:00.776 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-18T01:38:00.787 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:38:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.012+0000 7f8d5f800640 1 -- 172.21.15.7:0/4270499544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58107e40 msgr2=0x7f8d58108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.012+0000 7f8d5f800640 1 --2- 172.21.15.7:0/4270499544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58107e40 0x7f8d58108260 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c009900 tx=0x7f8d4c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:38:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.013+0000 7f8d5f800640 1 -- 172.21.15.7:0/4270499544 shutdown_connections 2024-03-18T01:38:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.013+0000 7f8d5f800640 1 --2- 172.21.15.7:0/4270499544 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d58109950 0x7f8d58110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.013+0000 7f8d5f800640 1 --2- 172.21.15.7:0/4270499544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d58108f90 0x7f8d58109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.013+0000 7f8d5f800640 1 --2- 172.21.15.7:0/4270499544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58107e40 0x7f8d58108260 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.013+0000 7f8d5f800640 1 -- 172.21.15.7:0/4270499544 >> 172.21.15.7:0/4270499544 conn(0x7f8d58076dd0 msgr2=0x7f8d58079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.013+0000 7f8d5f800640 1 -- 172.21.15.7:0/4270499544 shutdown_connections 2024-03-18T01:38:01.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.013+0000 7f8d5f800640 1 -- 172.21.15.7:0/4270499544 wait complete. 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5f800640 1 Processor -- start 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5f800640 1 -- start start 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5f800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d58107e40 0x7f8d5819e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5f800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d58108f90 0x7f8d5819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5f800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58109950 0x7f8d581a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5f800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8d58113b30 con 0x7f8d58107e40 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5f800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8d581139b0 con 0x7f8d58109950 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5f800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8d58113cb0 con 0x7f8d58108f90 2024-03-18T01:38:01.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5ce00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d58107e40 0x7f8d5819e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5ce00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d58107e40 0x7f8d5819e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53814/0 (socket says 172.21.15.7:53814) 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5ce00640 1 -- 172.21.15.7:0/635095633 learned_addr learned my addr 172.21.15.7:0/635095633 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d57e00640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d58108f90 0x7f8d5819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.014+0000 7f8d5d800640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58109950 0x7f8d581a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d57e00640 1 -- 172.21.15.7:0/635095633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58109950 msgr2=0x7f8d581a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d57e00640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58109950 0x7f8d581a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d57e00640 1 -- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d58107e40 msgr2=0x7f8d5819e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d57e00640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d58107e40 0x7f8d5819e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d57e00640 1 -- 172.21.15.7:0/635095633 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d581a5d40 con 0x7f8d58108f90 2024-03-18T01:38:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d5ce00640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d58107e40 0x7f8d5819e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d5d800640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58109950 0x7f8d581a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:01.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d57e00640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d58108f90 0x7f8d5819e660 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400e9c0 tx=0x7f8d4400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.015+0000 7f8d55600640 1 -- 172.21.15.7:0/635095633 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4400ccc0 con 0x7f8d58108f90 2024-03-18T01:38:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.016+0000 7f8d55600640 1 -- 172.21.15.7:0/635095633 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d440035f0 con 0x7f8d58108f90 2024-03-18T01:38:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.016+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d581a5fd0 con 0x7f8d58108f90 2024-03-18T01:38:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.016+0000 7f8d55600640 1 -- 172.21.15.7:0/635095633 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4400ce60 con 0x7f8d58108f90 2024-03-18T01:38:01.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.016+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d581a64a0 con 0x7f8d58108f90 2024-03-18T01:38:01.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:00 smithi046 ceph-mon[36108]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T01:38:01.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:00 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1569668299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T01:38:01.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:00 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T01:38:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.017+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d5810f0f0 con 0x7f8d58108f90 2024-03-18T01:38:01.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.019+0000 7f8d55600640 1 -- 172.21.15.7:0/635095633 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f8d440037a0 con 0x7f8d58108f90 2024-03-18T01:38:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.022+0000 7f8d55600640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d400a2800 0x7f8d400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.022+0000 7f8d55600640 1 -- 172.21.15.7:0/635095633 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8d440cf360 con 0x7f8d58108f90 2024-03-18T01:38:01.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.022+0000 7f8d5ce00640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d400a2800 0x7f8d400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.023+0000 7f8d5ce00640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d400a2800 0x7f8d400a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c02c8e0 tx=0x7f8d4c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:01.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.025+0000 7f8d55600640 1 -- 172.21.15.7:0/635095633 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d4409c040 con 0x7f8d58108f90 2024-03-18T01:38:01.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:00 smithi007 ceph-mon[31996]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T01:38:01.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:00 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1569668299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T01:38:01.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:00 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T01:38:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:00 smithi203 ceph-mon[36345]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T01:38:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:00 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1569668299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T01:38:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:00 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T01:38:01.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.230+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d580623e0 con 0x7f8d58108f90 2024-03-18T01:38:01.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.232+0000 7f8d55600640 1 -- 172.21.15.7:0/635095633 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7f8d4409b940 con 0x7f8d58108f90 2024-03-18T01:38:01.234 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:01.234 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.234+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d400a2800 msgr2=0x7f8d400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.234+0000 7f8d5f800640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d400a2800 0x7f8d400a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c02c8e0 tx=0x7f8d4c02df90 comp rx=0 tx=0).stop 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.234+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d58108f90 msgr2=0x7f8d5819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.234+0000 7f8d5f800640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d58108f90 0x7f8d5819e660 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400e9c0 tx=0x7f8d4400ee80 comp rx=0 tx=0).stop 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.235+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 shutdown_connections 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.235+0000 7f8d5f800640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d58109950 0x7f8d581a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.235+0000 7f8d5f800640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d58108f90 0x7f8d5819e660 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.235+0000 7f8d5f800640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d400a2800 0x7f8d400a4cc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.235+0000 7f8d5f800640 1 --2- 172.21.15.7:0/635095633 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8d58107e40 0x7f8d5819e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.235+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 >> 172.21.15.7:0/635095633 conn(0x7f8d58076dd0 msgr2=0x7f8d5810c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:01.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.235+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 shutdown_connections 2024-03-18T01:38:01.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.235+0000 7f8d5f800640 1 -- 172.21.15.7:0/635095633 wait complete. 2024-03-18T01:38:01.255 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:38:01.255 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:38:01.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 -- 172.21.15.7:0/1633964503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 msgr2=0x7f5b8c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 --2- 172.21.15.7:0/1633964503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 0x7f5b8c108240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009900 tx=0x7f5b7c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:38:01.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 -- 172.21.15.7:0/1633964503 shutdown_connections 2024-03-18T01:38:01.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 --2- 172.21.15.7:0/1633964503 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b8c109930 0x7f5b8c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 --2- 172.21.15.7:0/1633964503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c108f70 0x7f5b8c1093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 --2- 172.21.15.7:0/1633964503 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 0x7f5b8c108240 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 -- 172.21.15.7:0/1633964503 >> 172.21.15.7:0/1633964503 conn(0x7f5b8c076dd0 msgr2=0x7f5b8c0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:01.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 -- 172.21.15.7:0/1633964503 shutdown_connections 2024-03-18T01:38:01.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.476+0000 7f5b91000640 1 -- 172.21.15.7:0/1633964503 wait complete. 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b91000640 1 Processor -- start 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b91000640 1 -- start start 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b91000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 0x7f5b8c19e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b91000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b8c108f70 0x7f5b8c19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b91000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c109930 0x7f5b8c1a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b91000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c113af0 con 0x7f5b8c108f70 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b91000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c113970 con 0x7f5b8c109930 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b91000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c113c70 con 0x7f5b8c107e20 2024-03-18T01:38:01.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b89e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 0x7f5b8c19e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b89e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 0x7f5b8c19e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52266/0 (socket says 172.21.15.7:52266) 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.477+0000 7f5b89e00640 1 -- 172.21.15.7:0/1491548466 learned_addr learned my addr 172.21.15.7:0/1491548466 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.478+0000 7f5b8a800640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c109930 0x7f5b8c1a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.478+0000 7f5b89400640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b8c108f70 0x7f5b8c19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.478+0000 7f5b89e00640 1 -- 172.21.15.7:0/1491548466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c109930 msgr2=0x7f5b8c1a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.478+0000 7f5b89e00640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c109930 0x7f5b8c1a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.478+0000 7f5b89e00640 1 -- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b8c108f70 msgr2=0x7f5b8c19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.478+0000 7f5b89e00640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b8c108f70 0x7f5b8c19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.478+0000 7f5b89e00640 1 -- 172.21.15.7:0/1491548466 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8c1a5d20 con 0x7f5b8c107e20 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.478+0000 7f5b89e00640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 0x7f5b8c19e100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000c00 tx=0x7f5b7c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.479+0000 7f5b82a00640 1 -- 172.21.15.7:0/1491548466 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c043020 con 0x7f5b8c107e20 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.479+0000 7f5b82a00640 1 -- 172.21.15.7:0/1491548466 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b7c0054e0 con 0x7f5b8c107e20 2024-03-18T01:38:01.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.479+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8c1a5fb0 con 0x7f5b8c107e20 2024-03-18T01:38:01.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.479+0000 7f5b82a00640 1 -- 172.21.15.7:0/1491548466 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c003c40 con 0x7f5b8c107e20 2024-03-18T01:38:01.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.479+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8c1a6470 con 0x7f5b8c107e20 2024-03-18T01:38:01.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.480+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b8c10f130 con 0x7f5b8c107e20 2024-03-18T01:38:01.485 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.484+0000 7f5b82a00640 1 -- 172.21.15.7:0/1491548466 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f5b7c046020 con 0x7f5b8c107e20 2024-03-18T01:38:01.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.485+0000 7f5b82a00640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5b700a2880 0x7f5b700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.486+0000 7f5b82a00640 1 -- 172.21.15.7:0/1491548466 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5b7c0f1340 con 0x7f5b8c107e20 2024-03-18T01:38:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.486+0000 7f5b89400640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5b700a2880 0x7f5b700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.486+0000 7f5b82a00640 1 -- 172.21.15.7:0/1491548466 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5b7c04d020 con 0x7f5b8c107e20 2024-03-18T01:38:01.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.487+0000 7f5b89400640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5b700a2880 0x7f5b700a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5b780095f0 tx=0x7f5b78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:01.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.692+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b8c02d400 con 0x7f5b8c107e20 2024-03-18T01:38:01.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.692+0000 7f5b82a00640 1 -- 172.21.15.7:0/1491548466 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f5b7c0be020 con 0x7f5b8c107e20 2024-03-18T01:38:01.693 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.696+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5b700a2880 msgr2=0x7f5b700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5b700a2880 0x7f5b700a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5b780095f0 tx=0x7f5b78005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 msgr2=0x7f5b8c19e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 0x7f5b8c19e100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000c00 tx=0x7f5b7c02c650 comp rx=0 tx=0).stop 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 shutdown_connections 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b8c109930 0x7f5b8c1a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5b700a2880 0x7f5b700a4d40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5b8c108f70 0x7f5b8c19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 --2- 172.21.15.7:0/1491548466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b8c107e20 0x7f5b8c19e100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 >> 172.21.15.7:0/1491548466 conn(0x7f5b8c076dd0 msgr2=0x7f5b8c10c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:01.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 shutdown_connections 2024-03-18T01:38:01.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.697+0000 7f5b91000640 1 -- 172.21.15.7:0/1491548466 wait complete. 2024-03-18T01:38:01.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 -- 172.21.15.7:0/2649073649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec075820 msgr2=0x7f59ec073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 --2- 172.21.15.7:0/2649073649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec075820 0x7f59ec073c70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f59dc009900 tx=0x7f59dc02bc30 comp rx=0 tx=0).stop 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 -- 172.21.15.7:0/2649073649 shutdown_connections 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 --2- 172.21.15.7:0/2649073649 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59ec1094f0 0x7f59ec10b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 --2- 172.21.15.7:0/2649073649 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59ec074240 0x7f59ec0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 --2- 172.21.15.7:0/2649073649 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec075820 0x7f59ec073c70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 -- 172.21.15.7:0/2649073649 >> 172.21.15.7:0/2649073649 conn(0x7f59ec0fcba0 msgr2=0x7f59ec0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 -- 172.21.15.7:0/2649073649 shutdown_connections 2024-03-18T01:38:01.969 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.967+0000 7f59f0c00640 1 -- 172.21.15.7:0/2649073649 wait complete. 2024-03-18T01:38:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.969+0000 7f59f0c00640 1 Processor -- start 2024-03-18T01:38:01.970 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59f0c00640 1 -- start start 2024-03-18T01:38:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59f0c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec074240 0x7f59ec199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59f0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59ec075820 0x7f59ec19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59f0c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59ec1094f0 0x7f59ec1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59f0c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec10f4e0 con 0x7f59ec1094f0 2024-03-18T01:38:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59f0c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec10f360 con 0x7f59ec075820 2024-03-18T01:38:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59f0c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f59ec10f660 con 0x7f59ec074240 2024-03-18T01:38:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59eb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec074240 0x7f59ec199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.972 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59eb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec074240 0x7f59ec199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52286/0 (socket says 172.21.15.7:52286) 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59eb400640 1 -- 172.21.15.7:0/1727009352 learned_addr learned my addr 172.21.15.7:0/1727009352 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.970+0000 7f59eaa00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59ec075820 0x7f59ec19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59ebe00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59ec1094f0 0x7f59ec1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59eb400640 1 -- 172.21.15.7:0/1727009352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59ec075820 msgr2=0x7f59ec19a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59eb400640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59ec075820 0x7f59ec19a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59eb400640 1 -- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59ec1094f0 msgr2=0x7f59ec1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59eb400640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59ec1094f0 0x7f59ec1a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59eb400640 1 -- 172.21.15.7:0/1727009352 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ec1a1770 con 0x7f59ec074240 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59ebe00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59ec1094f0 0x7f59ec1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:01.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59eb400640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec074240 0x7f59ec199b50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f59dc005ab0 tx=0x7f59dc02c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:01.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.971+0000 7f59e3e00640 1 -- 172.21.15.7:0/1727009352 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc043020 con 0x7f59ec074240 2024-03-18T01:38:01.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.972+0000 7f59e3e00640 1 -- 172.21.15.7:0/1727009352 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59dc0054e0 con 0x7f59ec074240 2024-03-18T01:38:01.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.972+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ec1a1a00 con 0x7f59ec074240 2024-03-18T01:38:01.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.972+0000 7f59e3e00640 1 -- 172.21.15.7:0/1727009352 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc003ce0 con 0x7f59ec074240 2024-03-18T01:38:01.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.972+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ec1a1fa0 con 0x7f59ec074240 2024-03-18T01:38:01.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.973+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59ec04f540 con 0x7f59ec074240 2024-03-18T01:38:01.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.975+0000 7f59e3e00640 1 -- 172.21.15.7:0/1727009352 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f59dc046020 con 0x7f59ec074240 2024-03-18T01:38:01.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.976+0000 7f59e3e00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f59d00a2880 0x7f59d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:01.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.976+0000 7f59e3e00640 1 -- 172.21.15.7:0/1727009352 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f59dc0f0f50 con 0x7f59ec074240 2024-03-18T01:38:01.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.976+0000 7f59eaa00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f59d00a2880 0x7f59d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:01.977 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.977+0000 7f59eaa00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f59d00a2880 0x7f59d00a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f59ec19b060 tx=0x7f59d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:01.981 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:01.980+0000 7f59e3e00640 1 -- 172.21.15.7:0/1727009352 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59dc04d020 con 0x7f59ec074240 2024-03-18T01:38:02.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:01 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T01:38:02.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:01 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:38:02.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:01 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/635095633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:02.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:01 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1491548466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:02.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:01 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T01:38:02.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:01 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:38:02.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:01 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/635095633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:02.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:01 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1491548466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:02.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:01 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T01:38:02.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:01 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:38:02.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:01 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/635095633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:02.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:01 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1491548466' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:02.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.194+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f59ec062430 con 0x7f59ec074240 2024-03-18T01:38:02.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.194+0000 7f59e3e00640 1 -- 172.21.15.7:0/1727009352 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f59dc0bdc30 con 0x7f59ec074240 2024-03-18T01:38:02.196 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:38:02.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f59d00a2880 msgr2=0x7f59d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f59d00a2880 0x7f59d00a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f59ec19b060 tx=0x7f59d8005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:02.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec074240 msgr2=0x7f59ec199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec074240 0x7f59ec199b50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f59dc005ab0 tx=0x7f59dc02c6c0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 shutdown_connections 2024-03-18T01:38:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f59d00a2880 0x7f59d00a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f59ec1094f0 0x7f59ec1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59ec075820 0x7f59ec19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 --2- 172.21.15.7:0/1727009352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec074240 0x7f59ec199b50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 >> 172.21.15.7:0/1727009352 conn(0x7f59ec0fcba0 msgr2=0x7f59ec1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 shutdown_connections 2024-03-18T01:38:02.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.197+0000 7f59f0c00640 1 -- 172.21.15.7:0/1727009352 wait complete. 2024-03-18T01:38:02.221 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:38:02.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 -- 172.21.15.7:0/3358868664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c103820 msgr2=0x7fec4c103c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 --2- 172.21.15.7:0/3358868664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c103820 0x7fec4c103c40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009900 tx=0x7fec3c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:38:02.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 -- 172.21.15.7:0/3358868664 shutdown_connections 2024-03-18T01:38:02.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 --2- 172.21.15.7:0/3358868664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec4c1053d0 0x7fec4c10bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 --2- 172.21.15.7:0/3358868664 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec4c104a10 0x7fec4c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 --2- 172.21.15.7:0/3358868664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c103820 0x7fec4c103c40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 -- 172.21.15.7:0/3358868664 >> 172.21.15.7:0/3358868664 conn(0x7fec4c0fedb0 msgr2=0x7fec4c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:02.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 -- 172.21.15.7:0/3358868664 shutdown_connections 2024-03-18T01:38:02.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.436+0000 7fec4b200640 1 -- 172.21.15.7:0/3358868664 wait complete. 2024-03-18T01:38:02.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.437+0000 7fec4b200640 1 Processor -- start 2024-03-18T01:38:02.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.437+0000 7fec4b200640 1 -- start start 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec4b200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec4c103820 0x7fec4c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec4b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c104a10 0x7fec4c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec4b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec4c1053d0 0x7fec4c1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec4b200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10f4f0 con 0x7fec4c103820 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec4b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10f370 con 0x7fec4c104a10 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec4b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c10f670 con 0x7fec4c1053d0 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c104a10 0x7fec4c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c104a10 0x7fec4c19a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:43874/0 (socket says 172.21.15.7:43874) 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49400640 1 -- 172.21.15.7:0/3943610677 learned_addr learned my addr 172.21.15.7:0/3943610677 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec4a800640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec4c1053d0 0x7fec4c1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49e00640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec4c103820 0x7fec4c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:02.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49400640 1 -- 172.21.15.7:0/3943610677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec4c1053d0 msgr2=0x7fec4c1a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49400640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec4c1053d0 0x7fec4c1a1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49400640 1 -- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec4c103820 msgr2=0x7fec4c199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49400640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec4c103820 0x7fec4c199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.438+0000 7fec49400640 1 -- 172.21.15.7:0/3943610677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4c1a1780 con 0x7fec4c104a10 2024-03-18T01:38:02.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.439+0000 7fec49400640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c104a10 0x7fec4c19a0a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fec3800e9f0 tx=0x7fec3800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:02.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.439+0000 7fec42a00640 1 -- 172.21.15.7:0/3943610677 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3800cd80 con 0x7fec4c104a10 2024-03-18T01:38:02.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.439+0000 7fec42a00640 1 -- 172.21.15.7:0/3943610677 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec38010910 con 0x7fec4c104a10 2024-03-18T01:38:02.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.439+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4c1a1a70 con 0x7fec4c104a10 2024-03-18T01:38:02.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.439+0000 7fec42a00640 1 -- 172.21.15.7:0/3943610677 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec38018990 con 0x7fec4c104a10 2024-03-18T01:38:02.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.439+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4c1a2010 con 0x7fec4c104a10 2024-03-18T01:38:02.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.440+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec4c10abc0 con 0x7fec4c104a10 2024-03-18T01:38:02.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.443+0000 7fec42a00640 1 -- 172.21.15.7:0/3943610677 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fec38010ac0 con 0x7fec4c104a10 2024-03-18T01:38:02.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.444+0000 7fec42a00640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec300a2880 0x7fec300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:02.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.444+0000 7fec42a00640 1 -- 172.21.15.7:0/3943610677 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fec380cf0b0 con 0x7fec4c104a10 2024-03-18T01:38:02.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.445+0000 7fec49e00640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec300a2880 0x7fec300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:02.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.445+0000 7fec49e00640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec300a2880 0x7fec300a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fec3c02c170 tx=0x7fec3c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:02.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.448+0000 7fec42a00640 1 -- 172.21.15.7:0/3943610677 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fec3809b740 con 0x7fec4c104a10 2024-03-18T01:38:02.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.655+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fec4c062410 con 0x7fec4c104a10 2024-03-18T01:38:02.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.656+0000 7fec42a00640 1 -- 172.21.15.7:0/3943610677 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fec3809f7e0 con 0x7fec4c104a10 2024-03-18T01:38:02.657 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:38:02.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec300a2880 msgr2=0x7fec300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec300a2880 0x7fec300a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fec3c02c170 tx=0x7fec3c02df90 comp rx=0 tx=0).stop 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c104a10 msgr2=0x7fec4c19a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c104a10 0x7fec4c19a0a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fec3800e9f0 tx=0x7fec3800eeb0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 shutdown_connections 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec4c1053d0 0x7fec4c1a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec4c104a10 0x7fec4c19a0a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec300a2880 0x7fec300a4d40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 --2- 172.21.15.7:0/3943610677 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fec4c103820 0x7fec4c199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.658+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 >> 172.21.15.7:0/3943610677 conn(0x7fec4c0fedb0 msgr2=0x7fec4c107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.659+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 shutdown_connections 2024-03-18T01:38:02.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.659+0000 7fec4b200640 1 -- 172.21.15.7:0/3943610677 wait complete. 2024-03-18T01:38:02.678 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:38:02.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 -- 172.21.15.7:0/2020950781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01017f0 msgr2=0x7f69f0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 --2- 172.21.15.7:0/2020950781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01017f0 0x7f69f0101c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009900 tx=0x7f69e402bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:02.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 -- 172.21.15.7:0/2020950781 shutdown_connections 2024-03-18T01:38:02.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 --2- 172.21.15.7:0/2020950781 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69f01033a0 0x7f69f0109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 --2- 172.21.15.7:0/2020950781 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f01029e0 0x7f69f0102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 --2- 172.21.15.7:0/2020950781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01017f0 0x7f69f0101c10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 -- 172.21.15.7:0/2020950781 >> 172.21.15.7:0/2020950781 conn(0x7f69f00fcd60 msgr2=0x7f69f00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:02.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 -- 172.21.15.7:0/2020950781 shutdown_connections 2024-03-18T01:38:02.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.893+0000 7f69f7c00640 1 -- 172.21.15.7:0/2020950781 wait complete. 2024-03-18T01:38:02.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.894+0000 7f69f7c00640 1 Processor -- start 2024-03-18T01:38:02.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f7c00640 1 -- start start 2024-03-18T01:38:02.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f7c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f01017f0 0x7f69f0199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:02.897 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f7c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69f01029e0 0x7f69f019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f7c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01033a0 0x7f69f01a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f7c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f69f010d4b0 con 0x7f69f01029e0 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f7c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f69f010d330 con 0x7f69f01033a0 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f7c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69f010d630 con 0x7f69f01017f0 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f01017f0 0x7f69f0199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f01017f0 0x7f69f0199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52310/0 (socket says 172.21.15.7:52310) 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f5200640 1 -- 172.21.15.7:0/1587800347 learned_addr learned my addr 172.21.15.7:0/1587800347 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69f5c00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01033a0 0x7f69f01a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.895+0000 7f69efe00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69f01029e0 0x7f69f019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69f5200640 1 -- 172.21.15.7:0/1587800347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01033a0 msgr2=0x7f69f01a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69f5200640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01033a0 0x7f69f01a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.898 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69f5200640 1 -- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69f01029e0 msgr2=0x7f69f019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69f5200640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69f01029e0 0x7f69f019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69f5200640 1 -- 172.21.15.7:0/1587800347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f01a1990 con 0x7f69f01017f0 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69efe00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69f01029e0 0x7f69f019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69f5200640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f01017f0 0x7f69f0199d70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f69e402efd0 tx=0x7f69e402c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69ed600640 1 -- 172.21.15.7:0/1587800347 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4041070 con 0x7f69f01017f0 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69ed600640 1 -- 172.21.15.7:0/1587800347 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69e4004450 con 0x7f69f01017f0 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f01a1c20 con 0x7f69f01017f0 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.896+0000 7f69ed600640 1 -- 172.21.15.7:0/1587800347 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4003e50 con 0x7f69f01017f0 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.897+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f01a2130 con 0x7f69f01017f0 2024-03-18T01:38:02.899 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.898+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69f01a2300 con 0x7f69f01017f0 2024-03-18T01:38:02.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.899+0000 7f69ed600640 1 -- 172.21.15.7:0/1587800347 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f69e4044070 con 0x7f69f01017f0 2024-03-18T01:38:02.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.901+0000 7f69ed600640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69d80a2a20 0x7f69d80a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:02.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.901+0000 7f69ed600640 1 -- 172.21.15.7:0/1587800347 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f69e40f0990 con 0x7f69f01017f0 2024-03-18T01:38:02.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.901+0000 7f69efe00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69d80a2a20 0x7f69d80a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:02.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.902+0000 7f69efe00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69d80a2a20 0x7f69d80a4ee0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f69f019b280 tx=0x7f69dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:02.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:02.909+0000 7f69ed600640 1 -- 172.21.15.7:0/1587800347 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f69e40bd670 con 0x7f69f01017f0 2024-03-18T01:38:03.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:02 smithi046 ceph-mon[36108]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T01:38:03.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:02 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1727009352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:03.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:02 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3943610677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:03.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:02 smithi007 ceph-mon[31996]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T01:38:03.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:02 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1727009352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:03.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:02 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3943610677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:03.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.113+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f69f01a24d0 con 0x7f69f01017f0 2024-03-18T01:38:03.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.114+0000 7f69ed600640 1 -- 172.21.15.7:0/1587800347 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f69e40bcf70 con 0x7f69f01017f0 2024-03-18T01:38:03.115 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:38:03.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69d80a2a20 msgr2=0x7f69d80a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69d80a2a20 0x7f69d80a4ee0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f69f019b280 tx=0x7f69dc005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:03.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f01017f0 msgr2=0x7f69f0199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f01017f0 0x7f69f0199d70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f69e402efd0 tx=0x7f69e402c660 comp rx=0 tx=0).stop 2024-03-18T01:38:03.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 shutdown_connections 2024-03-18T01:38:03.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69f01033a0 0x7f69f01a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f69d80a2a20 0x7f69d80a4ee0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f69f01029e0 0x7f69f019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 --2- 172.21.15.7:0/1587800347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f01017f0 0x7f69f0199d70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 >> 172.21.15.7:0/1587800347 conn(0x7f69f00fcd60 msgr2=0x7f69f0073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:03.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 shutdown_connections 2024-03-18T01:38:03.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.117+0000 7f69f7c00640 1 -- 172.21.15.7:0/1587800347 wait complete. 2024-03-18T01:38:03.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:38:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:02 smithi203 ceph-mon[36345]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T01:38:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:02 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1727009352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:02 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3943610677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.360+0000 7f907e800640 1 -- 172.21.15.7:0/2226905810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781053d0 msgr2=0x7f907810bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.360+0000 7f907e800640 1 --2- 172.21.15.7:0/2226905810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781053d0 0x7f907810bc60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9070009ca0 tx=0x7f907002c0f0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.360+0000 7f907e800640 1 -- 172.21.15.7:0/2226905810 shutdown_connections 2024-03-18T01:38:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.360+0000 7f907e800640 1 --2- 172.21.15.7:0/2226905810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781053d0 0x7f907810bc60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.360+0000 7f907e800640 1 --2- 172.21.15.7:0/2226905810 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9078104a10 0x7f9078104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.360+0000 7f907e800640 1 --2- 172.21.15.7:0/2226905810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9078103820 0x7f9078103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.360+0000 7f907e800640 1 -- 172.21.15.7:0/2226905810 >> 172.21.15.7:0/2226905810 conn(0x7f90780fedb0 msgr2=0x7f90781011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:03.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.361+0000 7f907e800640 1 -- 172.21.15.7:0/2226905810 shutdown_connections 2024-03-18T01:38:03.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.361+0000 7f907e800640 1 -- 172.21.15.7:0/2226905810 wait complete. 2024-03-18T01:38:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907e800640 1 Processor -- start 2024-03-18T01:38:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907e800640 1 -- start start 2024-03-18T01:38:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907e800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9078103820 0x7f9078199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9078104a10 0x7f9078199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781053d0 0x7f90781a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907e800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f907810f480 con 0x7f9078103820 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907e800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f907810f300 con 0x7f9078104a10 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907e800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f907810f600 con 0x7f90781053d0 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9078104a10 0x7f9078199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9078104a10 0x7f9078199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:43914/0 (socket says 172.21.15.7:43914) 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907d400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9078103820 0x7f9078199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.362+0000 7f907ca00640 1 -- 172.21.15.7:0/1958650434 learned_addr learned my addr 172.21.15.7:0/1958650434 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.363+0000 7f907de00640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781053d0 0x7f90781a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.363+0000 7f907ca00640 1 -- 172.21.15.7:0/1958650434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781053d0 msgr2=0x7f90781a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.363+0000 7f907ca00640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781053d0 0x7f90781a0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.363+0000 7f907ca00640 1 -- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9078103820 msgr2=0x7f9078199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.363+0000 7f907ca00640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9078103820 0x7f9078199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.363+0000 7f907ca00640 1 -- 172.21.15.7:0/1958650434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90781a1590 con 0x7f9078104a10 2024-03-18T01:38:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.363+0000 7f907ca00640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9078104a10 0x7f9078199eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f906400e9f0 tx=0x7f906400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.363+0000 7f9076000640 1 -- 172.21.15.7:0/1958650434 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906400cd80 con 0x7f9078104a10 2024-03-18T01:38:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.364+0000 7f9076000640 1 -- 172.21.15.7:0/1958650434 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9064010910 con 0x7f9078104a10 2024-03-18T01:38:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.364+0000 7f9076000640 1 -- 172.21.15.7:0/1958650434 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9064018990 con 0x7f9078104a10 2024-03-18T01:38:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.364+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90781a1880 con 0x7f9078104a10 2024-03-18T01:38:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.364+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90781a1d80 con 0x7f9078104a10 2024-03-18T01:38:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.365+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f907810aa70 con 0x7f9078104a10 2024-03-18T01:38:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.368+0000 7f9076000640 1 -- 172.21.15.7:0/1958650434 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9064010ac0 con 0x7f9078104a10 2024-03-18T01:38:03.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.369+0000 7f9076000640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f90600a2920 0x7f90600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:03.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.369+0000 7f9076000640 1 -- 172.21.15.7:0/1958650434 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f90640cf230 con 0x7f9078104a10 2024-03-18T01:38:03.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.369+0000 7f907d400640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f90600a2920 0x7f90600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:03.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.370+0000 7f907d400640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f90600a2920 0x7f90600a4de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f906c0095f0 tx=0x7f906c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:03.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.370+0000 7f9076000640 1 -- 172.21.15.7:0/1958650434 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f906409b8c0 con 0x7f9078104a10 2024-03-18T01:38:03.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.604+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f907802d380 con 0x7f9078104a10 2024-03-18T01:38:03.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.606+0000 7f9076000640 1 -- 172.21.15.7:0/1958650434 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7f906409f960 con 0x7f9078104a10 2024-03-18T01:38:03.607 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:03.608 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:38:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f90600a2920 msgr2=0x7f90600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f90600a2920 0x7f90600a4de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f906c0095f0 tx=0x7f906c005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9078104a10 msgr2=0x7f9078199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9078104a10 0x7f9078199eb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f906400e9f0 tx=0x7f906400eeb0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 shutdown_connections 2024-03-18T01:38:03.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90781053d0 0x7f90781a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9078104a10 0x7f9078199eb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f90600a2920 0x7f90600a4de0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 --2- 172.21.15.7:0/1958650434 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9078103820 0x7f9078199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 >> 172.21.15.7:0/1958650434 conn(0x7f90780fedb0 msgr2=0x7f9078107c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 shutdown_connections 2024-03-18T01:38:03.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.609+0000 7f907e800640 1 -- 172.21.15.7:0/1958650434 wait complete. 2024-03-18T01:38:03.630 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:38:03.631 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 -- 172.21.15.7:0/3926841740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a181033c0 msgr2=0x7f8a18109c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 --2- 172.21.15.7:0/3926841740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a181033c0 0x7f8a18109c50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009ca0 tx=0x7f8a1002c100 comp rx=0 tx=0).stop 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 -- 172.21.15.7:0/3926841740 shutdown_connections 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 --2- 172.21.15.7:0/3926841740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a181033c0 0x7f8a18109c50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 --2- 172.21.15.7:0/3926841740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a18102a00 0x7f8a18102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 --2- 172.21.15.7:0/3926841740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a18101810 0x7f8a18101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 -- 172.21.15.7:0/3926841740 >> 172.21.15.7:0/3926841740 conn(0x7f8a180fcda0 msgr2=0x7f8a180ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 -- 172.21.15.7:0/3926841740 shutdown_connections 2024-03-18T01:38:03.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.854+0000 7f8a1e800640 1 -- 172.21.15.7:0/3926841740 wait complete. 2024-03-18T01:38:03.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.856+0000 7f8a1e800640 1 Processor -- start 2024-03-18T01:38:03.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.856+0000 7f8a1e800640 1 -- start start 2024-03-18T01:38:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.856+0000 7f8a1e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a18101810 0x7f8a18199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.856+0000 7f8a1e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a18102a00 0x7f8a1819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.856+0000 7f8a1e800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a181033c0 0x7f8a181a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.856+0000 7f8a1e800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1810d490 con 0x7f8a181033c0 2024-03-18T01:38:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a1e800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1810d310 con 0x7f8a18102a00 2024-03-18T01:38:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a1e800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1810d610 con 0x7f8a18101810 2024-03-18T01:38:03.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a18101810 0x7f8a18199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a18101810 0x7f8a18199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52352/0 (socket says 172.21.15.7:52352) 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17400640 1 -- 172.21.15.7:0/298894529 learned_addr learned my addr 172.21.15.7:0/298894529 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17e00640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a181033c0 0x7f8a181a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a16a00640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a18102a00 0x7f8a1819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17400640 1 -- 172.21.15.7:0/298894529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a18102a00 msgr2=0x7f8a1819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17400640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a18102a00 0x7f8a1819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17400640 1 -- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a181033c0 msgr2=0x7f8a181a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17400640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a181033c0 0x7f8a181a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a17400640 1 -- 172.21.15.7:0/298894529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a181a1920 con 0x7f8a18101810 2024-03-18T01:38:03.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.857+0000 7f8a16a00640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a18102a00 0x7f8a1819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:03.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.858+0000 7f8a17400640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a18101810 0x7f8a18199d10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800e9c0 tx=0x7f8a0800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:03.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.858+0000 7f8a0fe00640 1 -- 172.21.15.7:0/298894529 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0800cd80 con 0x7f8a18101810 2024-03-18T01:38:03.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.858+0000 7f8a0fe00640 1 -- 172.21.15.7:0/298894529 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a080035f0 con 0x7f8a18101810 2024-03-18T01:38:03.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.858+0000 7f8a0fe00640 1 -- 172.21.15.7:0/298894529 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08010690 con 0x7f8a18101810 2024-03-18T01:38:03.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.858+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a181a1c10 con 0x7f8a18101810 2024-03-18T01:38:03.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.858+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a181a2110 con 0x7f8a18101810 2024-03-18T01:38:03.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.859+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a181a22e0 con 0x7f8a18101810 2024-03-18T01:38:03.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.863+0000 7f8a0fe00640 1 -- 172.21.15.7:0/298894529 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f8a080037a0 con 0x7f8a18101810 2024-03-18T01:38:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.865+0000 7f8a0fe00640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f89fc0a2880 0x7f89fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.865+0000 7f8a0fe00640 1 -- 172.21.15.7:0/298894529 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8a080cf1c0 con 0x7f8a18101810 2024-03-18T01:38:03.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.865+0000 7f8a16a00640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f89fc0a2880 0x7f89fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:03.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.865+0000 7f8a0fe00640 1 -- 172.21.15.7:0/298894529 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a0809bea0 con 0x7f8a18101810 2024-03-18T01:38:03.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:03.866+0000 7f8a16a00640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f89fc0a2880 0x7f89fc0a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009fc0 tx=0x7f8a04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:04.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:03 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1587800347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:04.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:03 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1958650434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:04.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:03 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1587800347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:04.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:03 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1958650434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:04.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.072+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8a181a24b0 con 0x7f8a18101810 2024-03-18T01:38:04.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.073+0000 7f8a0fe00640 1 -- 172.21.15.7:0/298894529 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f8a0809b7a0 con 0x7f8a18101810 2024-03-18T01:38:04.074 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:38:04.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f89fc0a2880 msgr2=0x7f89fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f89fc0a2880 0x7f89fc0a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009fc0 tx=0x7f8a04005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a18101810 msgr2=0x7f8a18199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a18101810 0x7f8a18199d10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800e9c0 tx=0x7f8a0800ee80 comp rx=0 tx=0).stop 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 shutdown_connections 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f89fc0a2880 0x7f89fc0a4d40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8a181033c0 0x7f8a181a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a18102a00 0x7f8a1819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 --2- 172.21.15.7:0/298894529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a18101810 0x7f8a18199d10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 >> 172.21.15.7:0/298894529 conn(0x7f8a180fcda0 msgr2=0x7f8a18073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.075+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 shutdown_connections 2024-03-18T01:38:04.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.076+0000 7f8a1e800640 1 -- 172.21.15.7:0/298894529 wait complete. 2024-03-18T01:38:04.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:38:04.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:03 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1587800347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:04.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:03 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1958650434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:04.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 -- 172.21.15.7:0/2799269435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc1018f0 msgr2=0x7f09dc101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 --2- 172.21.15.7:0/2799269435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc1018f0 0x7f09dc101d70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f09c4009860 tx=0x7f09c402bb70 comp rx=0 tx=0).stop 2024-03-18T01:38:04.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 -- 172.21.15.7:0/2799269435 shutdown_connections 2024-03-18T01:38:04.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 --2- 172.21.15.7:0/2799269435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09dc1022b0 0x7f09dc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 --2- 172.21.15.7:0/2799269435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc1018f0 0x7f09dc101d70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 --2- 172.21.15.7:0/2799269435 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc100700 0x7f09dc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 -- 172.21.15.7:0/2799269435 >> 172.21.15.7:0/2799269435 conn(0x7f09dc0fbc70 msgr2=0x7f09dc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:04.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 -- 172.21.15.7:0/2799269435 shutdown_connections 2024-03-18T01:38:04.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.309+0000 7f09e2000640 1 -- 172.21.15.7:0/2799269435 wait complete. 2024-03-18T01:38:04.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.310+0000 7f09e2000640 1 Processor -- start 2024-03-18T01:38:04.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.310+0000 7f09e2000640 1 -- start start 2024-03-18T01:38:04.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.310+0000 7f09e2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc100700 0x7f09dc199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.310+0000 7f09e2000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09dc1018f0 0x7f09dc19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.310+0000 7f09e2000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc1022b0 0x7f09dc1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.310+0000 7f09e2000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc069500 con 0x7f09dc1022b0 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.310+0000 7f09e2000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc069380 con 0x7f09dc100700 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09e2000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc069680 con 0x7f09dc1018f0 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09db400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc100700 0x7f09dc199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09d2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09dc1018f0 0x7f09dc19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09dbe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc1022b0 0x7f09dc1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09db400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc100700 0x7f09dc199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:43946/0 (socket says 172.21.15.7:43946) 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09db400640 1 -- 172.21.15.7:0/1643703556 learned_addr learned my addr 172.21.15.7:0/1643703556 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09d2a00640 1 -- 172.21.15.7:0/1643703556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc100700 msgr2=0x7f09dc199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09d2a00640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc100700 0x7f09dc199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09d2a00640 1 -- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc1022b0 msgr2=0x7f09dc1a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09d2a00640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc1022b0 0x7f09dc1a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09d2a00640 1 -- 172.21.15.7:0/1643703556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc1a1940 con 0x7f09dc1018f0 2024-03-18T01:38:04.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.311+0000 7f09db400640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc100700 0x7f09dc199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:04.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.312+0000 7f09d2a00640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09dc1018f0 0x7f09dc19a270 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c402c0b0 tx=0x7f09c40042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:04.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.312+0000 7f09d8c00640 1 -- 172.21.15.7:0/1643703556 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4041020 con 0x7f09dc1018f0 2024-03-18T01:38:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.312+0000 7f09d8c00640 1 -- 172.21.15.7:0/1643703556 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09c40323b0 con 0x7f09dc1018f0 2024-03-18T01:38:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.312+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09dc1a1bd0 con 0x7f09dc1018f0 2024-03-18T01:38:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.312+0000 7f09d8c00640 1 -- 172.21.15.7:0/1643703556 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4039040 con 0x7f09dc1018f0 2024-03-18T01:38:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.312+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09dc1a20e0 con 0x7f09dc1018f0 2024-03-18T01:38:04.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.313+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09dc1a22b0 con 0x7f09dc1018f0 2024-03-18T01:38:04.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.315+0000 7f09d8c00640 1 -- 172.21.15.7:0/1643703556 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f09c4044020 con 0x7f09dc1018f0 2024-03-18T01:38:04.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.316+0000 7f09d8c00640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f09c00a2880 0x7f09c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:04.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.317+0000 7f09d8c00640 1 -- 172.21.15.7:0/1643703556 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f09c40f06a0 con 0x7f09dc1018f0 2024-03-18T01:38:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.317+0000 7f09db400640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f09c00a2880 0x7f09c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:04.318 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.318+0000 7f09db400640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f09c00a2880 0x7f09c00a4d40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f09dc19b200 tx=0x7f09cc0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:04.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.321+0000 7f09d8c00640 1 -- 172.21.15.7:0/1643703556 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f09c404b020 con 0x7f09dc1018f0 2024-03-18T01:38:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.529+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f09dc1a2640 con 0x7f09dc1018f0 2024-03-18T01:38:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.529+0000 7f09d8c00640 1 -- 172.21.15.7:0/1643703556 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f09c40bd380 con 0x7f09dc1018f0 2024-03-18T01:38:04.531 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f09c00a2880 msgr2=0x7f09c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f09c00a2880 0x7f09c00a4d40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f09dc19b200 tx=0x7f09cc0095c0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09dc1018f0 msgr2=0x7f09dc19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09dc1018f0 0x7f09dc19a270 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c402c0b0 tx=0x7f09c40042f0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 shutdown_connections 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f09c00a2880 0x7f09c00a4d40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f09dc1022b0 0x7f09dc1a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09dc1018f0 0x7f09dc19a270 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 --2- 172.21.15.7:0/1643703556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09dc100700 0x7f09dc199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 >> 172.21.15.7:0/1643703556 conn(0x7f09dc0fbc70 msgr2=0x7f09dc10b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 shutdown_connections 2024-03-18T01:38:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.533+0000 7f09e2000640 1 -- 172.21.15.7:0/1643703556 wait complete. 2024-03-18T01:38:04.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.764+0000 7fcc13600640 1 -- 172.21.15.7:0/3200058609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc0c104c10 msgr2=0x7fcc0c105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.764+0000 7fcc13600640 1 --2- 172.21.15.7:0/3200058609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc0c104c10 0x7fcc0c105090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009860 tx=0x7fcbf802bb70 comp rx=0 tx=0).stop 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.765+0000 7fcc13600640 1 -- 172.21.15.7:0/3200058609 shutdown_connections 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.765+0000 7fcc13600640 1 --2- 172.21.15.7:0/3200058609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc0c1055d0 0x7fcc0c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.765+0000 7fcc13600640 1 --2- 172.21.15.7:0/3200058609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc0c104c10 0x7fcc0c105090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.765+0000 7fcc13600640 1 --2- 172.21.15.7:0/3200058609 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc0c103a20 0x7fcc0c103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.765+0000 7fcc13600640 1 -- 172.21.15.7:0/3200058609 >> 172.21.15.7:0/3200058609 conn(0x7fcc0c0fef90 msgr2=0x7fcc0c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.765+0000 7fcc13600640 1 -- 172.21.15.7:0/3200058609 shutdown_connections 2024-03-18T01:38:04.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.765+0000 7fcc13600640 1 -- 172.21.15.7:0/3200058609 wait complete. 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc13600640 1 Processor -- start 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc13600640 1 -- start start 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc13600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc0c103a20 0x7fcc0c079590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc13600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc0c104c10 0x7fcc0c079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc13600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc0c1055d0 0x7fcc0c076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc13600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c10f6a0 con 0x7fcc0c1055d0 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc13600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c10f520 con 0x7fcc0c104c10 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc13600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c10f820 con 0x7fcc0c103a20 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc10c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc0c103a20 0x7fcc0c079590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc10c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc0c103a20 0x7fcc0c079590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52390/0 (socket says 172.21.15.7:52390) 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc10c00640 1 -- 172.21.15.7:0/2750394294 learned_addr learned my addr 172.21.15.7:0/2750394294 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:04.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.766+0000 7fcc0be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc0c104c10 0x7fcc0c079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc11600640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc0c1055d0 0x7fcc0c076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc10c00640 1 -- 172.21.15.7:0/2750394294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc0c104c10 msgr2=0x7fcc0c079ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc10c00640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc0c104c10 0x7fcc0c079ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc10c00640 1 -- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc0c1055d0 msgr2=0x7fcc0c076100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc10c00640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc0c1055d0 0x7fcc0c076100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc10c00640 1 -- 172.21.15.7:0/2750394294 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c0769c0 con 0x7fcc0c103a20 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc10c00640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc0c103a20 0x7fcc0c079590 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000e9a0 tx=0x7fcc0000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc09600640 1 -- 172.21.15.7:0/2750394294 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0000cd80 con 0x7fcc0c103a20 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc09600640 1 -- 172.21.15.7:0/2750394294 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc00010910 con 0x7fcc0c103a20 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc09600640 1 -- 172.21.15.7:0/2750394294 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc00018a30 con 0x7fcc0c103a20 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.767+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0c1a6160 con 0x7fcc0c103a20 2024-03-18T01:38:04.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.768+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0c1a6510 con 0x7fcc0c103a20 2024-03-18T01:38:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.769+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc0c068ab0 con 0x7fcc0c103a20 2024-03-18T01:38:04.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.770+0000 7fcc09600640 1 -- 172.21.15.7:0/2750394294 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fcc00010ac0 con 0x7fcc0c103a20 2024-03-18T01:38:04.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.773+0000 7fcc09600640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcbf40a2880 0x7fcbf40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:04.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.773+0000 7fcc09600640 1 -- 172.21.15.7:0/2750394294 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcc000cf1b0 con 0x7fcc0c103a20 2024-03-18T01:38:04.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.773+0000 7fcc0be00640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcbf40a2880 0x7fcbf40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:04.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.774+0000 7fcc0be00640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcbf40a2880 0x7fcbf40a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009830 tx=0x7fcbf8034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:04.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.775+0000 7fcc09600640 1 -- 172.21.15.7:0/2750394294 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc00053740 con 0x7fcc0c103a20 2024-03-18T01:38:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.991+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcc0c10ad40 con 0x7fcc0c103a20 2024-03-18T01:38:04.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.991+0000 7fcc09600640 1 -- 172.21.15.7:0/2750394294 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fcc0c10ad40 con 0x7fcc0c103a20 2024-03-18T01:38:04.992 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:38:04.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcbf40a2880 msgr2=0x7fcbf40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcbf40a2880 0x7fcbf40a4d40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009830 tx=0x7fcbf8034ab0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc0c103a20 msgr2=0x7fcc0c079590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc0c103a20 0x7fcc0c079590 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000e9a0 tx=0x7fcc0000ee60 comp rx=0 tx=0).stop 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 shutdown_connections 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcbf40a2880 0x7fcbf40a4d40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcc0c1055d0 0x7fcc0c076100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc0c104c10 0x7fcc0c079ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 --2- 172.21.15.7:0/2750394294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc0c103a20 0x7fcc0c079590 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.994+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 >> 172.21.15.7:0/2750394294 conn(0x7fcc0c0fef90 msgr2=0x7fcc0c107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.995+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 shutdown_connections 2024-03-18T01:38:04.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:04.995+0000 7fcc13600640 1 -- 172.21.15.7:0/2750394294 wait complete. 2024-03-18T01:38:05.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:04 smithi007 ceph-mon[31996]: pgmap v414: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T01:38:05.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:04 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/298894529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:05.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:04 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1643703556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:05.014 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:38:05.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:04 smithi046 ceph-mon[36108]: pgmap v414: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T01:38:05.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:04 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/298894529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:05.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:04 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1643703556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:04 smithi203 ceph-mon[36345]: pgmap v414: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T01:38:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:04 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/298894529' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:04 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1643703556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:05.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 -- 172.21.15.7:0/1340375784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281000f0 msgr2=0x7f3328100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 --2- 172.21.15.7:0/1340375784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281000f0 0x7f3328100510 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3318009900 tx=0x7f331802bcd0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 -- 172.21.15.7:0/1340375784 shutdown_connections 2024-03-18T01:38:05.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 --2- 172.21.15.7:0/1340375784 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f33281013f0 0x7f3328101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 --2- 172.21.15.7:0/1340375784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3328100a50 0x7f3328100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 --2- 172.21.15.7:0/1340375784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281000f0 0x7f3328100510 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 -- 172.21.15.7:0/1340375784 >> 172.21.15.7:0/1340375784 conn(0x7f33280fbc70 msgr2=0x7f33280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:05.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 -- 172.21.15.7:0/1340375784 shutdown_connections 2024-03-18T01:38:05.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.244+0000 7f332d000640 1 -- 172.21.15.7:0/1340375784 wait complete. 2024-03-18T01:38:05.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.245+0000 7f332d000640 1 Processor -- start 2024-03-18T01:38:05.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.245+0000 7f332d000640 1 -- start start 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f332d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33281000f0 0x7f3328199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f332d000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3328100a50 0x7f332819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f332d000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281013f0 0x7f33281a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f332d000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f332810f650 con 0x7f3328100a50 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f332d000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f332810f4d0 con 0x7f33281013f0 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f332d000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f332810f7d0 con 0x7f33281000f0 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f3325e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33281000f0 0x7f3328199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f3326800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281013f0 0x7f33281a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f3325e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33281000f0 0x7f3328199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52420/0 (socket says 172.21.15.7:52420) 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f3325e00640 1 -- 172.21.15.7:0/1009995873 learned_addr learned my addr 172.21.15.7:0/1009995873 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.246+0000 7f3325400640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3328100a50 0x7f332819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f3325e00640 1 -- 172.21.15.7:0/1009995873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281013f0 msgr2=0x7f33281a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f3325e00640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281013f0 0x7f33281a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f3325e00640 1 -- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3328100a50 msgr2=0x7f332819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f3325e00640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3328100a50 0x7f332819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f3325e00640 1 -- 172.21.15.7:0/1009995873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33281a1950 con 0x7f33281000f0 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f3326800640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281013f0 0x7f33281a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f3325e00640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33281000f0 0x7f3328199d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3318005ab0 tx=0x7f331802c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f331ea00640 1 -- 172.21.15.7:0/1009995873 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3318041070 con 0x7f33281000f0 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f331ea00640 1 -- 172.21.15.7:0/1009995873 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3318004450 con 0x7f33281000f0 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33281a1be0 con 0x7f33281000f0 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.247+0000 7f331ea00640 1 -- 172.21.15.7:0/1009995873 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3318034440 con 0x7f33281000f0 2024-03-18T01:38:05.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.248+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33281a20d0 con 0x7f33281000f0 2024-03-18T01:38:05.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.249+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f332810b030 con 0x7f33281000f0 2024-03-18T01:38:05.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.251+0000 7f331ea00640 1 -- 172.21.15.7:0/1009995873 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f3318044070 con 0x7f33281000f0 2024-03-18T01:38:05.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.252+0000 7f331ea00640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f330c0a2800 0x7f330c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:05.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.253+0000 7f331ea00640 1 -- 172.21.15.7:0/1009995873 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f33180f0940 con 0x7f33281000f0 2024-03-18T01:38:05.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.253+0000 7f3325400640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f330c0a2800 0x7f330c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:05.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.254+0000 7f3325400640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f330c0a2800 0x7f330c0a4cc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3314009bd0 tx=0x7f3314005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:05.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.257+0000 7f331ea00640 1 -- 172.21.15.7:0/1009995873 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f33180bcfd0 con 0x7f33281000f0 2024-03-18T01:38:05.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.466+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f332802d3d0 con 0x7f33281000f0 2024-03-18T01:38:05.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.466+0000 7f331ea00640 1 -- 172.21.15.7:0/1009995873 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f33180c1070 con 0x7f33281000f0 2024-03-18T01:38:05.468 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f330c0a2800 msgr2=0x7f330c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f330c0a2800 0x7f330c0a4cc0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3314009bd0 tx=0x7f3314005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33281000f0 msgr2=0x7f3328199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33281000f0 0x7f3328199d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3318005ab0 tx=0x7f331802c6e0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 shutdown_connections 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33281013f0 0x7f33281a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f330c0a2800 0x7f330c0a4cc0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3328100a50 0x7f332819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 --2- 172.21.15.7:0/1009995873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33281000f0 0x7f3328199d40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.469+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 >> 172.21.15.7:0/1009995873 conn(0x7f33280fbc70 msgr2=0x7f33280fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.470+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 shutdown_connections 2024-03-18T01:38:05.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.470+0000 7f332d000640 1 -- 172.21.15.7:0/1009995873 wait complete. 2024-03-18T01:38:05.491 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-18T01:38:05.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:38:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.725+0000 7fb142e00640 1 -- 172.21.15.7:0/2438995706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c1018d0 msgr2=0x7fb13c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.725+0000 7fb142e00640 1 --2- 172.21.15.7:0/2438995706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c1018d0 0x7fb13c101d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb128009900 tx=0x7fb12802bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.726+0000 7fb142e00640 1 -- 172.21.15.7:0/2438995706 shutdown_connections 2024-03-18T01:38:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.726+0000 7fb142e00640 1 --2- 172.21.15.7:0/2438995706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c102290 0x7fb13c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.726+0000 7fb142e00640 1 --2- 172.21.15.7:0/2438995706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c1018d0 0x7fb13c101d50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.726+0000 7fb142e00640 1 --2- 172.21.15.7:0/2438995706 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c1006e0 0x7fb13c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.726+0000 7fb142e00640 1 -- 172.21.15.7:0/2438995706 >> 172.21.15.7:0/2438995706 conn(0x7fb13c0fbc70 msgr2=0x7fb13c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:05.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.726+0000 7fb142e00640 1 -- 172.21.15.7:0/2438995706 shutdown_connections 2024-03-18T01:38:05.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.726+0000 7fb142e00640 1 -- 172.21.15.7:0/2438995706 wait complete. 2024-03-18T01:38:05.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb142e00640 1 Processor -- start 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb142e00640 1 -- start start 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb142e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c1006e0 0x7fb13c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb142e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c1018d0 0x7fb13c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb142e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c102290 0x7fb13c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb142e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d480 con 0x7fb13c1018d0 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb142e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d300 con 0x7fb13c102290 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb142e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb13c10d600 con 0x7fb13c1006e0 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb13be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c1006e0 0x7fb13c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb13be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c1006e0 0x7fb13c197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:52438/0 (socket says 172.21.15.7:52438) 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb13be00640 1 -- 172.21.15.7:0/1997917221 learned_addr learned my addr 172.21.15.7:0/1997917221 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb13b400640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c1018d0 0x7fb13c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:05.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.727+0000 7fb140e00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c102290 0x7fb13c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb13be00640 1 -- 172.21.15.7:0/1997917221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c102290 msgr2=0x7fb13c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb13be00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c102290 0x7fb13c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb13be00640 1 -- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c1018d0 msgr2=0x7fb13c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb13be00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c1018d0 0x7fb13c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb13be00640 1 -- 172.21.15.7:0/1997917221 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13c19f6a0 con 0x7fb13c1006e0 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb13be00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c1006e0 0x7fb13c197a80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00e9c0 tx=0x7fb12c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb138c00640 1 -- 172.21.15.7:0/1997917221 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c00cde0 con 0x7fb13c1006e0 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb138c00640 1 -- 172.21.15.7:0/1997917221 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb12c010910 con 0x7fb13c1006e0 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13c19f990 con 0x7fb13c1006e0 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.728+0000 7fb138c00640 1 -- 172.21.15.7:0/1997917221 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12c018a00 con 0x7fb13c1006e0 2024-03-18T01:38:05.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.729+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb13c19feb0 con 0x7fb13c1006e0 2024-03-18T01:38:05.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.730+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb13c107a00 con 0x7fb13c1006e0 2024-03-18T01:38:05.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.731+0000 7fb138c00640 1 -- 172.21.15.7:0/1997917221 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fb12c01f020 con 0x7fb13c1006e0 2024-03-18T01:38:05.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.734+0000 7fb138c00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1200a2800 0x7fb1200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:05.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.734+0000 7fb138c00640 1 -- 172.21.15.7:0/1997917221 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb12c0cefe0 con 0x7fb13c1006e0 2024-03-18T01:38:05.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.734+0000 7fb13b400640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1200a2800 0x7fb1200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:05.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.735+0000 7fb13b400640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1200a2800 0x7fb1200a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb12802c7c0 tx=0x7fb128005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:05.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.737+0000 7fb138c00640 1 -- 172.21.15.7:0/1997917221 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb12c09bce0 con 0x7fb13c1006e0 2024-03-18T01:38:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.991+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb13c02d1b0 con 0x7fb13c1006e0 2024-03-18T01:38:05.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.992+0000 7fb138c00640 1 -- 172.21.15.7:0/1997917221 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11918 (secure 0 0 0) 0x7fb12c09b5e0 con 0x7fb13c1006e0 2024-03-18T01:38:05.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:05.993 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:37:59.950482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24562,24596,24542,24544,24523]},"id":1}]} 2024-03-18T01:38:05.994 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2024-03-18T01:38:05.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1200a2800 msgr2=0x7fb1200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1200a2800 0x7fb1200a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb12802c7c0 tx=0x7fb128005fd0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c1006e0 msgr2=0x7fb13c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c1006e0 0x7fb13c197a80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00e9c0 tx=0x7fb12c00ee80 comp rx=0 tx=0).stop 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 shutdown_connections 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb13c102290 0x7fb13c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1200a2800 0x7fb1200a4cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb13c1018d0 0x7fb13c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 --2- 172.21.15.7:0/1997917221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb13c1006e0 0x7fb13c197a80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.994+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 >> 172.21.15.7:0/1997917221 conn(0x7fb13c0fbc70 msgr2=0x7fb13c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.995+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 shutdown_connections 2024-03-18T01:38:05.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:05.995+0000 7fb142e00640 1 -- 172.21.15.7:0/1997917221 wait complete. 2024-03-18T01:38:06.011 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:05 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2750394294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:06.011 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:05 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1009995873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:06.015 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-18T01:38:06.015 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-18T01:38:06.015 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T01:38:06.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:05 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2750394294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:06.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:05 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1009995873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:06.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:05 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2750394294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:06.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:05 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1009995873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:06.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.226+0000 7fecb5000640 1 -- 172.21.15.7:0/1864878057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecb0103a30 msgr2=0x7fecb0103e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:06.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.226+0000 7fecb5000640 1 --2- 172.21.15.7:0/1864878057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecb0103a30 0x7fecb0103e50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7feca0009ca0 tx=0x7feca002c100 comp rx=0 tx=0).stop 2024-03-18T01:38:06.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.226+0000 7fecb5000640 1 -- 172.21.15.7:0/1864878057 shutdown_connections 2024-03-18T01:38:06.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.226+0000 7fecb5000640 1 --2- 172.21.15.7:0/1864878057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fecb01055e0 0x7fecb010be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.226+0000 7fecb5000640 1 --2- 172.21.15.7:0/1864878057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecb0104c20 0x7fecb01050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.226+0000 7fecb5000640 1 --2- 172.21.15.7:0/1864878057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecb0103a30 0x7fecb0103e50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.226+0000 7fecb5000640 1 -- 172.21.15.7:0/1864878057 >> 172.21.15.7:0/1864878057 conn(0x7fecb00fefa0 msgr2=0x7fecb01013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:06.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.227+0000 7fecb5000640 1 -- 172.21.15.7:0/1864878057 shutdown_connections 2024-03-18T01:38:06.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.227+0000 7fecb5000640 1 -- 172.21.15.7:0/1864878057 wait complete. 2024-03-18T01:38:06.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.228+0000 7fecb5000640 1 Processor -- start 2024-03-18T01:38:06.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.228+0000 7fecb5000640 1 -- start start 2024-03-18T01:38:06.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.228+0000 7fecb5000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecb0103a30 0x7fecb0199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:06.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.228+0000 7fecb5000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fecb0104c20 0x7fecb019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:06.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.228+0000 7fecb5000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecb01055e0 0x7fecb01a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:06.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.228+0000 7fecb5000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fecb010f680 con 0x7fecb0104c20 2024-03-18T01:38:06.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecb5000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fecb010f500 con 0x7fecb01055e0 2024-03-18T01:38:06.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecb5000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fecb010f800 con 0x7fecb0103a30 2024-03-18T01:38:06.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecb0103a30 0x7fecb0199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:06.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecb0103a30 0x7fecb0199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44552/0 (socket says 172.21.15.7:44552) 2024-03-18T01:38:06.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 -- 172.21.15.7:0/2063497290 learned_addr learned my addr 172.21.15.7:0/2063497290 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:06.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecae800640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecb01055e0 0x7fecb01a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:06.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7feca5400640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fecb0104c20 0x7fecb019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:06.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 -- 172.21.15.7:0/2063497290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecb01055e0 msgr2=0x7fecb01a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:06.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecb01055e0 0x7fecb01a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 -- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fecb0104c20 msgr2=0x7fecb019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:06.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fecb0104c20 0x7fecb019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 -- 172.21.15.7:0/2063497290 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb01a1920 con 0x7fecb0103a30 2024-03-18T01:38:06.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.229+0000 7fecade00640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecb0103a30 0x7fecb0199d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feca0000f80 tx=0x7feca0035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:06.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.230+0000 7feca7400640 1 -- 172.21.15.7:0/2063497290 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0039040 con 0x7fecb0103a30 2024-03-18T01:38:06.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.230+0000 7feca7400640 1 -- 172.21.15.7:0/2063497290 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca0044070 con 0x7fecb0103a30 2024-03-18T01:38:06.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.230+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecb01a1bb0 con 0x7fecb0103a30 2024-03-18T01:38:06.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.230+0000 7feca7400640 1 -- 172.21.15.7:0/2063497290 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca0004430 con 0x7fecb0103a30 2024-03-18T01:38:06.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.230+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecb01a20c0 con 0x7fecb0103a30 2024-03-18T01:38:06.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.231+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecb010ad50 con 0x7fecb0103a30 2024-03-18T01:38:06.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.233+0000 7feca7400640 1 -- 172.21.15.7:0/2063497290 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7feca00045d0 con 0x7fecb0103a30 2024-03-18T01:38:06.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.235+0000 7feca7400640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec940a2880 0x7fec940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:06.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.235+0000 7feca7400640 1 -- 172.21.15.7:0/2063497290 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7feca00f1550 con 0x7fecb0103a30 2024-03-18T01:38:06.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.235+0000 7feca5400640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec940a2880 0x7fec940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:06.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.236+0000 7feca5400640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec940a2880 0x7fec940a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fec98009b60 tx=0x7fec98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:06.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.237+0000 7feca7400640 1 -- 172.21.15.7:0/2063497290 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feca00be350 con 0x7fecb0103a30 2024-03-18T01:38:06.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.435+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fecb0061c20 con 0x7fec940a2880 2024-03-18T01:38:06.452 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.451+0000 7feca7400640 1 -- 172.21.15.7:0/2063497290 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340539 (secure 0 0 0) 0x7fecb0061c20 con 0x7fec940a2880 2024-03-18T01:38:06.458 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:06.460 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.457+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec940a2880 msgr2=0x7fec940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec940a2880 0x7fec940a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fec98009b60 tx=0x7fec98005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecb0103a30 msgr2=0x7fecb0199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecb0103a30 0x7fecb0199d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feca0000f80 tx=0x7feca0035d10 comp rx=0 tx=0).stop 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 shutdown_connections 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fecb01055e0 0x7fecb01a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fec940a2880 0x7fec940a4d40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fecb0104c20 0x7fecb019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 --2- 172.21.15.7:0/2063497290 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fecb0103a30 0x7fecb0199d10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.458+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 >> 172.21.15.7:0/2063497290 conn(0x7fecb00fefa0 msgr2=0x7fecb0107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.459+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 shutdown_connections 2024-03-18T01:38:06.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.459+0000 7fecb5000640 1 -- 172.21.15.7:0/2063497290 wait complete. 2024-03-18T01:38:06.480 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-03-18T01:38:05.255039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664811,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1688,"num_read_kb":970,"num_write":440,"num_write_kb":3094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":860,"ondisk_log_size":860,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367560,"kb_used_data":14248,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574264,"statfs":{"total":1151940427776,"available":1151564046336,"internally_reserved":0,"allocated":14589952,"data_stored":7975925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":61,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005313"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202699+0000","last_change":"2024-03-18T01:34:56.661330+0000","last_active":"2024-03-18T01:36:57.202699+0000","last_peered":"2024-03-18T01:36:57.202699+0000","last_clean":"2024-03-18T01:36:57.202699+0000","last_became_active":"2024-03-18T01:34:56.660949+0000","last_became_peered":"2024-03-18T01:34:56.660949+0000","last_unstale":"2024-03-18T01:36:57.202699+0000","last_undegraded":"2024-03-18T01:36:57.202699+0000","last_fullsized":"2024-03-18T01:36:57.202699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:40.962487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.209926+0000","last_change":"2024-03-18T01:34:56.652417+0000","last_active":"2024-03-18T01:36:57.209926+0000","last_peered":"2024-03-18T01:36:57.209926+0000","last_clean":"2024-03-18T01:36:57.209926+0000","last_became_active":"2024-03-18T01:34:56.651851+0000","last_became_peered":"2024-03-18T01:34:56.651851+0000","last_unstale":"2024-03-18T01:36:57.209926+0000","last_undegraded":"2024-03-18T01:36:57.209926+0000","last_fullsized":"2024-03-18T01:36:57.209926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:14:24.172177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214561+0000","last_change":"2024-03-18T01:34:56.647657+0000","last_active":"2024-03-18T01:36:57.214561+0000","last_peered":"2024-03-18T01:36:57.214561+0000","last_clean":"2024-03-18T01:36:57.214561+0000","last_became_active":"2024-03-18T01:34:56.647241+0000","last_became_peered":"2024-03-18T01:34:56.647241+0000","last_unstale":"2024-03-18T01:36:57.214561+0000","last_undegraded":"2024-03-18T01:36:57.214561+0000","last_fullsized":"2024-03-18T01:36:57.214561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:01.706351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.200510+0000","last_change":"2024-03-18T01:35:58.815017+0000","last_active":"2024-03-18T01:36:57.200510+0000","last_peered":"2024-03-18T01:36:57.200510+0000","last_clean":"2024-03-18T01:36:57.200510+0000","last_became_active":"2024-03-18T01:35:58.814140+0000","last_became_peered":"2024-03-18T01:35:58.814140+0000","last_unstale":"2024-03-18T01:36:57.200510+0000","last_undegraded":"2024-03-18T01:36:57.200510+0000","last_fullsized":"2024-03-18T01:36:57.200510+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:13:21.960119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.212414+0000","last_change":"2024-03-18T01:34:56.643992+0000","last_active":"2024-03-18T01:36:57.212414+0000","last_peered":"2024-03-18T01:36:57.212414+0000","last_clean":"2024-03-18T01:36:57.212414+0000","last_became_active":"2024-03-18T01:34:56.643652+0000","last_became_peered":"2024-03-18T01:34:56.643652+0000","last_unstale":"2024-03-18T01:36:57.212414+0000","last_undegraded":"2024-03-18T01:36:57.212414+0000","last_fullsized":"2024-03-18T01:36:57.212414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:12.005357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202984+0000","last_change":"2024-03-18T01:34:56.649955+0000","last_active":"2024-03-18T01:36:57.202984+0000","last_peered":"2024-03-18T01:36:57.202984+0000","last_clean":"2024-03-18T01:36:57.202984+0000","last_became_active":"2024-03-18T01:34:56.649609+0000","last_became_peered":"2024-03-18T01:34:56.649609+0000","last_unstale":"2024-03-18T01:36:57.202984+0000","last_undegraded":"2024-03-18T01:36:57.202984+0000","last_fullsized":"2024-03-18T01:36:57.202984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:34:28.113733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193161+0000","last_change":"2024-03-18T01:34:56.647561+0000","last_active":"2024-03-18T01:36:57.193161+0000","last_peered":"2024-03-18T01:36:57.193161+0000","last_clean":"2024-03-18T01:36:57.193161+0000","last_became_active":"2024-03-18T01:34:56.647040+0000","last_became_peered":"2024-03-18T01:34:56.647040+0000","last_unstale":"2024-03-18T01:36:57.193161+0000","last_undegraded":"2024-03-18T01:36:57.193161+0000","last_fullsized":"2024-03-18T01:36:57.193161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:42.433751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220346+0000","last_change":"2024-03-18T01:34:56.652073+0000","last_active":"2024-03-18T01:36:57.220346+0000","last_peered":"2024-03-18T01:36:57.220346+0000","last_clean":"2024-03-18T01:36:57.220346+0000","last_became_active":"2024-03-18T01:34:56.651748+0000","last_became_peered":"2024-03-18T01:34:56.651748+0000","last_unstale":"2024-03-18T01:36:57.220346+0000","last_undegraded":"2024-03-18T01:36:57.220346+0000","last_fullsized":"2024-03-18T01:36:57.220346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:33.148823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.200974+0000","last_change":"2024-03-18T01:34:56.645782+0000","last_active":"2024-03-18T01:36:57.200974+0000","last_peered":"2024-03-18T01:36:57.200974+0000","last_clean":"2024-03-18T01:36:57.200974+0000","last_became_active":"2024-03-18T01:34:56.645439+0000","last_became_peered":"2024-03-18T01:34:56.645439+0000","last_unstale":"2024-03-18T01:36:57.200974+0000","last_undegraded":"2024-03-18T01:36:57.200974+0000","last_fullsized":"2024-03-18T01:36:57.200974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:40.242348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.188909+0000","last_change":"2024-03-18T01:34:56.639697+0000","last_active":"2024-03-18T01:36:57.188909+0000","last_peered":"2024-03-18T01:36:57.188909+0000","last_clean":"2024-03-18T01:36:57.188909+0000","last_became_active":"2024-03-18T01:34:56.639339+0000","last_became_peered":"2024-03-18T01:34:56.639339+0000","last_unstale":"2024-03-18T01:36:57.188909+0000","last_undegraded":"2024-03-18T01:36:57.188909+0000","last_fullsized":"2024-03-18T01:36:57.188909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:25:22.741437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.204096+0000","last_change":"2024-03-18T01:34:56.655341+0000","last_active":"2024-03-18T01:36:57.204096+0000","last_peered":"2024-03-18T01:36:57.204096+0000","last_clean":"2024-03-18T01:36:57.204096+0000","last_became_active":"2024-03-18T01:34:56.655015+0000","last_became_peered":"2024-03-18T01:34:56.655015+0000","last_unstale":"2024-03-18T01:36:57.204096+0000","last_undegraded":"2024-03-18T01:36:57.204096+0000","last_fullsized":"2024-03-18T01:36:57.204096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:50.766000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.185565+0000","last_change":"2024-03-18T01:34:56.648399+0000","last_active":"2024-03-18T01:36:57.185565+0000","last_peered":"2024-03-18T01:36:57.185565+0000","last_clean":"2024-03-18T01:36:57.185565+0000","last_became_active":"2024-03-18T01:34:56.648042+0000","last_became_peered":"2024-03-18T01:34:56.648042+0000","last_unstale":"2024-03-18T01:36:57.185565+0000","last_undegraded":"2024-03-18T01:36:57.185565+0000","last_fullsized":"2024-03-18T01:36:57.185565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:04.972230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211790+0000","last_change":"2024-03-18T01:34:56.653465+0000","last_active":"2024-03-18T01:36:57.211790+0000","last_peered":"2024-03-18T01:36:57.211790+0000","last_clean":"2024-03-18T01:36:57.211790+0000","last_became_active":"2024-03-18T01:34:56.651183+0000","last_became_peered":"2024-03-18T01:34:56.651183+0000","last_unstale":"2024-03-18T01:36:57.211790+0000","last_undegraded":"2024-03-18T01:36:57.211790+0000","last_fullsized":"2024-03-18T01:36:57.211790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:57.983330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211583+0000","last_change":"2024-03-18T01:34:56.658845+0000","last_active":"2024-03-18T01:36:57.211583+0000","last_peered":"2024-03-18T01:36:57.211583+0000","last_clean":"2024-03-18T01:36:57.211583+0000","last_became_active":"2024-03-18T01:34:56.658435+0000","last_became_peered":"2024-03-18T01:34:56.658435+0000","last_unstale":"2024-03-18T01:36:57.211583+0000","last_undegraded":"2024-03-18T01:36:57.211583+0000","last_fullsized":"2024-03-18T01:36:57.211583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:02.176325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211816+0000","last_change":"2024-03-18T01:34:56.660752+0000","last_active":"2024-03-18T01:36:57.211816+0000","last_peered":"2024-03-18T01:36:57.211816+0000","last_clean":"2024-03-18T01:36:57.211816+0000","last_became_active":"2024-03-18T01:34:56.660378+0000","last_became_peered":"2024-03-18T01:34:56.660378+0000","last_unstale":"2024-03-18T01:36:57.211816+0000","last_undegraded":"2024-03-18T01:36:57.211816+0000","last_fullsized":"2024-03-18T01:36:57.211816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:13.143481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214270+0000","last_change":"2024-03-18T01:34:56.656589+0000","last_active":"2024-03-18T01:36:57.214270+0000","last_peered":"2024-03-18T01:36:57.214270+0000","last_clean":"2024-03-18T01:36:57.214270+0000","last_became_active":"2024-03-18T01:34:56.656184+0000","last_became_peered":"2024-03-18T01:34:56.656184+0000","last_unstale":"2024-03-18T01:36:57.214270+0000","last_undegraded":"2024-03-18T01:36:57.214270+0000","last_fullsized":"2024-03-18T01:36:57.214270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:56.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202791+0000","last_change":"2024-03-18T01:34:56.661327+0000","last_active":"2024-03-18T01:36:57.202791+0000","last_peered":"2024-03-18T01:36:57.202791+0000","last_clean":"2024-03-18T01:36:57.202791+0000","last_became_active":"2024-03-18T01:34:56.660767+0000","last_became_peered":"2024-03-18T01:34:56.660767+0000","last_unstale":"2024-03-18T01:36:57.202791+0000","last_undegraded":"2024-03-18T01:36:57.202791+0000","last_fullsized":"2024-03-18T01:36:57.202791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:07:59.066028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203762+0000","last_change":"2024-03-18T01:34:56.650149+0000","last_active":"2024-03-18T01:36:57.203762+0000","last_peered":"2024-03-18T01:36:57.203762+0000","last_clean":"2024-03-18T01:36:57.203762+0000","last_became_active":"2024-03-18T01:34:56.649780+0000","last_became_peered":"2024-03-18T01:34:56.649780+0000","last_unstale":"2024-03-18T01:36:57.203762+0000","last_undegraded":"2024-03-18T01:36:57.203762+0000","last_fullsized":"2024-03-18T01:36:57.203762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:41.605563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.188792+0000","last_change":"2024-03-18T01:34:56.653919+0000","last_active":"2024-03-18T01:36:57.188792+0000","last_peered":"2024-03-18T01:36:57.188792+0000","last_clean":"2024-03-18T01:36:57.188792+0000","last_became_active":"2024-03-18T01:34:56.653512+0000","last_became_peered":"2024-03-18T01:34:56.653512+0000","last_unstale":"2024-03-18T01:36:57.188792+0000","last_undegraded":"2024-03-18T01:36:57.188792+0000","last_fullsized":"2024-03-18T01:36:57.188792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:27.419518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810162+0000","last_change":"2024-03-18T01:35:01.149999+0000","last_active":"2024-03-18T01:35:58.810162+0000","last_peered":"2024-03-18T01:35:58.810162+0000","last_clean":"2024-03-18T01:35:58.810162+0000","last_became_active":"2024-03-18T01:35:01.149603+0000","last_became_peered":"2024-03-18T01:35:01.149603+0000","last_unstale":"2024-03-18T01:35:58.810162+0000","last_undegraded":"2024-03-18T01:35:58.810162+0000","last_fullsized":"2024-03-18T01:35:58.810162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:37.619517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":192,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:58.868646+0000","last_change":"2024-03-18T01:34:55.650404+0000","last_active":"2024-03-18T01:37:58.868646+0000","last_peered":"2024-03-18T01:37:58.868646+0000","last_clean":"2024-03-18T01:37:58.868646+0000","last_became_active":"2024-03-18T01:34:55.650068+0000","last_became_peered":"2024-03-18T01:34:55.650068+0000","last_unstale":"2024-03-18T01:37:58.868646+0000","last_undegraded":"2024-03-18T01:37:58.868646+0000","last_fullsized":"2024-03-18T01:37:58.868646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:11.396529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":151,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211353+0000","last_change":"2024-03-18T01:34:56.654265+0000","last_active":"2024-03-18T01:36:57.211353+0000","last_peered":"2024-03-18T01:36:57.211353+0000","last_clean":"2024-03-18T01:36:57.211353+0000","last_became_active":"2024-03-18T01:34:56.653886+0000","last_became_peered":"2024-03-18T01:34:56.653886+0000","last_unstale":"2024-03-18T01:36:57.211353+0000","last_undegraded":"2024-03-18T01:36:57.211353+0000","last_fullsized":"2024-03-18T01:36:57.211353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:01:44.069966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801492+0000","last_change":"2024-03-18T01:35:01.155838+0000","last_active":"2024-03-18T01:35:58.801492+0000","last_peered":"2024-03-18T01:35:58.801492+0000","last_clean":"2024-03-18T01:35:58.801492+0000","last_became_active":"2024-03-18T01:35:01.155391+0000","last_became_peered":"2024-03-18T01:35:01.155391+0000","last_unstale":"2024-03-18T01:35:58.801492+0000","last_undegraded":"2024-03-18T01:35:58.801492+0000","last_fullsized":"2024-03-18T01:35:58.801492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:31.885767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821989+0000","last_change":"2024-03-18T01:34:55.658830+0000","last_active":"2024-03-18T01:35:58.821989+0000","last_peered":"2024-03-18T01:35:58.821989+0000","last_clean":"2024-03-18T01:35:58.821989+0000","last_became_active":"2024-03-18T01:34:55.658395+0000","last_became_peered":"2024-03-18T01:34:55.658395+0000","last_unstale":"2024-03-18T01:35:58.821989+0000","last_undegraded":"2024-03-18T01:35:58.821989+0000","last_fullsized":"2024-03-18T01:35:58.821989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:07.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.215807+0000","last_change":"2024-03-18T01:34:56.656874+0000","last_active":"2024-03-18T01:36:57.215807+0000","last_peered":"2024-03-18T01:36:57.215807+0000","last_clean":"2024-03-18T01:36:57.215807+0000","last_became_active":"2024-03-18T01:34:56.656525+0000","last_became_peered":"2024-03-18T01:34:56.656525+0000","last_unstale":"2024-03-18T01:36:57.215807+0000","last_undegraded":"2024-03-18T01:36:57.215807+0000","last_fullsized":"2024-03-18T01:36:57.215807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:35:19.363586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.822875+0000","last_change":"2024-03-18T01:35:58.822392+0000","last_active":"2024-03-18T01:35:58.822875+0000","last_peered":"2024-03-18T01:35:58.822875+0000","last_clean":"2024-03-18T01:35:58.822875+0000","last_became_active":"2024-03-18T01:35:58.821792+0000","last_became_peered":"2024-03-18T01:35:58.821792+0000","last_unstale":"2024-03-18T01:35:58.822875+0000","last_undegraded":"2024-03-18T01:35:58.822875+0000","last_fullsized":"2024-03-18T01:35:58.822875+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:43:26.345066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802737+0000","last_change":"2024-03-18T01:34:55.648218+0000","last_active":"2024-03-18T01:35:58.802737+0000","last_peered":"2024-03-18T01:35:58.802737+0000","last_clean":"2024-03-18T01:35:58.802737+0000","last_became_active":"2024-03-18T01:34:55.647779+0000","last_became_peered":"2024-03-18T01:34:55.647779+0000","last_unstale":"2024-03-18T01:35:58.802737+0000","last_undegraded":"2024-03-18T01:35:58.802737+0000","last_fullsized":"2024-03-18T01:35:58.802737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:22:08.136973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205385+0000","last_change":"2024-03-18T01:34:56.649900+0000","last_active":"2024-03-18T01:36:57.205385+0000","last_peered":"2024-03-18T01:36:57.205385+0000","last_clean":"2024-03-18T01:36:57.205385+0000","last_became_active":"2024-03-18T01:34:56.649564+0000","last_became_peered":"2024-03-18T01:34:56.649564+0000","last_unstale":"2024-03-18T01:36:57.205385+0000","last_undegraded":"2024-03-18T01:36:57.205385+0000","last_fullsized":"2024-03-18T01:36:57.205385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:29.861861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801607+0000","last_change":"2024-03-18T01:35:01.147062+0000","last_active":"2024-03-18T01:35:58.801607+0000","last_peered":"2024-03-18T01:35:58.801607+0000","last_clean":"2024-03-18T01:35:58.801607+0000","last_became_active":"2024-03-18T01:35:01.146666+0000","last_became_peered":"2024-03-18T01:35:01.146666+0000","last_unstale":"2024-03-18T01:35:58.801607+0000","last_undegraded":"2024-03-18T01:35:58.801607+0000","last_fullsized":"2024-03-18T01:35:58.801607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:15:21.433263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.817437+0000","last_change":"2024-03-18T01:35:58.816601+0000","last_active":"2024-03-18T01:35:58.817437+0000","last_peered":"2024-03-18T01:35:58.817437+0000","last_clean":"2024-03-18T01:35:58.817437+0000","last_became_active":"2024-03-18T01:35:58.808678+0000","last_became_peered":"2024-03-18T01:35:58.808678+0000","last_unstale":"2024-03-18T01:35:58.817437+0000","last_undegraded":"2024-03-18T01:35:58.817437+0000","last_fullsized":"2024-03-18T01:35:58.817437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:37:42.998856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192461+0000","last_change":"2024-03-18T01:34:56.650449+0000","last_active":"2024-03-18T01:36:57.192461+0000","last_peered":"2024-03-18T01:36:57.192461+0000","last_clean":"2024-03-18T01:36:57.192461+0000","last_became_active":"2024-03-18T01:34:56.650103+0000","last_became_peered":"2024-03-18T01:34:56.650103+0000","last_unstale":"2024-03-18T01:36:57.192461+0000","last_undegraded":"2024-03-18T01:36:57.192461+0000","last_fullsized":"2024-03-18T01:36:57.192461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:07:55.003592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809597+0000","last_change":"2024-03-18T01:35:01.132227+0000","last_active":"2024-03-18T01:35:58.809597+0000","last_peered":"2024-03-18T01:35:58.809597+0000","last_clean":"2024-03-18T01:35:58.809597+0000","last_became_active":"2024-03-18T01:35:01.131767+0000","last_became_peered":"2024-03-18T01:35:01.131767+0000","last_unstale":"2024-03-18T01:35:58.809597+0000","last_undegraded":"2024-03-18T01:35:58.809597+0000","last_fullsized":"2024-03-18T01:35:58.809597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:54.335482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":189,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:02.803783+0000","last_change":"2024-03-18T01:34:55.649277+0000","last_active":"2024-03-18T01:38:02.803783+0000","last_peered":"2024-03-18T01:38:02.803783+0000","last_clean":"2024-03-18T01:38:02.803783+0000","last_became_active":"2024-03-18T01:34:55.648917+0000","last_became_peered":"2024-03-18T01:34:55.648917+0000","last_unstale":"2024-03-18T01:38:02.803783+0000","last_undegraded":"2024-03-18T01:38:02.803783+0000","last_fullsized":"2024-03-18T01:38:02.803783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:00.117647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":156,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193004+0000","last_change":"2024-03-18T01:34:56.648394+0000","last_active":"2024-03-18T01:36:57.193004+0000","last_peered":"2024-03-18T01:36:57.193004+0000","last_clean":"2024-03-18T01:36:57.193004+0000","last_became_active":"2024-03-18T01:34:56.648043+0000","last_became_peered":"2024-03-18T01:34:56.648043+0000","last_unstale":"2024-03-18T01:36:57.193004+0000","last_undegraded":"2024-03-18T01:36:57.193004+0000","last_fullsized":"2024-03-18T01:36:57.193004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:54:52.531851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809510+0000","last_change":"2024-03-18T01:35:01.151346+0000","last_active":"2024-03-18T01:35:58.809510+0000","last_peered":"2024-03-18T01:35:58.809510+0000","last_clean":"2024-03-18T01:35:58.809510+0000","last_became_active":"2024-03-18T01:35:01.150916+0000","last_became_peered":"2024-03-18T01:35:01.150916+0000","last_unstale":"2024-03-18T01:35:58.809510+0000","last_undegraded":"2024-03-18T01:35:58.809510+0000","last_fullsized":"2024-03-18T01:35:58.809510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:44:36.000983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804000+0000","last_change":"2024-03-18T01:34:55.648498+0000","last_active":"2024-03-18T01:35:58.804000+0000","last_peered":"2024-03-18T01:35:58.804000+0000","last_clean":"2024-03-18T01:35:58.804000+0000","last_became_active":"2024-03-18T01:34:55.648152+0000","last_became_peered":"2024-03-18T01:34:55.648152+0000","last_unstale":"2024-03-18T01:35:58.804000+0000","last_undegraded":"2024-03-18T01:35:58.804000+0000","last_fullsized":"2024-03-18T01:35:58.804000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:33.138620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211371+0000","last_change":"2024-03-18T01:34:56.651420+0000","last_active":"2024-03-18T01:36:57.211371+0000","last_peered":"2024-03-18T01:36:57.211371+0000","last_clean":"2024-03-18T01:36:57.211371+0000","last_became_active":"2024-03-18T01:34:56.650909+0000","last_became_peered":"2024-03-18T01:34:56.650909+0000","last_unstale":"2024-03-18T01:36:57.211371+0000","last_undegraded":"2024-03-18T01:36:57.211371+0000","last_fullsized":"2024-03-18T01:36:57.211371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:43:40.548970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804705+0000","last_change":"2024-03-18T01:35:01.157759+0000","last_active":"2024-03-18T01:35:58.804705+0000","last_peered":"2024-03-18T01:35:58.804705+0000","last_clean":"2024-03-18T01:35:58.804705+0000","last_became_active":"2024-03-18T01:35:01.157288+0000","last_became_peered":"2024-03-18T01:35:01.157288+0000","last_unstale":"2024-03-18T01:35:58.804705+0000","last_undegraded":"2024-03-18T01:35:58.804705+0000","last_fullsized":"2024-03-18T01:35:58.804705+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:05:41.272727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.818781+0000","last_change":"2024-03-18T01:34:55.650388+0000","last_active":"2024-03-18T01:35:58.818781+0000","last_peered":"2024-03-18T01:35:58.818781+0000","last_clean":"2024-03-18T01:35:58.818781+0000","last_became_active":"2024-03-18T01:34:55.650055+0000","last_became_peered":"2024-03-18T01:34:55.650055+0000","last_unstale":"2024-03-18T01:35:58.818781+0000","last_undegraded":"2024-03-18T01:35:58.818781+0000","last_fullsized":"2024-03-18T01:35:58.818781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:29:40.758561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220432+0000","last_change":"2024-03-18T01:34:56.658634+0000","last_active":"2024-03-18T01:36:57.220432+0000","last_peered":"2024-03-18T01:36:57.220432+0000","last_clean":"2024-03-18T01:36:57.220432+0000","last_became_active":"2024-03-18T01:34:56.657196+0000","last_became_peered":"2024-03-18T01:34:56.657196+0000","last_unstale":"2024-03-18T01:36:57.220432+0000","last_undegraded":"2024-03-18T01:36:57.220432+0000","last_fullsized":"2024-03-18T01:36:57.220432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:12:36.265949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811261+0000","last_change":"2024-03-18T01:35:01.150431+0000","last_active":"2024-03-18T01:35:58.811261+0000","last_peered":"2024-03-18T01:35:58.811261+0000","last_clean":"2024-03-18T01:35:58.811261+0000","last_became_active":"2024-03-18T01:35:01.149981+0000","last_became_peered":"2024-03-18T01:35:01.149981+0000","last_unstale":"2024-03-18T01:35:58.811261+0000","last_undegraded":"2024-03-18T01:35:58.811261+0000","last_fullsized":"2024-03-18T01:35:58.811261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:34:55.371860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807035+0000","last_change":"2024-03-18T01:34:55.649346+0000","last_active":"2024-03-18T01:35:58.807035+0000","last_peered":"2024-03-18T01:35:58.807035+0000","last_clean":"2024-03-18T01:35:58.807035+0000","last_became_active":"2024-03-18T01:34:55.648958+0000","last_became_peered":"2024-03-18T01:34:55.648958+0000","last_unstale":"2024-03-18T01:35:58.807035+0000","last_undegraded":"2024-03-18T01:35:58.807035+0000","last_fullsized":"2024-03-18T01:35:58.807035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:53.788081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.219078+0000","last_change":"2024-03-18T01:34:56.653525+0000","last_active":"2024-03-18T01:36:57.219078+0000","last_peered":"2024-03-18T01:36:57.219078+0000","last_clean":"2024-03-18T01:36:57.219078+0000","last_became_active":"2024-03-18T01:34:56.653218+0000","last_became_peered":"2024-03-18T01:34:56.653218+0000","last_unstale":"2024-03-18T01:36:57.219078+0000","last_undegraded":"2024-03-18T01:36:57.219078+0000","last_fullsized":"2024-03-18T01:36:57.219078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:32:37.479782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816359+0000","last_change":"2024-03-18T01:35:01.135505+0000","last_active":"2024-03-18T01:35:58.816359+0000","last_peered":"2024-03-18T01:35:58.816359+0000","last_clean":"2024-03-18T01:35:58.816359+0000","last_became_active":"2024-03-18T01:35:01.135055+0000","last_became_peered":"2024-03-18T01:35:01.135055+0000","last_unstale":"2024-03-18T01:35:58.816359+0000","last_undegraded":"2024-03-18T01:35:58.816359+0000","last_fullsized":"2024-03-18T01:35:58.816359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:52:35.069313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":337,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:01.802027+0000","last_change":"2024-03-18T01:34:55.659389+0000","last_active":"2024-03-18T01:38:01.802027+0000","last_peered":"2024-03-18T01:38:01.802027+0000","last_clean":"2024-03-18T01:38:01.802027+0000","last_became_active":"2024-03-18T01:34:55.659090+0000","last_became_peered":"2024-03-18T01:34:55.659090+0000","last_unstale":"2024-03-18T01:38:01.802027+0000","last_undegraded":"2024-03-18T01:38:01.802027+0000","last_fullsized":"2024-03-18T01:38:01.802027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:55:51.731788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":304,"num_read_kb":304,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.206644+0000","last_change":"2024-03-18T01:34:56.650411+0000","last_active":"2024-03-18T01:36:57.206644+0000","last_peered":"2024-03-18T01:36:57.206644+0000","last_clean":"2024-03-18T01:36:57.206644+0000","last_became_active":"2024-03-18T01:34:56.650036+0000","last_became_peered":"2024-03-18T01:34:56.650036+0000","last_unstale":"2024-03-18T01:36:57.206644+0000","last_undegraded":"2024-03-18T01:36:57.206644+0000","last_fullsized":"2024-03-18T01:36:57.206644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:59.912207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811599+0000","last_change":"2024-03-18T01:35:58.811137+0000","last_active":"2024-03-18T01:35:58.811599+0000","last_peered":"2024-03-18T01:35:58.811599+0000","last_clean":"2024-03-18T01:35:58.811599+0000","last_became_active":"2024-03-18T01:35:58.810398+0000","last_became_peered":"2024-03-18T01:35:58.810398+0000","last_unstale":"2024-03-18T01:35:58.811599+0000","last_undegraded":"2024-03-18T01:35:58.811599+0000","last_fullsized":"2024-03-18T01:35:58.811599+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:08.271404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807107+0000","last_change":"2024-03-18T01:34:55.647879+0000","last_active":"2024-03-18T01:35:58.807107+0000","last_peered":"2024-03-18T01:35:58.807107+0000","last_clean":"2024-03-18T01:35:58.807107+0000","last_became_active":"2024-03-18T01:34:55.647549+0000","last_became_peered":"2024-03-18T01:34:55.647549+0000","last_unstale":"2024-03-18T01:35:58.807107+0000","last_undegraded":"2024-03-18T01:35:58.807107+0000","last_fullsized":"2024-03-18T01:35:58.807107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:12.261729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.210366+0000","last_change":"2024-03-18T01:35:58.813429+0000","last_active":"2024-03-18T01:36:57.210366+0000","last_peered":"2024-03-18T01:36:57.210366+0000","last_clean":"2024-03-18T01:36:57.210366+0000","last_became_active":"2024-03-18T01:35:58.812659+0000","last_became_peered":"2024-03-18T01:35:58.812659+0000","last_unstale":"2024-03-18T01:36:57.210366+0000","last_undegraded":"2024-03-18T01:36:57.210366+0000","last_fullsized":"2024-03-18T01:36:57.210366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:22:01.273649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802250+0000","last_change":"2024-03-18T01:35:01.153448+0000","last_active":"2024-03-18T01:35:58.802250+0000","last_peered":"2024-03-18T01:35:58.802250+0000","last_clean":"2024-03-18T01:35:58.802250+0000","last_became_active":"2024-03-18T01:35:01.152593+0000","last_became_peered":"2024-03-18T01:35:01.152593+0000","last_unstale":"2024-03-18T01:35:58.802250+0000","last_undegraded":"2024-03-18T01:35:58.802250+0000","last_fullsized":"2024-03-18T01:35:58.802250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:46.363662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808590+0000","last_change":"2024-03-18T01:34:55.655079+0000","last_active":"2024-03-18T01:35:58.808590+0000","last_peered":"2024-03-18T01:35:58.808590+0000","last_clean":"2024-03-18T01:35:58.808590+0000","last_became_active":"2024-03-18T01:34:55.654702+0000","last_became_peered":"2024-03-18T01:34:55.654702+0000","last_unstale":"2024-03-18T01:35:58.808590+0000","last_undegraded":"2024-03-18T01:35:58.808590+0000","last_fullsized":"2024-03-18T01:35:58.808590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:50:46.524614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205015+0000","last_change":"2024-03-18T01:34:56.644441+0000","last_active":"2024-03-18T01:36:57.205015+0000","last_peered":"2024-03-18T01:36:57.205015+0000","last_clean":"2024-03-18T01:36:57.205015+0000","last_became_active":"2024-03-18T01:34:56.643924+0000","last_became_peered":"2024-03-18T01:34:56.643924+0000","last_unstale":"2024-03-18T01:36:57.205015+0000","last_undegraded":"2024-03-18T01:36:57.205015+0000","last_fullsized":"2024-03-18T01:36:57.205015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:44.260300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810329+0000","last_change":"2024-03-18T01:35:01.150022+0000","last_active":"2024-03-18T01:35:58.810329+0000","last_peered":"2024-03-18T01:35:58.810329+0000","last_clean":"2024-03-18T01:35:58.810329+0000","last_became_active":"2024-03-18T01:35:01.149510+0000","last_became_peered":"2024-03-18T01:35:01.149510+0000","last_unstale":"2024-03-18T01:35:58.810329+0000","last_undegraded":"2024-03-18T01:35:58.810329+0000","last_fullsized":"2024-03-18T01:35:58.810329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:40:36.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":378,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:01.802481+0000","last_change":"2024-03-18T01:34:55.642397+0000","last_active":"2024-03-18T01:38:01.802481+0000","last_peered":"2024-03-18T01:38:01.802481+0000","last_clean":"2024-03-18T01:38:01.802481+0000","last_became_active":"2024-03-18T01:34:55.642046+0000","last_became_peered":"2024-03-18T01:34:55.642046+0000","last_unstale":"2024-03-18T01:38:01.802481+0000","last_undegraded":"2024-03-18T01:38:01.802481+0000","last_fullsized":"2024-03-18T01:38:01.802481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:48:31.631414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66945,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":312,"num_read_kb":222,"num_write":40,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.182441+0000","last_change":"2024-03-18T01:34:56.644388+0000","last_active":"2024-03-18T01:36:57.182441+0000","last_peered":"2024-03-18T01:36:57.182441+0000","last_clean":"2024-03-18T01:36:57.182441+0000","last_became_active":"2024-03-18T01:34:56.644014+0000","last_became_peered":"2024-03-18T01:34:56.644014+0000","last_unstale":"2024-03-18T01:36:57.182441+0000","last_undegraded":"2024-03-18T01:36:57.182441+0000","last_fullsized":"2024-03-18T01:36:57.182441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:06:18.193600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.823299+0000","last_change":"2024-03-18T01:35:58.822872+0000","last_active":"2024-03-18T01:35:58.823299+0000","last_peered":"2024-03-18T01:35:58.823299+0000","last_clean":"2024-03-18T01:35:58.823299+0000","last_became_active":"2024-03-18T01:35:58.822325+0000","last_became_peered":"2024-03-18T01:35:58.822325+0000","last_unstale":"2024-03-18T01:35:58.823299+0000","last_undegraded":"2024-03-18T01:35:58.823299+0000","last_fullsized":"2024-03-18T01:35:58.823299+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:54:21.392764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.834419+0000","last_change":"2024-03-18T01:35:58.834419+0000","last_active":"2024-03-18T01:35:58.834419+0000","last_peered":"2024-03-18T01:35:58.834419+0000","last_clean":"2024-03-18T01:35:58.834419+0000","last_became_active":"2024-03-18T01:35:58.822930+0000","last_became_peered":"2024-03-18T01:35:58.822930+0000","last_unstale":"2024-03-18T01:35:58.834419+0000","last_undegraded":"2024-03-18T01:35:58.834419+0000","last_fullsized":"2024-03-18T01:35:58.834419+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:15.777787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.222003+0000","last_change":"2024-03-18T01:34:56.657786+0000","last_active":"2024-03-18T01:36:57.222003+0000","last_peered":"2024-03-18T01:36:57.222003+0000","last_clean":"2024-03-18T01:36:57.222003+0000","last_became_active":"2024-03-18T01:34:56.657432+0000","last_became_peered":"2024-03-18T01:34:56.657432+0000","last_unstale":"2024-03-18T01:36:57.222003+0000","last_undegraded":"2024-03-18T01:36:57.222003+0000","last_fullsized":"2024-03-18T01:36:57.222003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:13:39.498562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804963+0000","last_change":"2024-03-18T01:35:01.150495+0000","last_active":"2024-03-18T01:35:58.804963+0000","last_peered":"2024-03-18T01:35:58.804963+0000","last_clean":"2024-03-18T01:35:58.804963+0000","last_became_active":"2024-03-18T01:35:01.150033+0000","last_became_peered":"2024-03-18T01:35:01.150033+0000","last_unstale":"2024-03-18T01:35:58.804963+0000","last_undegraded":"2024-03-18T01:35:58.804963+0000","last_fullsized":"2024-03-18T01:35:58.804963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:38.433827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807841+0000","last_change":"2024-03-18T01:34:55.658820+0000","last_active":"2024-03-18T01:35:58.807841+0000","last_peered":"2024-03-18T01:35:58.807841+0000","last_clean":"2024-03-18T01:35:58.807841+0000","last_became_active":"2024-03-18T01:34:55.658446+0000","last_became_peered":"2024-03-18T01:34:55.658446+0000","last_unstale":"2024-03-18T01:35:58.807841+0000","last_undegraded":"2024-03-18T01:35:58.807841+0000","last_fullsized":"2024-03-18T01:35:58.807841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:19:21.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.199889+0000","last_change":"2024-03-18T01:34:56.652890+0000","last_active":"2024-03-18T01:36:57.199889+0000","last_peered":"2024-03-18T01:36:57.199889+0000","last_clean":"2024-03-18T01:36:57.199889+0000","last_became_active":"2024-03-18T01:34:56.652444+0000","last_became_peered":"2024-03-18T01:34:56.652444+0000","last_unstale":"2024-03-18T01:36:57.199889+0000","last_undegraded":"2024-03-18T01:36:57.199889+0000","last_fullsized":"2024-03-18T01:36:57.199889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:58:47.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812407+0000","last_change":"2024-03-18T01:35:01.154700+0000","last_active":"2024-03-18T01:35:58.812407+0000","last_peered":"2024-03-18T01:35:58.812407+0000","last_clean":"2024-03-18T01:35:58.812407+0000","last_became_active":"2024-03-18T01:35:01.148453+0000","last_became_peered":"2024-03-18T01:35:01.148453+0000","last_unstale":"2024-03-18T01:35:58.812407+0000","last_undegraded":"2024-03-18T01:35:58.812407+0000","last_fullsized":"2024-03-18T01:35:58.812407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:08:40.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":193,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:03.952949+0000","last_change":"2024-03-18T01:34:55.643861+0000","last_active":"2024-03-18T01:38:03.952949+0000","last_peered":"2024-03-18T01:38:03.952949+0000","last_clean":"2024-03-18T01:38:03.952949+0000","last_became_active":"2024-03-18T01:34:55.643352+0000","last_became_peered":"2024-03-18T01:34:55.643352+0000","last_unstale":"2024-03-18T01:38:03.952949+0000","last_undegraded":"2024-03-18T01:38:03.952949+0000","last_fullsized":"2024-03-18T01:38:03.952949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:56:53.685571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":159,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193499+0000","last_change":"2024-03-18T01:35:58.814042+0000","last_active":"2024-03-18T01:36:57.193499+0000","last_peered":"2024-03-18T01:36:57.193499+0000","last_clean":"2024-03-18T01:36:57.193499+0000","last_became_active":"2024-03-18T01:35:58.813622+0000","last_became_peered":"2024-03-18T01:35:58.813622+0000","last_unstale":"2024-03-18T01:36:57.193499+0000","last_undegraded":"2024-03-18T01:36:57.193499+0000","last_fullsized":"2024-03-18T01:36:57.193499+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:52.815608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.825880+0000","last_change":"2024-03-18T01:35:58.824798+0000","last_active":"2024-03-18T01:35:58.825880+0000","last_peered":"2024-03-18T01:35:58.825880+0000","last_clean":"2024-03-18T01:35:58.825880+0000","last_became_active":"2024-03-18T01:35:58.823278+0000","last_became_peered":"2024-03-18T01:35:58.823278+0000","last_unstale":"2024-03-18T01:35:58.825880+0000","last_undegraded":"2024-03-18T01:35:58.825880+0000","last_fullsized":"2024-03-18T01:35:58.825880+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:18:53.871541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801980+0000","last_change":"2024-03-18T01:34:55.658245+0000","last_active":"2024-03-18T01:35:58.801980+0000","last_peered":"2024-03-18T01:35:58.801980+0000","last_clean":"2024-03-18T01:35:58.801980+0000","last_became_active":"2024-03-18T01:34:55.657916+0000","last_became_peered":"2024-03-18T01:34:55.657916+0000","last_unstale":"2024-03-18T01:35:58.801980+0000","last_undegraded":"2024-03-18T01:35:58.801980+0000","last_fullsized":"2024-03-18T01:35:58.801980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:18:49.207721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192214+0000","last_change":"2024-03-18T01:34:56.648646+0000","last_active":"2024-03-18T01:36:57.192214+0000","last_peered":"2024-03-18T01:36:57.192214+0000","last_clean":"2024-03-18T01:36:57.192214+0000","last_became_active":"2024-03-18T01:34:56.648257+0000","last_became_peered":"2024-03-18T01:34:56.648257+0000","last_unstale":"2024-03-18T01:36:57.192214+0000","last_undegraded":"2024-03-18T01:36:57.192214+0000","last_fullsized":"2024-03-18T01:36:57.192214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:53.788445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807611+0000","last_change":"2024-03-18T01:35:01.136912+0000","last_active":"2024-03-18T01:35:58.807611+0000","last_peered":"2024-03-18T01:35:58.807611+0000","last_clean":"2024-03-18T01:35:58.807611+0000","last_became_active":"2024-03-18T01:35:01.136611+0000","last_became_peered":"2024-03-18T01:35:01.136611+0000","last_unstale":"2024-03-18T01:35:58.807611+0000","last_undegraded":"2024-03-18T01:35:58.807611+0000","last_fullsized":"2024-03-18T01:35:58.807611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:31.089605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802559+0000","last_change":"2024-03-18T01:34:55.648345+0000","last_active":"2024-03-18T01:35:58.802559+0000","last_peered":"2024-03-18T01:35:58.802559+0000","last_clean":"2024-03-18T01:35:58.802559+0000","last_became_active":"2024-03-18T01:34:55.647113+0000","last_became_peered":"2024-03-18T01:34:55.647113+0000","last_unstale":"2024-03-18T01:35:58.802559+0000","last_undegraded":"2024-03-18T01:35:58.802559+0000","last_fullsized":"2024-03-18T01:35:58.802559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:20.904129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190181+0000","last_change":"2024-03-18T01:34:56.649147+0000","last_active":"2024-03-18T01:36:57.190181+0000","last_peered":"2024-03-18T01:36:57.190181+0000","last_clean":"2024-03-18T01:36:57.190181+0000","last_became_active":"2024-03-18T01:34:56.648815+0000","last_became_peered":"2024-03-18T01:34:56.648815+0000","last_unstale":"2024-03-18T01:36:57.190181+0000","last_undegraded":"2024-03-18T01:36:57.190181+0000","last_fullsized":"2024-03-18T01:36:57.190181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:53.560679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802064+0000","last_change":"2024-03-18T01:35:01.156778+0000","last_active":"2024-03-18T01:35:58.802064+0000","last_peered":"2024-03-18T01:35:58.802064+0000","last_clean":"2024-03-18T01:35:58.802064+0000","last_became_active":"2024-03-18T01:35:01.156367+0000","last_became_peered":"2024-03-18T01:35:01.156367+0000","last_unstale":"2024-03-18T01:35:58.802064+0000","last_undegraded":"2024-03-18T01:35:58.802064+0000","last_fullsized":"2024-03-18T01:35:58.802064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:46.223915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810972+0000","last_change":"2024-03-18T01:34:55.658722+0000","last_active":"2024-03-18T01:35:58.810972+0000","last_peered":"2024-03-18T01:35:58.810972+0000","last_clean":"2024-03-18T01:35:58.810972+0000","last_became_active":"2024-03-18T01:34:55.658321+0000","last_became_peered":"2024-03-18T01:34:55.658321+0000","last_unstale":"2024-03-18T01:35:58.810972+0000","last_undegraded":"2024-03-18T01:35:58.810972+0000","last_fullsized":"2024-03-18T01:35:58.810972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:05.753226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.221076+0000","last_change":"2024-03-18T01:34:56.649415+0000","last_active":"2024-03-18T01:36:57.221076+0000","last_peered":"2024-03-18T01:36:57.221076+0000","last_clean":"2024-03-18T01:36:57.221076+0000","last_became_active":"2024-03-18T01:34:56.649019+0000","last_became_peered":"2024-03-18T01:34:56.649019+0000","last_unstale":"2024-03-18T01:36:57.221076+0000","last_undegraded":"2024-03-18T01:36:57.221076+0000","last_fullsized":"2024-03-18T01:36:57.221076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:28.473683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819934+0000","last_change":"2024-03-18T01:35:01.148937+0000","last_active":"2024-03-18T01:35:58.819934+0000","last_peered":"2024-03-18T01:35:58.819934+0000","last_clean":"2024-03-18T01:35:58.819934+0000","last_became_active":"2024-03-18T01:35:01.148415+0000","last_became_peered":"2024-03-18T01:35:01.148415+0000","last_unstale":"2024-03-18T01:35:58.819934+0000","last_undegraded":"2024-03-18T01:35:58.819934+0000","last_fullsized":"2024-03-18T01:35:58.819934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:45.926970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":185,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:59.870480+0000","last_change":"2024-03-18T01:34:55.653443+0000","last_active":"2024-03-18T01:37:59.870480+0000","last_peered":"2024-03-18T01:37:59.870480+0000","last_clean":"2024-03-18T01:37:59.870480+0000","last_became_active":"2024-03-18T01:34:55.653072+0000","last_became_peered":"2024-03-18T01:34:55.653072+0000","last_unstale":"2024-03-18T01:37:59.870480+0000","last_undegraded":"2024-03-18T01:37:59.870480+0000","last_fullsized":"2024-03-18T01:37:59.870480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:26.376636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202535+0000","last_change":"2024-03-18T01:35:58.811178+0000","last_active":"2024-03-18T01:36:57.202535+0000","last_peered":"2024-03-18T01:36:57.202535+0000","last_clean":"2024-03-18T01:36:57.202535+0000","last_became_active":"2024-03-18T01:35:58.810764+0000","last_became_peered":"2024-03-18T01:35:58.810764+0000","last_unstale":"2024-03-18T01:36:57.202535+0000","last_undegraded":"2024-03-18T01:36:57.202535+0000","last_fullsized":"2024-03-18T01:36:57.202535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:49.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819478+0000","last_change":"2024-03-18T01:35:01.152767+0000","last_active":"2024-03-18T01:35:58.819478+0000","last_peered":"2024-03-18T01:35:58.819478+0000","last_clean":"2024-03-18T01:35:58.819478+0000","last_became_active":"2024-03-18T01:35:01.150619+0000","last_became_peered":"2024-03-18T01:35:01.150619+0000","last_unstale":"2024-03-18T01:35:58.819478+0000","last_undegraded":"2024-03-18T01:35:58.819478+0000","last_fullsized":"2024-03-18T01:35:58.819478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:57:28.258251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191740+0000","last_change":"2024-03-18T01:34:56.650541+0000","last_active":"2024-03-18T01:36:57.191740+0000","last_peered":"2024-03-18T01:36:57.191740+0000","last_clean":"2024-03-18T01:36:57.191740+0000","last_became_active":"2024-03-18T01:34:56.650072+0000","last_became_peered":"2024-03-18T01:34:56.650072+0000","last_unstale":"2024-03-18T01:36:57.191740+0000","last_undegraded":"2024-03-18T01:36:57.191740+0000","last_fullsized":"2024-03-18T01:36:57.191740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:51.224883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":189,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:02.950644+0000","last_change":"2024-03-18T01:34:55.664297+0000","last_active":"2024-03-18T01:38:02.950644+0000","last_peered":"2024-03-18T01:38:02.950644+0000","last_clean":"2024-03-18T01:38:02.950644+0000","last_became_active":"2024-03-18T01:34:55.663938+0000","last_became_peered":"2024-03-18T01:34:55.663938+0000","last_unstale":"2024-03-18T01:38:02.950644+0000","last_undegraded":"2024-03-18T01:38:02.950644+0000","last_fullsized":"2024-03-18T01:38:02.950644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:51.771987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":155,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806717+0000","last_change":"2024-03-18T01:35:01.141736+0000","last_active":"2024-03-18T01:35:58.806717+0000","last_peered":"2024-03-18T01:35:58.806717+0000","last_clean":"2024-03-18T01:35:58.806717+0000","last_became_active":"2024-03-18T01:35:01.141381+0000","last_became_peered":"2024-03-18T01:35:01.141381+0000","last_unstale":"2024-03-18T01:35:58.806717+0000","last_undegraded":"2024-03-18T01:35:58.806717+0000","last_fullsized":"2024-03-18T01:35:58.806717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:25:04.190250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216313+0000","last_change":"2024-03-18T01:34:56.666439+0000","last_active":"2024-03-18T01:36:57.216313+0000","last_peered":"2024-03-18T01:36:57.216313+0000","last_clean":"2024-03-18T01:36:57.216313+0000","last_became_active":"2024-03-18T01:34:56.664556+0000","last_became_peered":"2024-03-18T01:34:56.664556+0000","last_unstale":"2024-03-18T01:36:57.216313+0000","last_undegraded":"2024-03-18T01:36:57.216313+0000","last_fullsized":"2024-03-18T01:36:57.216313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:08:48.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802167+0000","last_change":"2024-03-18T01:34:55.656484+0000","last_active":"2024-03-18T01:35:58.802167+0000","last_peered":"2024-03-18T01:35:58.802167+0000","last_clean":"2024-03-18T01:35:58.802167+0000","last_became_active":"2024-03-18T01:34:55.656125+0000","last_became_peered":"2024-03-18T01:34:55.656125+0000","last_unstale":"2024-03-18T01:35:58.802167+0000","last_undegraded":"2024-03-18T01:35:58.802167+0000","last_fullsized":"2024-03-18T01:35:58.802167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:58:25.944403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806027+0000","last_change":"2024-03-18T01:33:35.824651+0000","last_active":"2024-03-18T01:35:58.806027+0000","last_peered":"2024-03-18T01:35:58.806027+0000","last_clean":"2024-03-18T01:35:58.806027+0000","last_became_active":"2024-03-18T01:33:35.824106+0000","last_became_peered":"2024-03-18T01:33:35.824106+0000","last_unstale":"2024-03-18T01:35:58.806027+0000","last_undegraded":"2024-03-18T01:35:58.806027+0000","last_fullsized":"2024-03-18T01:35:58.806027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_clean_scrub_stamp":"2024-03-18T01:29:41.724134+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:06.422316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808524+0000","last_change":"2024-03-18T01:35:01.157735+0000","last_active":"2024-03-18T01:35:58.808524+0000","last_peered":"2024-03-18T01:35:58.808524+0000","last_clean":"2024-03-18T01:35:58.808524+0000","last_became_active":"2024-03-18T01:35:01.156954+0000","last_became_peered":"2024-03-18T01:35:01.156954+0000","last_unstale":"2024-03-18T01:35:58.808524+0000","last_undegraded":"2024-03-18T01:35:58.808524+0000","last_fullsized":"2024-03-18T01:35:58.808524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:00.106809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193743+0000","last_change":"2024-03-18T01:34:56.646808+0000","last_active":"2024-03-18T01:36:57.193743+0000","last_peered":"2024-03-18T01:36:57.193743+0000","last_clean":"2024-03-18T01:36:57.193743+0000","last_became_active":"2024-03-18T01:34:56.646437+0000","last_became_peered":"2024-03-18T01:34:56.646437+0000","last_unstale":"2024-03-18T01:36:57.193743+0000","last_undegraded":"2024-03-18T01:36:57.193743+0000","last_fullsized":"2024-03-18T01:36:57.193743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:46:09.703889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806136+0000","last_change":"2024-03-18T01:34:55.645365+0000","last_active":"2024-03-18T01:35:58.806136+0000","last_peered":"2024-03-18T01:35:58.806136+0000","last_clean":"2024-03-18T01:35:58.806136+0000","last_became_active":"2024-03-18T01:34:55.645018+0000","last_became_peered":"2024-03-18T01:34:55.645018+0000","last_unstale":"2024-03-18T01:35:58.806136+0000","last_undegraded":"2024-03-18T01:35:58.806136+0000","last_fullsized":"2024-03-18T01:35:58.806136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:08:31.869328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804962+0000","last_change":"2024-03-18T01:35:01.147294+0000","last_active":"2024-03-18T01:35:58.804962+0000","last_peered":"2024-03-18T01:35:58.804962+0000","last_clean":"2024-03-18T01:35:58.804962+0000","last_became_active":"2024-03-18T01:35:01.146858+0000","last_became_peered":"2024-03-18T01:35:01.146858+0000","last_unstale":"2024-03-18T01:35:58.804962+0000","last_undegraded":"2024-03-18T01:35:58.804962+0000","last_fullsized":"2024-03-18T01:35:58.804962+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:45:24.010746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802584+0000","last_change":"2024-03-18T01:34:55.658110+0000","last_active":"2024-03-18T01:35:58.802584+0000","last_peered":"2024-03-18T01:35:58.802584+0000","last_clean":"2024-03-18T01:35:58.802584+0000","last_became_active":"2024-03-18T01:34:55.657802+0000","last_became_peered":"2024-03-18T01:34:55.657802+0000","last_unstale":"2024-03-18T01:35:58.802584+0000","last_undegraded":"2024-03-18T01:35:58.802584+0000","last_fullsized":"2024-03-18T01:35:58.802584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:34.264115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205099+0000","last_change":"2024-03-18T01:34:56.656689+0000","last_active":"2024-03-18T01:36:57.205099+0000","last_peered":"2024-03-18T01:36:57.205099+0000","last_clean":"2024-03-18T01:36:57.205099+0000","last_became_active":"2024-03-18T01:34:56.655653+0000","last_became_peered":"2024-03-18T01:34:56.655653+0000","last_unstale":"2024-03-18T01:36:57.205099+0000","last_undegraded":"2024-03-18T01:36:57.205099+0000","last_fullsized":"2024-03-18T01:36:57.205099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:58.370904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816539+0000","last_change":"2024-03-18T01:35:01.137419+0000","last_active":"2024-03-18T01:35:58.816539+0000","last_peered":"2024-03-18T01:35:58.816539+0000","last_clean":"2024-03-18T01:35:58.816539+0000","last_became_active":"2024-03-18T01:35:01.137033+0000","last_became_peered":"2024-03-18T01:35:01.137033+0000","last_unstale":"2024-03-18T01:35:58.816539+0000","last_undegraded":"2024-03-18T01:35:58.816539+0000","last_fullsized":"2024-03-18T01:35:58.816539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:13:48.382329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:05.854415+0000","last_change":"2024-03-18T01:34:55.663134+0000","last_active":"2024-03-18T01:37:05.854415+0000","last_peered":"2024-03-18T01:37:05.854415+0000","last_clean":"2024-03-18T01:37:05.854415+0000","last_became_active":"2024-03-18T01:34:55.662083+0000","last_became_peered":"2024-03-18T01:34:55.662083+0000","last_unstale":"2024-03-18T01:37:05.854415+0000","last_undegraded":"2024-03-18T01:37:05.854415+0000","last_fullsized":"2024-03-18T01:37:05.854415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:01:09.186599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216678+0000","last_change":"2024-03-18T01:34:56.664761+0000","last_active":"2024-03-18T01:36:57.216678+0000","last_peered":"2024-03-18T01:36:57.216678+0000","last_clean":"2024-03-18T01:36:57.216678+0000","last_became_active":"2024-03-18T01:34:56.663657+0000","last_became_peered":"2024-03-18T01:34:56.663657+0000","last_unstale":"2024-03-18T01:36:57.216678+0000","last_undegraded":"2024-03-18T01:36:57.216678+0000","last_fullsized":"2024-03-18T01:36:57.216678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:26:53.075000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809995+0000","last_change":"2024-03-18T01:35:01.133002+0000","last_active":"2024-03-18T01:35:58.809995+0000","last_peered":"2024-03-18T01:35:58.809995+0000","last_clean":"2024-03-18T01:35:58.809995+0000","last_became_active":"2024-03-18T01:35:01.132542+0000","last_became_peered":"2024-03-18T01:35:01.132542+0000","last_unstale":"2024-03-18T01:35:58.809995+0000","last_undegraded":"2024-03-18T01:35:58.809995+0000","last_fullsized":"2024-03-18T01:35:58.809995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:23.823409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801972+0000","last_change":"2024-03-18T01:34:55.648924+0000","last_active":"2024-03-18T01:35:58.801972+0000","last_peered":"2024-03-18T01:35:58.801972+0000","last_clean":"2024-03-18T01:35:58.801972+0000","last_became_active":"2024-03-18T01:34:55.648548+0000","last_became_peered":"2024-03-18T01:34:55.648548+0000","last_unstale":"2024-03-18T01:35:58.801972+0000","last_undegraded":"2024-03-18T01:35:58.801972+0000","last_fullsized":"2024-03-18T01:35:58.801972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:26:38.916856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.221812+0000","last_change":"2024-03-18T01:34:56.663142+0000","last_active":"2024-03-18T01:36:57.221812+0000","last_peered":"2024-03-18T01:36:57.221812+0000","last_clean":"2024-03-18T01:36:57.221812+0000","last_became_active":"2024-03-18T01:34:56.660256+0000","last_became_peered":"2024-03-18T01:34:56.660256+0000","last_unstale":"2024-03-18T01:36:57.221812+0000","last_undegraded":"2024-03-18T01:36:57.221812+0000","last_fullsized":"2024-03-18T01:36:57.221812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:59.503277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801730+0000","last_change":"2024-03-18T01:35:01.136186+0000","last_active":"2024-03-18T01:35:58.801730+0000","last_peered":"2024-03-18T01:35:58.801730+0000","last_clean":"2024-03-18T01:35:58.801730+0000","last_became_active":"2024-03-18T01:35:01.135803+0000","last_became_peered":"2024-03-18T01:35:01.135803+0000","last_unstale":"2024-03-18T01:35:58.801730+0000","last_undegraded":"2024-03-18T01:35:58.801730+0000","last_fullsized":"2024-03-18T01:35:58.801730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:20.027448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806474+0000","last_change":"2024-03-18T01:34:55.663090+0000","last_active":"2024-03-18T01:35:58.806474+0000","last_peered":"2024-03-18T01:35:58.806474+0000","last_clean":"2024-03-18T01:35:58.806474+0000","last_became_active":"2024-03-18T01:34:55.662774+0000","last_became_peered":"2024-03-18T01:34:55.662774+0000","last_unstale":"2024-03-18T01:35:58.806474+0000","last_undegraded":"2024-03-18T01:35:58.806474+0000","last_fullsized":"2024-03-18T01:35:58.806474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:30:35.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.178629+0000","last_change":"2024-03-18T01:34:56.639509+0000","last_active":"2024-03-18T01:36:57.178629+0000","last_peered":"2024-03-18T01:36:57.178629+0000","last_clean":"2024-03-18T01:36:57.178629+0000","last_became_active":"2024-03-18T01:34:56.639144+0000","last_became_peered":"2024-03-18T01:34:56.639144+0000","last_unstale":"2024-03-18T01:36:57.178629+0000","last_undegraded":"2024-03-18T01:36:57.178629+0000","last_fullsized":"2024-03-18T01:36:57.178629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:12:36.220793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812801+0000","last_change":"2024-03-18T01:35:01.141762+0000","last_active":"2024-03-18T01:35:58.812801+0000","last_peered":"2024-03-18T01:35:58.812801+0000","last_clean":"2024-03-18T01:35:58.812801+0000","last_became_active":"2024-03-18T01:35:01.141219+0000","last_became_peered":"2024-03-18T01:35:01.141219+0000","last_unstale":"2024-03-18T01:35:58.812801+0000","last_undegraded":"2024-03-18T01:35:58.812801+0000","last_fullsized":"2024-03-18T01:35:58.812801+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:55.839120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.815926+0000","last_change":"2024-03-18T01:34:55.662530+0000","last_active":"2024-03-18T01:35:58.815926+0000","last_peered":"2024-03-18T01:35:58.815926+0000","last_clean":"2024-03-18T01:35:58.815926+0000","last_became_active":"2024-03-18T01:34:55.661178+0000","last_became_peered":"2024-03-18T01:34:55.661178+0000","last_unstale":"2024-03-18T01:35:58.815926+0000","last_undegraded":"2024-03-18T01:35:58.815926+0000","last_fullsized":"2024-03-18T01:35:58.815926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:47:16.678553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191228+0000","last_change":"2024-03-18T01:34:56.660936+0000","last_active":"2024-03-18T01:36:57.191228+0000","last_peered":"2024-03-18T01:36:57.191228+0000","last_clean":"2024-03-18T01:36:57.191228+0000","last_became_active":"2024-03-18T01:34:56.660513+0000","last_became_peered":"2024-03-18T01:34:56.660513+0000","last_unstale":"2024-03-18T01:36:57.191228+0000","last_undegraded":"2024-03-18T01:36:57.191228+0000","last_fullsized":"2024-03-18T01:36:57.191228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:35:21.037366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802056+0000","last_change":"2024-03-18T01:35:01.136914+0000","last_active":"2024-03-18T01:35:58.802056+0000","last_peered":"2024-03-18T01:35:58.802056+0000","last_clean":"2024-03-18T01:35:58.802056+0000","last_became_active":"2024-03-18T01:35:01.136480+0000","last_became_peered":"2024-03-18T01:35:01.136480+0000","last_unstale":"2024-03-18T01:35:58.802056+0000","last_undegraded":"2024-03-18T01:35:58.802056+0000","last_fullsized":"2024-03-18T01:35:58.802056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:49.262255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:05.892340+0000","last_change":"2024-03-18T01:34:55.654938+0000","last_active":"2024-03-18T01:37:05.892340+0000","last_peered":"2024-03-18T01:37:05.892340+0000","last_clean":"2024-03-18T01:37:05.892340+0000","last_became_active":"2024-03-18T01:34:55.654611+0000","last_became_peered":"2024-03-18T01:34:55.654611+0000","last_unstale":"2024-03-18T01:37:05.892340+0000","last_undegraded":"2024-03-18T01:37:05.892340+0000","last_fullsized":"2024-03-18T01:37:05.892340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:23.210205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220083+0000","last_change":"2024-03-18T01:34:56.650731+0000","last_active":"2024-03-18T01:36:57.220083+0000","last_peered":"2024-03-18T01:36:57.220083+0000","last_clean":"2024-03-18T01:36:57.220083+0000","last_became_active":"2024-03-18T01:34:56.650363+0000","last_became_peered":"2024-03-18T01:34:56.650363+0000","last_unstale":"2024-03-18T01:36:57.220083+0000","last_undegraded":"2024-03-18T01:36:57.220083+0000","last_fullsized":"2024-03-18T01:36:57.220083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:09:50.305019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801699+0000","last_change":"2024-03-18T01:35:01.135101+0000","last_active":"2024-03-18T01:35:58.801699+0000","last_peered":"2024-03-18T01:35:58.801699+0000","last_clean":"2024-03-18T01:35:58.801699+0000","last_became_active":"2024-03-18T01:35:01.134690+0000","last_became_peered":"2024-03-18T01:35:01.134690+0000","last_unstale":"2024-03-18T01:35:58.801699+0000","last_undegraded":"2024-03-18T01:35:58.801699+0000","last_fullsized":"2024-03-18T01:35:58.801699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:33.381629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802175+0000","last_change":"2024-03-18T01:34:55.648108+0000","last_active":"2024-03-18T01:35:58.802175+0000","last_peered":"2024-03-18T01:35:58.802175+0000","last_clean":"2024-03-18T01:35:58.802175+0000","last_became_active":"2024-03-18T01:34:55.647754+0000","last_became_peered":"2024-03-18T01:34:55.647754+0000","last_unstale":"2024-03-18T01:35:58.802175+0000","last_undegraded":"2024-03-18T01:35:58.802175+0000","last_fullsized":"2024-03-18T01:35:58.802175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:05:48.508015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203569+0000","last_change":"2024-03-18T01:34:56.651589+0000","last_active":"2024-03-18T01:36:57.203569+0000","last_peered":"2024-03-18T01:36:57.203569+0000","last_clean":"2024-03-18T01:36:57.203569+0000","last_became_active":"2024-03-18T01:34:56.650929+0000","last_became_peered":"2024-03-18T01:34:56.650929+0000","last_unstale":"2024-03-18T01:36:57.203569+0000","last_undegraded":"2024-03-18T01:36:57.203569+0000","last_fullsized":"2024-03-18T01:36:57.203569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:43.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806823+0000","last_change":"2024-03-18T01:35:01.148347+0000","last_active":"2024-03-18T01:35:58.806823+0000","last_peered":"2024-03-18T01:35:58.806823+0000","last_clean":"2024-03-18T01:35:58.806823+0000","last_became_active":"2024-03-18T01:35:01.147936+0000","last_became_peered":"2024-03-18T01:35:01.147936+0000","last_unstale":"2024-03-18T01:35:58.806823+0000","last_undegraded":"2024-03-18T01:35:58.806823+0000","last_fullsized":"2024-03-18T01:35:58.806823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:12.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":188,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:59.800787+0000","last_change":"2024-03-18T01:34:55.653625+0000","last_active":"2024-03-18T01:37:59.800787+0000","last_peered":"2024-03-18T01:37:59.800787+0000","last_clean":"2024-03-18T01:37:59.800787+0000","last_became_active":"2024-03-18T01:34:55.653186+0000","last_became_peered":"2024-03-18T01:34:55.653186+0000","last_unstale":"2024-03-18T01:37:59.800787+0000","last_undegraded":"2024-03-18T01:37:59.800787+0000","last_fullsized":"2024-03-18T01:37:59.800787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:10:51.193296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1480,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":154,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203868+0000","last_change":"2024-03-18T01:34:56.652703+0000","last_active":"2024-03-18T01:36:57.203868+0000","last_peered":"2024-03-18T01:36:57.203868+0000","last_clean":"2024-03-18T01:36:57.203868+0000","last_became_active":"2024-03-18T01:34:56.651768+0000","last_became_peered":"2024-03-18T01:34:56.651768+0000","last_unstale":"2024-03-18T01:36:57.203868+0000","last_undegraded":"2024-03-18T01:36:57.203868+0000","last_fullsized":"2024-03-18T01:36:57.203868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:59.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809369+0000","last_change":"2024-03-18T01:35:01.144900+0000","last_active":"2024-03-18T01:35:58.809369+0000","last_peered":"2024-03-18T01:35:58.809369+0000","last_clean":"2024-03-18T01:35:58.809369+0000","last_became_active":"2024-03-18T01:35:01.144431+0000","last_became_peered":"2024-03-18T01:35:01.144431+0000","last_unstale":"2024-03-18T01:35:58.809369+0000","last_undegraded":"2024-03-18T01:35:58.809369+0000","last_fullsized":"2024-03-18T01:35:58.809369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:36.199904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808782+0000","last_change":"2024-03-18T01:34:55.661259+0000","last_active":"2024-03-18T01:35:58.808782+0000","last_peered":"2024-03-18T01:35:58.808782+0000","last_clean":"2024-03-18T01:35:58.808782+0000","last_became_active":"2024-03-18T01:34:55.660964+0000","last_became_peered":"2024-03-18T01:34:55.660964+0000","last_unstale":"2024-03-18T01:35:58.808782+0000","last_undegraded":"2024-03-18T01:35:58.808782+0000","last_fullsized":"2024-03-18T01:35:58.808782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:14.723905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213011+0000","last_change":"2024-03-18T01:34:56.663075+0000","last_active":"2024-03-18T01:36:57.213011+0000","last_peered":"2024-03-18T01:36:57.213011+0000","last_clean":"2024-03-18T01:36:57.213011+0000","last_became_active":"2024-03-18T01:34:56.662458+0000","last_became_peered":"2024-03-18T01:34:56.662458+0000","last_unstale":"2024-03-18T01:36:57.213011+0000","last_undegraded":"2024-03-18T01:36:57.213011+0000","last_fullsized":"2024-03-18T01:36:57.213011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:31:31.837493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806874+0000","last_change":"2024-03-18T01:35:01.133317+0000","last_active":"2024-03-18T01:35:58.806874+0000","last_peered":"2024-03-18T01:35:58.806874+0000","last_clean":"2024-03-18T01:35:58.806874+0000","last_became_active":"2024-03-18T01:35:01.131798+0000","last_became_peered":"2024-03-18T01:35:01.131798+0000","last_unstale":"2024-03-18T01:35:58.806874+0000","last_undegraded":"2024-03-18T01:35:58.806874+0000","last_fullsized":"2024-03-18T01:35:58.806874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:55.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819510+0000","last_change":"2024-03-18T01:34:55.660887+0000","last_active":"2024-03-18T01:35:58.819510+0000","last_peered":"2024-03-18T01:35:58.819510+0000","last_clean":"2024-03-18T01:35:58.819510+0000","last_became_active":"2024-03-18T01:34:55.660524+0000","last_became_peered":"2024-03-18T01:34:55.660524+0000","last_unstale":"2024-03-18T01:35:58.819510+0000","last_undegraded":"2024-03-18T01:35:58.819510+0000","last_fullsized":"2024-03-18T01:35:58.819510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:24.816951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192183+0000","last_change":"2024-03-18T01:34:56.654107+0000","last_active":"2024-03-18T01:36:57.192183+0000","last_peered":"2024-03-18T01:36:57.192183+0000","last_clean":"2024-03-18T01:36:57.192183+0000","last_became_active":"2024-03-18T01:34:56.653742+0000","last_became_peered":"2024-03-18T01:34:56.653742+0000","last_unstale":"2024-03-18T01:36:57.192183+0000","last_undegraded":"2024-03-18T01:36:57.192183+0000","last_fullsized":"2024-03-18T01:36:57.192183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:58:46.093589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191185+0000","last_change":"2024-03-18T01:34:56.647818+0000","last_active":"2024-03-18T01:36:57.191185+0000","last_peered":"2024-03-18T01:36:57.191185+0000","last_clean":"2024-03-18T01:36:57.191185+0000","last_became_active":"2024-03-18T01:34:56.647174+0000","last_became_peered":"2024-03-18T01:34:56.647174+0000","last_unstale":"2024-03-18T01:36:57.191185+0000","last_undegraded":"2024-03-18T01:36:57.191185+0000","last_fullsized":"2024-03-18T01:36:57.191185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:55.656083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191353+0000","last_change":"2024-03-18T01:34:56.643221+0000","last_active":"2024-03-18T01:36:57.191353+0000","last_peered":"2024-03-18T01:36:57.191353+0000","last_clean":"2024-03-18T01:36:57.191353+0000","last_became_active":"2024-03-18T01:34:56.642871+0000","last_became_peered":"2024-03-18T01:34:56.642871+0000","last_unstale":"2024-03-18T01:36:57.191353+0000","last_undegraded":"2024-03-18T01:36:57.191353+0000","last_fullsized":"2024-03-18T01:36:57.191353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:23:25.106060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213467+0000","last_change":"2024-03-18T01:34:56.656227+0000","last_active":"2024-03-18T01:36:57.213467+0000","last_peered":"2024-03-18T01:36:57.213467+0000","last_clean":"2024-03-18T01:36:57.213467+0000","last_became_active":"2024-03-18T01:34:56.655859+0000","last_became_peered":"2024-03-18T01:34:56.655859+0000","last_unstale":"2024-03-18T01:36:57.213467+0000","last_undegraded":"2024-03-18T01:36:57.213467+0000","last_fullsized":"2024-03-18T01:36:57.213467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:51:54.424342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.215195+0000","last_change":"2024-03-18T01:34:56.664655+0000","last_active":"2024-03-18T01:36:57.215195+0000","last_peered":"2024-03-18T01:36:57.215195+0000","last_clean":"2024-03-18T01:36:57.215195+0000","last_became_active":"2024-03-18T01:34:56.662526+0000","last_became_peered":"2024-03-18T01:34:56.662526+0000","last_unstale":"2024-03-18T01:36:57.215195+0000","last_undegraded":"2024-03-18T01:36:57.215195+0000","last_fullsized":"2024-03-18T01:36:57.215195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:45:06.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214796+0000","last_change":"2024-03-18T01:34:56.657045+0000","last_active":"2024-03-18T01:36:57.214796+0000","last_peered":"2024-03-18T01:36:57.214796+0000","last_clean":"2024-03-18T01:36:57.214796+0000","last_became_active":"2024-03-18T01:34:56.656339+0000","last_became_peered":"2024-03-18T01:34:56.656339+0000","last_unstale":"2024-03-18T01:36:57.214796+0000","last_undegraded":"2024-03-18T01:36:57.214796+0000","last_fullsized":"2024-03-18T01:36:57.214796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:41.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.184826+0000","last_change":"2024-03-18T01:34:56.647626+0000","last_active":"2024-03-18T01:36:57.184826+0000","last_peered":"2024-03-18T01:36:57.184826+0000","last_clean":"2024-03-18T01:36:57.184826+0000","last_became_active":"2024-03-18T01:34:56.647216+0000","last_became_peered":"2024-03-18T01:34:56.647216+0000","last_unstale":"2024-03-18T01:36:57.184826+0000","last_undegraded":"2024-03-18T01:36:57.184826+0000","last_fullsized":"2024-03-18T01:36:57.184826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:12:33.330925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.180192+0000","last_change":"2024-03-18T01:34:56.645091+0000","last_active":"2024-03-18T01:36:57.180192+0000","last_peered":"2024-03-18T01:36:57.180192+0000","last_clean":"2024-03-18T01:36:57.180192+0000","last_became_active":"2024-03-18T01:34:56.644725+0000","last_became_peered":"2024-03-18T01:34:56.644725+0000","last_unstale":"2024-03-18T01:36:57.180192+0000","last_undegraded":"2024-03-18T01:36:57.180192+0000","last_fullsized":"2024-03-18T01:36:57.180192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211414+0000","last_change":"2024-03-18T01:34:56.651282+0000","last_active":"2024-03-18T01:36:57.211414+0000","last_peered":"2024-03-18T01:36:57.211414+0000","last_clean":"2024-03-18T01:36:57.211414+0000","last_became_active":"2024-03-18T01:34:56.650872+0000","last_became_peered":"2024-03-18T01:34:56.650872+0000","last_unstale":"2024-03-18T01:36:57.211414+0000","last_undegraded":"2024-03-18T01:36:57.211414+0000","last_fullsized":"2024-03-18T01:36:57.211414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:14:45.106848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211139+0000","last_change":"2024-03-18T01:34:56.651347+0000","last_active":"2024-03-18T01:36:57.211139+0000","last_peered":"2024-03-18T01:36:57.211139+0000","last_clean":"2024-03-18T01:36:57.211139+0000","last_became_active":"2024-03-18T01:34:56.650935+0000","last_became_peered":"2024-03-18T01:34:56.650935+0000","last_unstale":"2024-03-18T01:36:57.211139+0000","last_undegraded":"2024-03-18T01:36:57.211139+0000","last_fullsized":"2024-03-18T01:36:57.211139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:41.849883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211642+0000","last_change":"2024-03-18T01:34:56.667333+0000","last_active":"2024-03-18T01:36:57.211642+0000","last_peered":"2024-03-18T01:36:57.211642+0000","last_clean":"2024-03-18T01:36:57.211642+0000","last_became_active":"2024-03-18T01:34:56.666845+0000","last_became_peered":"2024-03-18T01:34:56.666845+0000","last_unstale":"2024-03-18T01:36:57.211642+0000","last_undegraded":"2024-03-18T01:36:57.211642+0000","last_fullsized":"2024-03-18T01:36:57.211642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:49:23.813455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.217436+0000","last_change":"2024-03-18T01:34:56.662677+0000","last_active":"2024-03-18T01:36:57.217436+0000","last_peered":"2024-03-18T01:36:57.217436+0000","last_clean":"2024-03-18T01:36:57.217436+0000","last_became_active":"2024-03-18T01:34:56.662306+0000","last_became_peered":"2024-03-18T01:34:56.662306+0000","last_unstale":"2024-03-18T01:36:57.217436+0000","last_undegraded":"2024-03-18T01:36:57.217436+0000","last_fullsized":"2024-03-18T01:36:57.217436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:53:56.868984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216916+0000","last_change":"2024-03-18T01:34:56.657721+0000","last_active":"2024-03-18T01:36:57.216916+0000","last_peered":"2024-03-18T01:36:57.216916+0000","last_clean":"2024-03-18T01:36:57.216916+0000","last_became_active":"2024-03-18T01:34:56.657072+0000","last_became_peered":"2024-03-18T01:34:56.657072+0000","last_unstale":"2024-03-18T01:36:57.216916+0000","last_undegraded":"2024-03-18T01:36:57.216916+0000","last_fullsized":"2024-03-18T01:36:57.216916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:21.120391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190992+0000","last_change":"2024-03-18T01:34:56.647327+0000","last_active":"2024-03-18T01:36:57.190992+0000","last_peered":"2024-03-18T01:36:57.190992+0000","last_clean":"2024-03-18T01:36:57.190992+0000","last_became_active":"2024-03-18T01:34:56.646910+0000","last_became_peered":"2024-03-18T01:34:56.646910+0000","last_unstale":"2024-03-18T01:36:57.190992+0000","last_undegraded":"2024-03-18T01:36:57.190992+0000","last_fullsized":"2024-03-18T01:36:57.190992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:37:25.970272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74209,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1562,"num_read_kb":861,"num_write":257,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":222627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383732,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":502393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645368,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1155726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808893,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070533,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":501159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332170,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1098379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495696,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":564842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":560990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182496,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920859,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":501473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757335,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1093429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:38:06.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T01:38:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 -- 172.21.15.7:0/1974214531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41020a0 msgr2=0x7fa7f4108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/1974214531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41020a0 0x7fa7f4108930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec009ca0 tx=0x7fa7ec02c100 comp rx=0 tx=0).stop 2024-03-18T01:38:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 -- 172.21.15.7:0/1974214531 shutdown_connections 2024-03-18T01:38:06.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/1974214531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41020a0 0x7fa7f4108930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/1974214531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f41016e0 0x7fa7f4101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/1974214531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41004f0 0x7fa7f4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 -- 172.21.15.7:0/1974214531 >> 172.21.15.7:0/1974214531 conn(0x7fa7f40fbaa0 msgr2=0x7fa7f40fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:06.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 -- 172.21.15.7:0/1974214531 shutdown_connections 2024-03-18T01:38:06.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.742+0000 7fa7f3e00640 1 -- 172.21.15.7:0/1974214531 wait complete. 2024-03-18T01:38:06.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.743+0000 7fa7f3e00640 1 Processor -- start 2024-03-18T01:38:06.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.743+0000 7fa7f3e00640 1 -- start start 2024-03-18T01:38:06.745 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f41004f0 0x7fa7f41999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41016e0 0x7fa7f4199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41020a0 0x7fa7f41a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f3e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f410c150 con 0x7fa7f41004f0 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f3e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f410bfd0 con 0x7fa7f41016e0 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f3e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f410c2d0 con 0x7fa7f41020a0 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f41004f0 0x7fa7f41999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7ea000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41016e0 0x7fa7f4199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f41004f0 0x7fa7f41999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60014/0 (socket says 172.21.15.7:60014) 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7ea000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41016e0 0x7fa7f4199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:42304/0 (socket says 172.21.15.7:42304) 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f2a00640 1 -- 172.21.15.7:0/3149741907 learned_addr learned my addr 172.21.15.7:0/3149741907 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.744+0000 7fa7f3400640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41020a0 0x7fa7f41a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7ea000640 1 -- 172.21.15.7:0/3149741907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41020a0 msgr2=0x7fa7f41a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:06.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7ea000640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41020a0 0x7fa7f41a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7ea000640 1 -- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f41004f0 msgr2=0x7fa7f41999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:06.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7ea000640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f41004f0 0x7fa7f41999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:06.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7ea000640 1 -- 172.21.15.7:0/3149741907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f41a15e0 con 0x7fa7f41016e0 2024-03-18T01:38:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7ea000640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41016e0 0x7fa7f4199f00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00e9f0 tx=0x7fa7dc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7f8a00640 1 -- 172.21.15.7:0/3149741907 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc00ccc0 con 0x7fa7f41016e0 2024-03-18T01:38:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7f8a00640 1 -- 172.21.15.7:0/3149741907 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7dc010910 con 0x7fa7f41016e0 2024-03-18T01:38:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f41a18d0 con 0x7fa7f41016e0 2024-03-18T01:38:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7f8a00640 1 -- 172.21.15.7:0/3149741907 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc018960 con 0x7fa7f41016e0 2024-03-18T01:38:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.745+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f41a1e70 con 0x7fa7f41016e0 2024-03-18T01:38:06.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.746+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7f41a20a0 con 0x7fa7f41016e0 2024-03-18T01:38:06.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.751+0000 7fa7f8a00640 1 -- 172.21.15.7:0/3149741907 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fa7dc010ac0 con 0x7fa7f41016e0 2024-03-18T01:38:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.751+0000 7fa7f8a00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7d80a2920 0x7fa7d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.751+0000 7fa7f8a00640 1 -- 172.21.15.7:0/3149741907 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa7dc0cf110 con 0x7fa7f41016e0 2024-03-18T01:38:06.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.752+0000 7fa7f2a00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7d80a2920 0x7fa7d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:06.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.752+0000 7fa7f2a00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7d80a2920 0x7fa7d80a4de0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40058c0 tx=0x7fa7e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:06.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.756+0000 7fa7f8a00640 1 -- 172.21.15.7:0/3149741907 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa7dc09b7a0 con 0x7fa7f41016e0 2024-03-18T01:38:06.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:06.986+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa7f40fdb10 con 0x7fa7d80a2920 2024-03-18T01:38:07.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.004+0000 7fa7f8a00640 1 -- 172.21.15.7:0/3149741907 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340539 (secure 0 0 0) 0x7fa7f40fdb10 con 0x7fa7d80a2920 2024-03-18T01:38:07.014 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:07.016 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-18T01:38:07.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:06 smithi046 ceph-mon[36108]: pgmap v415: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:07.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:06 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1997917221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:07.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7d80a2920 msgr2=0x7fa7d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7d80a2920 0x7fa7d80a4de0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa7e40058c0 tx=0x7fa7e4005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:07.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41016e0 msgr2=0x7fa7f4199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41016e0 0x7fa7f4199f00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc00e9f0 tx=0x7fa7dc00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 shutdown_connections 2024-03-18T01:38:07.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7f41020a0 0x7fa7f41a0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7f41016e0 0x7fa7f4199f00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7d80a2920 0x7fa7d80a4de0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 --2- 172.21.15.7:0/3149741907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7f41004f0 0x7fa7f41999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.010+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 >> 172.21.15.7:0/3149741907 conn(0x7fa7f40fbaa0 msgr2=0x7fa7f40fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:07.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.011+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 shutdown_connections 2024-03-18T01:38:07.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.011+0000 7fa7f3e00640 1 -- 172.21.15.7:0/3149741907 wait complete. 2024-03-18T01:38:07.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:06 smithi007 ceph-mon[31996]: pgmap v415: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:07.031 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:06 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1997917221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:07.034 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-03-18T01:38:05.255039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664811,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1688,"num_read_kb":970,"num_write":440,"num_write_kb":3094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":860,"ondisk_log_size":860,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367560,"kb_used_data":14248,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574264,"statfs":{"total":1151940427776,"available":1151564046336,"internally_reserved":0,"allocated":14589952,"data_stored":7975925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":61,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005313"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202699+0000","last_change":"2024-03-18T01:34:56.661330+0000","last_active":"2024-03-18T01:36:57.202699+0000","last_peered":"2024-03-18T01:36:57.202699+0000","last_clean":"2024-03-18T01:36:57.202699+0000","last_became_active":"2024-03-18T01:34:56.660949+0000","last_became_peered":"2024-03-18T01:34:56.660949+0000","last_unstale":"2024-03-18T01:36:57.202699+0000","last_undegraded":"2024-03-18T01:36:57.202699+0000","last_fullsized":"2024-03-18T01:36:57.202699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:40.962487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.209926+0000","last_change":"2024-03-18T01:34:56.652417+0000","last_active":"2024-03-18T01:36:57.209926+0000","last_peered":"2024-03-18T01:36:57.209926+0000","last_clean":"2024-03-18T01:36:57.209926+0000","last_became_active":"2024-03-18T01:34:56.651851+0000","last_became_peered":"2024-03-18T01:34:56.651851+0000","last_unstale":"2024-03-18T01:36:57.209926+0000","last_undegraded":"2024-03-18T01:36:57.209926+0000","last_fullsized":"2024-03-18T01:36:57.209926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:14:24.172177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214561+0000","last_change":"2024-03-18T01:34:56.647657+0000","last_active":"2024-03-18T01:36:57.214561+0000","last_peered":"2024-03-18T01:36:57.214561+0000","last_clean":"2024-03-18T01:36:57.214561+0000","last_became_active":"2024-03-18T01:34:56.647241+0000","last_became_peered":"2024-03-18T01:34:56.647241+0000","last_unstale":"2024-03-18T01:36:57.214561+0000","last_undegraded":"2024-03-18T01:36:57.214561+0000","last_fullsized":"2024-03-18T01:36:57.214561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:01.706351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.200510+0000","last_change":"2024-03-18T01:35:58.815017+0000","last_active":"2024-03-18T01:36:57.200510+0000","last_peered":"2024-03-18T01:36:57.200510+0000","last_clean":"2024-03-18T01:36:57.200510+0000","last_became_active":"2024-03-18T01:35:58.814140+0000","last_became_peered":"2024-03-18T01:35:58.814140+0000","last_unstale":"2024-03-18T01:36:57.200510+0000","last_undegraded":"2024-03-18T01:36:57.200510+0000","last_fullsized":"2024-03-18T01:36:57.200510+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:13:21.960119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.212414+0000","last_change":"2024-03-18T01:34:56.643992+0000","last_active":"2024-03-18T01:36:57.212414+0000","last_peered":"2024-03-18T01:36:57.212414+0000","last_clean":"2024-03-18T01:36:57.212414+0000","last_became_active":"2024-03-18T01:34:56.643652+0000","last_became_peered":"2024-03-18T01:34:56.643652+0000","last_unstale":"2024-03-18T01:36:57.212414+0000","last_undegraded":"2024-03-18T01:36:57.212414+0000","last_fullsized":"2024-03-18T01:36:57.212414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:12.005357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202984+0000","last_change":"2024-03-18T01:34:56.649955+0000","last_active":"2024-03-18T01:36:57.202984+0000","last_peered":"2024-03-18T01:36:57.202984+0000","last_clean":"2024-03-18T01:36:57.202984+0000","last_became_active":"2024-03-18T01:34:56.649609+0000","last_became_peered":"2024-03-18T01:34:56.649609+0000","last_unstale":"2024-03-18T01:36:57.202984+0000","last_undegraded":"2024-03-18T01:36:57.202984+0000","last_fullsized":"2024-03-18T01:36:57.202984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:34:28.113733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193161+0000","last_change":"2024-03-18T01:34:56.647561+0000","last_active":"2024-03-18T01:36:57.193161+0000","last_peered":"2024-03-18T01:36:57.193161+0000","last_clean":"2024-03-18T01:36:57.193161+0000","last_became_active":"2024-03-18T01:34:56.647040+0000","last_became_peered":"2024-03-18T01:34:56.647040+0000","last_unstale":"2024-03-18T01:36:57.193161+0000","last_undegraded":"2024-03-18T01:36:57.193161+0000","last_fullsized":"2024-03-18T01:36:57.193161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:42.433751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220346+0000","last_change":"2024-03-18T01:34:56.652073+0000","last_active":"2024-03-18T01:36:57.220346+0000","last_peered":"2024-03-18T01:36:57.220346+0000","last_clean":"2024-03-18T01:36:57.220346+0000","last_became_active":"2024-03-18T01:34:56.651748+0000","last_became_peered":"2024-03-18T01:34:56.651748+0000","last_unstale":"2024-03-18T01:36:57.220346+0000","last_undegraded":"2024-03-18T01:36:57.220346+0000","last_fullsized":"2024-03-18T01:36:57.220346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:33.148823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.200974+0000","last_change":"2024-03-18T01:34:56.645782+0000","last_active":"2024-03-18T01:36:57.200974+0000","last_peered":"2024-03-18T01:36:57.200974+0000","last_clean":"2024-03-18T01:36:57.200974+0000","last_became_active":"2024-03-18T01:34:56.645439+0000","last_became_peered":"2024-03-18T01:34:56.645439+0000","last_unstale":"2024-03-18T01:36:57.200974+0000","last_undegraded":"2024-03-18T01:36:57.200974+0000","last_fullsized":"2024-03-18T01:36:57.200974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:40.242348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.188909+0000","last_change":"2024-03-18T01:34:56.639697+0000","last_active":"2024-03-18T01:36:57.188909+0000","last_peered":"2024-03-18T01:36:57.188909+0000","last_clean":"2024-03-18T01:36:57.188909+0000","last_became_active":"2024-03-18T01:34:56.639339+0000","last_became_peered":"2024-03-18T01:34:56.639339+0000","last_unstale":"2024-03-18T01:36:57.188909+0000","last_undegraded":"2024-03-18T01:36:57.188909+0000","last_fullsized":"2024-03-18T01:36:57.188909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:25:22.741437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.204096+0000","last_change":"2024-03-18T01:34:56.655341+0000","last_active":"2024-03-18T01:36:57.204096+0000","last_peered":"2024-03-18T01:36:57.204096+0000","last_clean":"2024-03-18T01:36:57.204096+0000","last_became_active":"2024-03-18T01:34:56.655015+0000","last_became_peered":"2024-03-18T01:34:56.655015+0000","last_unstale":"2024-03-18T01:36:57.204096+0000","last_undegraded":"2024-03-18T01:36:57.204096+0000","last_fullsized":"2024-03-18T01:36:57.204096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:50.766000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.185565+0000","last_change":"2024-03-18T01:34:56.648399+0000","last_active":"2024-03-18T01:36:57.185565+0000","last_peered":"2024-03-18T01:36:57.185565+0000","last_clean":"2024-03-18T01:36:57.185565+0000","last_became_active":"2024-03-18T01:34:56.648042+0000","last_became_peered":"2024-03-18T01:34:56.648042+0000","last_unstale":"2024-03-18T01:36:57.185565+0000","last_undegraded":"2024-03-18T01:36:57.185565+0000","last_fullsized":"2024-03-18T01:36:57.185565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:04.972230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211790+0000","last_change":"2024-03-18T01:34:56.653465+0000","last_active":"2024-03-18T01:36:57.211790+0000","last_peered":"2024-03-18T01:36:57.211790+0000","last_clean":"2024-03-18T01:36:57.211790+0000","last_became_active":"2024-03-18T01:34:56.651183+0000","last_became_peered":"2024-03-18T01:34:56.651183+0000","last_unstale":"2024-03-18T01:36:57.211790+0000","last_undegraded":"2024-03-18T01:36:57.211790+0000","last_fullsized":"2024-03-18T01:36:57.211790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:57.983330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211583+0000","last_change":"2024-03-18T01:34:56.658845+0000","last_active":"2024-03-18T01:36:57.211583+0000","last_peered":"2024-03-18T01:36:57.211583+0000","last_clean":"2024-03-18T01:36:57.211583+0000","last_became_active":"2024-03-18T01:34:56.658435+0000","last_became_peered":"2024-03-18T01:34:56.658435+0000","last_unstale":"2024-03-18T01:36:57.211583+0000","last_undegraded":"2024-03-18T01:36:57.211583+0000","last_fullsized":"2024-03-18T01:36:57.211583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:02.176325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211816+0000","last_change":"2024-03-18T01:34:56.660752+0000","last_active":"2024-03-18T01:36:57.211816+0000","last_peered":"2024-03-18T01:36:57.211816+0000","last_clean":"2024-03-18T01:36:57.211816+0000","last_became_active":"2024-03-18T01:34:56.660378+0000","last_became_peered":"2024-03-18T01:34:56.660378+0000","last_unstale":"2024-03-18T01:36:57.211816+0000","last_undegraded":"2024-03-18T01:36:57.211816+0000","last_fullsized":"2024-03-18T01:36:57.211816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:13.143481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214270+0000","last_change":"2024-03-18T01:34:56.656589+0000","last_active":"2024-03-18T01:36:57.214270+0000","last_peered":"2024-03-18T01:36:57.214270+0000","last_clean":"2024-03-18T01:36:57.214270+0000","last_became_active":"2024-03-18T01:34:56.656184+0000","last_became_peered":"2024-03-18T01:34:56.656184+0000","last_unstale":"2024-03-18T01:36:57.214270+0000","last_undegraded":"2024-03-18T01:36:57.214270+0000","last_fullsized":"2024-03-18T01:36:57.214270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:56.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202791+0000","last_change":"2024-03-18T01:34:56.661327+0000","last_active":"2024-03-18T01:36:57.202791+0000","last_peered":"2024-03-18T01:36:57.202791+0000","last_clean":"2024-03-18T01:36:57.202791+0000","last_became_active":"2024-03-18T01:34:56.660767+0000","last_became_peered":"2024-03-18T01:34:56.660767+0000","last_unstale":"2024-03-18T01:36:57.202791+0000","last_undegraded":"2024-03-18T01:36:57.202791+0000","last_fullsized":"2024-03-18T01:36:57.202791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:07:59.066028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203762+0000","last_change":"2024-03-18T01:34:56.650149+0000","last_active":"2024-03-18T01:36:57.203762+0000","last_peered":"2024-03-18T01:36:57.203762+0000","last_clean":"2024-03-18T01:36:57.203762+0000","last_became_active":"2024-03-18T01:34:56.649780+0000","last_became_peered":"2024-03-18T01:34:56.649780+0000","last_unstale":"2024-03-18T01:36:57.203762+0000","last_undegraded":"2024-03-18T01:36:57.203762+0000","last_fullsized":"2024-03-18T01:36:57.203762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:41.605563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.188792+0000","last_change":"2024-03-18T01:34:56.653919+0000","last_active":"2024-03-18T01:36:57.188792+0000","last_peered":"2024-03-18T01:36:57.188792+0000","last_clean":"2024-03-18T01:36:57.188792+0000","last_became_active":"2024-03-18T01:34:56.653512+0000","last_became_peered":"2024-03-18T01:34:56.653512+0000","last_unstale":"2024-03-18T01:36:57.188792+0000","last_undegraded":"2024-03-18T01:36:57.188792+0000","last_fullsized":"2024-03-18T01:36:57.188792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:27.419518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810162+0000","last_change":"2024-03-18T01:35:01.149999+0000","last_active":"2024-03-18T01:35:58.810162+0000","last_peered":"2024-03-18T01:35:58.810162+0000","last_clean":"2024-03-18T01:35:58.810162+0000","last_became_active":"2024-03-18T01:35:01.149603+0000","last_became_peered":"2024-03-18T01:35:01.149603+0000","last_unstale":"2024-03-18T01:35:58.810162+0000","last_undegraded":"2024-03-18T01:35:58.810162+0000","last_fullsized":"2024-03-18T01:35:58.810162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:37.619517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":192,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:58.868646+0000","last_change":"2024-03-18T01:34:55.650404+0000","last_active":"2024-03-18T01:37:58.868646+0000","last_peered":"2024-03-18T01:37:58.868646+0000","last_clean":"2024-03-18T01:37:58.868646+0000","last_became_active":"2024-03-18T01:34:55.650068+0000","last_became_peered":"2024-03-18T01:34:55.650068+0000","last_unstale":"2024-03-18T01:37:58.868646+0000","last_undegraded":"2024-03-18T01:37:58.868646+0000","last_fullsized":"2024-03-18T01:37:58.868646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:11.396529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":151,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211353+0000","last_change":"2024-03-18T01:34:56.654265+0000","last_active":"2024-03-18T01:36:57.211353+0000","last_peered":"2024-03-18T01:36:57.211353+0000","last_clean":"2024-03-18T01:36:57.211353+0000","last_became_active":"2024-03-18T01:34:56.653886+0000","last_became_peered":"2024-03-18T01:34:56.653886+0000","last_unstale":"2024-03-18T01:36:57.211353+0000","last_undegraded":"2024-03-18T01:36:57.211353+0000","last_fullsized":"2024-03-18T01:36:57.211353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:01:44.069966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801492+0000","last_change":"2024-03-18T01:35:01.155838+0000","last_active":"2024-03-18T01:35:58.801492+0000","last_peered":"2024-03-18T01:35:58.801492+0000","last_clean":"2024-03-18T01:35:58.801492+0000","last_became_active":"2024-03-18T01:35:01.155391+0000","last_became_peered":"2024-03-18T01:35:01.155391+0000","last_unstale":"2024-03-18T01:35:58.801492+0000","last_undegraded":"2024-03-18T01:35:58.801492+0000","last_fullsized":"2024-03-18T01:35:58.801492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:31.885767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821989+0000","last_change":"2024-03-18T01:34:55.658830+0000","last_active":"2024-03-18T01:35:58.821989+0000","last_peered":"2024-03-18T01:35:58.821989+0000","last_clean":"2024-03-18T01:35:58.821989+0000","last_became_active":"2024-03-18T01:34:55.658395+0000","last_became_peered":"2024-03-18T01:34:55.658395+0000","last_unstale":"2024-03-18T01:35:58.821989+0000","last_undegraded":"2024-03-18T01:35:58.821989+0000","last_fullsized":"2024-03-18T01:35:58.821989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:07.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.215807+0000","last_change":"2024-03-18T01:34:56.656874+0000","last_active":"2024-03-18T01:36:57.215807+0000","last_peered":"2024-03-18T01:36:57.215807+0000","last_clean":"2024-03-18T01:36:57.215807+0000","last_became_active":"2024-03-18T01:34:56.656525+0000","last_became_peered":"2024-03-18T01:34:56.656525+0000","last_unstale":"2024-03-18T01:36:57.215807+0000","last_undegraded":"2024-03-18T01:36:57.215807+0000","last_fullsized":"2024-03-18T01:36:57.215807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:35:19.363586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.822875+0000","last_change":"2024-03-18T01:35:58.822392+0000","last_active":"2024-03-18T01:35:58.822875+0000","last_peered":"2024-03-18T01:35:58.822875+0000","last_clean":"2024-03-18T01:35:58.822875+0000","last_became_active":"2024-03-18T01:35:58.821792+0000","last_became_peered":"2024-03-18T01:35:58.821792+0000","last_unstale":"2024-03-18T01:35:58.822875+0000","last_undegraded":"2024-03-18T01:35:58.822875+0000","last_fullsized":"2024-03-18T01:35:58.822875+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:43:26.345066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802737+0000","last_change":"2024-03-18T01:34:55.648218+0000","last_active":"2024-03-18T01:35:58.802737+0000","last_peered":"2024-03-18T01:35:58.802737+0000","last_clean":"2024-03-18T01:35:58.802737+0000","last_became_active":"2024-03-18T01:34:55.647779+0000","last_became_peered":"2024-03-18T01:34:55.647779+0000","last_unstale":"2024-03-18T01:35:58.802737+0000","last_undegraded":"2024-03-18T01:35:58.802737+0000","last_fullsized":"2024-03-18T01:35:58.802737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:22:08.136973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205385+0000","last_change":"2024-03-18T01:34:56.649900+0000","last_active":"2024-03-18T01:36:57.205385+0000","last_peered":"2024-03-18T01:36:57.205385+0000","last_clean":"2024-03-18T01:36:57.205385+0000","last_became_active":"2024-03-18T01:34:56.649564+0000","last_became_peered":"2024-03-18T01:34:56.649564+0000","last_unstale":"2024-03-18T01:36:57.205385+0000","last_undegraded":"2024-03-18T01:36:57.205385+0000","last_fullsized":"2024-03-18T01:36:57.205385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:29.861861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801607+0000","last_change":"2024-03-18T01:35:01.147062+0000","last_active":"2024-03-18T01:35:58.801607+0000","last_peered":"2024-03-18T01:35:58.801607+0000","last_clean":"2024-03-18T01:35:58.801607+0000","last_became_active":"2024-03-18T01:35:01.146666+0000","last_became_peered":"2024-03-18T01:35:01.146666+0000","last_unstale":"2024-03-18T01:35:58.801607+0000","last_undegraded":"2024-03-18T01:35:58.801607+0000","last_fullsized":"2024-03-18T01:35:58.801607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:15:21.433263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.817437+0000","last_change":"2024-03-18T01:35:58.816601+0000","last_active":"2024-03-18T01:35:58.817437+0000","last_peered":"2024-03-18T01:35:58.817437+0000","last_clean":"2024-03-18T01:35:58.817437+0000","last_became_active":"2024-03-18T01:35:58.808678+0000","last_became_peered":"2024-03-18T01:35:58.808678+0000","last_unstale":"2024-03-18T01:35:58.817437+0000","last_undegraded":"2024-03-18T01:35:58.817437+0000","last_fullsized":"2024-03-18T01:35:58.817437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:37:42.998856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192461+0000","last_change":"2024-03-18T01:34:56.650449+0000","last_active":"2024-03-18T01:36:57.192461+0000","last_peered":"2024-03-18T01:36:57.192461+0000","last_clean":"2024-03-18T01:36:57.192461+0000","last_became_active":"2024-03-18T01:34:56.650103+0000","last_became_peered":"2024-03-18T01:34:56.650103+0000","last_unstale":"2024-03-18T01:36:57.192461+0000","last_undegraded":"2024-03-18T01:36:57.192461+0000","last_fullsized":"2024-03-18T01:36:57.192461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:07:55.003592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809597+0000","last_change":"2024-03-18T01:35:01.132227+0000","last_active":"2024-03-18T01:35:58.809597+0000","last_peered":"2024-03-18T01:35:58.809597+0000","last_clean":"2024-03-18T01:35:58.809597+0000","last_became_active":"2024-03-18T01:35:01.131767+0000","last_became_peered":"2024-03-18T01:35:01.131767+0000","last_unstale":"2024-03-18T01:35:58.809597+0000","last_undegraded":"2024-03-18T01:35:58.809597+0000","last_fullsized":"2024-03-18T01:35:58.809597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:54.335482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":189,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:02.803783+0000","last_change":"2024-03-18T01:34:55.649277+0000","last_active":"2024-03-18T01:38:02.803783+0000","last_peered":"2024-03-18T01:38:02.803783+0000","last_clean":"2024-03-18T01:38:02.803783+0000","last_became_active":"2024-03-18T01:34:55.648917+0000","last_became_peered":"2024-03-18T01:34:55.648917+0000","last_unstale":"2024-03-18T01:38:02.803783+0000","last_undegraded":"2024-03-18T01:38:02.803783+0000","last_fullsized":"2024-03-18T01:38:02.803783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:00.117647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":156,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193004+0000","last_change":"2024-03-18T01:34:56.648394+0000","last_active":"2024-03-18T01:36:57.193004+0000","last_peered":"2024-03-18T01:36:57.193004+0000","last_clean":"2024-03-18T01:36:57.193004+0000","last_became_active":"2024-03-18T01:34:56.648043+0000","last_became_peered":"2024-03-18T01:34:56.648043+0000","last_unstale":"2024-03-18T01:36:57.193004+0000","last_undegraded":"2024-03-18T01:36:57.193004+0000","last_fullsized":"2024-03-18T01:36:57.193004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:54:52.531851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809510+0000","last_change":"2024-03-18T01:35:01.151346+0000","last_active":"2024-03-18T01:35:58.809510+0000","last_peered":"2024-03-18T01:35:58.809510+0000","last_clean":"2024-03-18T01:35:58.809510+0000","last_became_active":"2024-03-18T01:35:01.150916+0000","last_became_peered":"2024-03-18T01:35:01.150916+0000","last_unstale":"2024-03-18T01:35:58.809510+0000","last_undegraded":"2024-03-18T01:35:58.809510+0000","last_fullsized":"2024-03-18T01:35:58.809510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:44:36.000983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804000+0000","last_change":"2024-03-18T01:34:55.648498+0000","last_active":"2024-03-18T01:35:58.804000+0000","last_peered":"2024-03-18T01:35:58.804000+0000","last_clean":"2024-03-18T01:35:58.804000+0000","last_became_active":"2024-03-18T01:34:55.648152+0000","last_became_peered":"2024-03-18T01:34:55.648152+0000","last_unstale":"2024-03-18T01:35:58.804000+0000","last_undegraded":"2024-03-18T01:35:58.804000+0000","last_fullsized":"2024-03-18T01:35:58.804000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:33.138620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211371+0000","last_change":"2024-03-18T01:34:56.651420+0000","last_active":"2024-03-18T01:36:57.211371+0000","last_peered":"2024-03-18T01:36:57.211371+0000","last_clean":"2024-03-18T01:36:57.211371+0000","last_became_active":"2024-03-18T01:34:56.650909+0000","last_became_peered":"2024-03-18T01:34:56.650909+0000","last_unstale":"2024-03-18T01:36:57.211371+0000","last_undegraded":"2024-03-18T01:36:57.211371+0000","last_fullsized":"2024-03-18T01:36:57.211371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:43:40.548970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804705+0000","last_change":"2024-03-18T01:35:01.157759+0000","last_active":"2024-03-18T01:35:58.804705+0000","last_peered":"2024-03-18T01:35:58.804705+0000","last_clean":"2024-03-18T01:35:58.804705+0000","last_became_active":"2024-03-18T01:35:01.157288+0000","last_became_peered":"2024-03-18T01:35:01.157288+0000","last_unstale":"2024-03-18T01:35:58.804705+0000","last_undegraded":"2024-03-18T01:35:58.804705+0000","last_fullsized":"2024-03-18T01:35:58.804705+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:05:41.272727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.818781+0000","last_change":"2024-03-18T01:34:55.650388+0000","last_active":"2024-03-18T01:35:58.818781+0000","last_peered":"2024-03-18T01:35:58.818781+0000","last_clean":"2024-03-18T01:35:58.818781+0000","last_became_active":"2024-03-18T01:34:55.650055+0000","last_became_peered":"2024-03-18T01:34:55.650055+0000","last_unstale":"2024-03-18T01:35:58.818781+0000","last_undegraded":"2024-03-18T01:35:58.818781+0000","last_fullsized":"2024-03-18T01:35:58.818781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:29:40.758561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220432+0000","last_change":"2024-03-18T01:34:56.658634+0000","last_active":"2024-03-18T01:36:57.220432+0000","last_peered":"2024-03-18T01:36:57.220432+0000","last_clean":"2024-03-18T01:36:57.220432+0000","last_became_active":"2024-03-18T01:34:56.657196+0000","last_became_peered":"2024-03-18T01:34:56.657196+0000","last_unstale":"2024-03-18T01:36:57.220432+0000","last_undegraded":"2024-03-18T01:36:57.220432+0000","last_fullsized":"2024-03-18T01:36:57.220432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:12:36.265949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811261+0000","last_change":"2024-03-18T01:35:01.150431+0000","last_active":"2024-03-18T01:35:58.811261+0000","last_peered":"2024-03-18T01:35:58.811261+0000","last_clean":"2024-03-18T01:35:58.811261+0000","last_became_active":"2024-03-18T01:35:01.149981+0000","last_became_peered":"2024-03-18T01:35:01.149981+0000","last_unstale":"2024-03-18T01:35:58.811261+0000","last_undegraded":"2024-03-18T01:35:58.811261+0000","last_fullsized":"2024-03-18T01:35:58.811261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:34:55.371860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807035+0000","last_change":"2024-03-18T01:34:55.649346+0000","last_active":"2024-03-18T01:35:58.807035+0000","last_peered":"2024-03-18T01:35:58.807035+0000","last_clean":"2024-03-18T01:35:58.807035+0000","last_became_active":"2024-03-18T01:34:55.648958+0000","last_became_peered":"2024-03-18T01:34:55.648958+0000","last_unstale":"2024-03-18T01:35:58.807035+0000","last_undegraded":"2024-03-18T01:35:58.807035+0000","last_fullsized":"2024-03-18T01:35:58.807035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:53.788081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.219078+0000","last_change":"2024-03-18T01:34:56.653525+0000","last_active":"2024-03-18T01:36:57.219078+0000","last_peered":"2024-03-18T01:36:57.219078+0000","last_clean":"2024-03-18T01:36:57.219078+0000","last_became_active":"2024-03-18T01:34:56.653218+0000","last_became_peered":"2024-03-18T01:34:56.653218+0000","last_unstale":"2024-03-18T01:36:57.219078+0000","last_undegraded":"2024-03-18T01:36:57.219078+0000","last_fullsized":"2024-03-18T01:36:57.219078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:32:37.479782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816359+0000","last_change":"2024-03-18T01:35:01.135505+0000","last_active":"2024-03-18T01:35:58.816359+0000","last_peered":"2024-03-18T01:35:58.816359+0000","last_clean":"2024-03-18T01:35:58.816359+0000","last_became_active":"2024-03-18T01:35:01.135055+0000","last_became_peered":"2024-03-18T01:35:01.135055+0000","last_unstale":"2024-03-18T01:35:58.816359+0000","last_undegraded":"2024-03-18T01:35:58.816359+0000","last_fullsized":"2024-03-18T01:35:58.816359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:52:35.069313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":337,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:01.802027+0000","last_change":"2024-03-18T01:34:55.659389+0000","last_active":"2024-03-18T01:38:01.802027+0000","last_peered":"2024-03-18T01:38:01.802027+0000","last_clean":"2024-03-18T01:38:01.802027+0000","last_became_active":"2024-03-18T01:34:55.659090+0000","last_became_peered":"2024-03-18T01:34:55.659090+0000","last_unstale":"2024-03-18T01:38:01.802027+0000","last_undegraded":"2024-03-18T01:38:01.802027+0000","last_fullsized":"2024-03-18T01:38:01.802027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:55:51.731788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":304,"num_read_kb":304,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.206644+0000","last_change":"2024-03-18T01:34:56.650411+0000","last_active":"2024-03-18T01:36:57.206644+0000","last_peered":"2024-03-18T01:36:57.206644+0000","last_clean":"2024-03-18T01:36:57.206644+0000","last_became_active":"2024-03-18T01:34:56.650036+0000","last_became_peered":"2024-03-18T01:34:56.650036+0000","last_unstale":"2024-03-18T01:36:57.206644+0000","last_undegraded":"2024-03-18T01:36:57.206644+0000","last_fullsized":"2024-03-18T01:36:57.206644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:59.912207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811599+0000","last_change":"2024-03-18T01:35:58.811137+0000","last_active":"2024-03-18T01:35:58.811599+0000","last_peered":"2024-03-18T01:35:58.811599+0000","last_clean":"2024-03-18T01:35:58.811599+0000","last_became_active":"2024-03-18T01:35:58.810398+0000","last_became_peered":"2024-03-18T01:35:58.810398+0000","last_unstale":"2024-03-18T01:35:58.811599+0000","last_undegraded":"2024-03-18T01:35:58.811599+0000","last_fullsized":"2024-03-18T01:35:58.811599+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:08.271404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807107+0000","last_change":"2024-03-18T01:34:55.647879+0000","last_active":"2024-03-18T01:35:58.807107+0000","last_peered":"2024-03-18T01:35:58.807107+0000","last_clean":"2024-03-18T01:35:58.807107+0000","last_became_active":"2024-03-18T01:34:55.647549+0000","last_became_peered":"2024-03-18T01:34:55.647549+0000","last_unstale":"2024-03-18T01:35:58.807107+0000","last_undegraded":"2024-03-18T01:35:58.807107+0000","last_fullsized":"2024-03-18T01:35:58.807107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:12.261729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.210366+0000","last_change":"2024-03-18T01:35:58.813429+0000","last_active":"2024-03-18T01:36:57.210366+0000","last_peered":"2024-03-18T01:36:57.210366+0000","last_clean":"2024-03-18T01:36:57.210366+0000","last_became_active":"2024-03-18T01:35:58.812659+0000","last_became_peered":"2024-03-18T01:35:58.812659+0000","last_unstale":"2024-03-18T01:36:57.210366+0000","last_undegraded":"2024-03-18T01:36:57.210366+0000","last_fullsized":"2024-03-18T01:36:57.210366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:22:01.273649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802250+0000","last_change":"2024-03-18T01:35:01.153448+0000","last_active":"2024-03-18T01:35:58.802250+0000","last_peered":"2024-03-18T01:35:58.802250+0000","last_clean":"2024-03-18T01:35:58.802250+0000","last_became_active":"2024-03-18T01:35:01.152593+0000","last_became_peered":"2024-03-18T01:35:01.152593+0000","last_unstale":"2024-03-18T01:35:58.802250+0000","last_undegraded":"2024-03-18T01:35:58.802250+0000","last_fullsized":"2024-03-18T01:35:58.802250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:46.363662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808590+0000","last_change":"2024-03-18T01:34:55.655079+0000","last_active":"2024-03-18T01:35:58.808590+0000","last_peered":"2024-03-18T01:35:58.808590+0000","last_clean":"2024-03-18T01:35:58.808590+0000","last_became_active":"2024-03-18T01:34:55.654702+0000","last_became_peered":"2024-03-18T01:34:55.654702+0000","last_unstale":"2024-03-18T01:35:58.808590+0000","last_undegraded":"2024-03-18T01:35:58.808590+0000","last_fullsized":"2024-03-18T01:35:58.808590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:50:46.524614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205015+0000","last_change":"2024-03-18T01:34:56.644441+0000","last_active":"2024-03-18T01:36:57.205015+0000","last_peered":"2024-03-18T01:36:57.205015+0000","last_clean":"2024-03-18T01:36:57.205015+0000","last_became_active":"2024-03-18T01:34:56.643924+0000","last_became_peered":"2024-03-18T01:34:56.643924+0000","last_unstale":"2024-03-18T01:36:57.205015+0000","last_undegraded":"2024-03-18T01:36:57.205015+0000","last_fullsized":"2024-03-18T01:36:57.205015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:44.260300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810329+0000","last_change":"2024-03-18T01:35:01.150022+0000","last_active":"2024-03-18T01:35:58.810329+0000","last_peered":"2024-03-18T01:35:58.810329+0000","last_clean":"2024-03-18T01:35:58.810329+0000","last_became_active":"2024-03-18T01:35:01.149510+0000","last_became_peered":"2024-03-18T01:35:01.149510+0000","last_unstale":"2024-03-18T01:35:58.810329+0000","last_undegraded":"2024-03-18T01:35:58.810329+0000","last_fullsized":"2024-03-18T01:35:58.810329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:40:36.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":378,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:01.802481+0000","last_change":"2024-03-18T01:34:55.642397+0000","last_active":"2024-03-18T01:38:01.802481+0000","last_peered":"2024-03-18T01:38:01.802481+0000","last_clean":"2024-03-18T01:38:01.802481+0000","last_became_active":"2024-03-18T01:34:55.642046+0000","last_became_peered":"2024-03-18T01:34:55.642046+0000","last_unstale":"2024-03-18T01:38:01.802481+0000","last_undegraded":"2024-03-18T01:38:01.802481+0000","last_fullsized":"2024-03-18T01:38:01.802481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:48:31.631414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66945,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":312,"num_read_kb":222,"num_write":40,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.182441+0000","last_change":"2024-03-18T01:34:56.644388+0000","last_active":"2024-03-18T01:36:57.182441+0000","last_peered":"2024-03-18T01:36:57.182441+0000","last_clean":"2024-03-18T01:36:57.182441+0000","last_became_active":"2024-03-18T01:34:56.644014+0000","last_became_peered":"2024-03-18T01:34:56.644014+0000","last_unstale":"2024-03-18T01:36:57.182441+0000","last_undegraded":"2024-03-18T01:36:57.182441+0000","last_fullsized":"2024-03-18T01:36:57.182441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:06:18.193600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.823299+0000","last_change":"2024-03-18T01:35:58.822872+0000","last_active":"2024-03-18T01:35:58.823299+0000","last_peered":"2024-03-18T01:35:58.823299+0000","last_clean":"2024-03-18T01:35:58.823299+0000","last_became_active":"2024-03-18T01:35:58.822325+0000","last_became_peered":"2024-03-18T01:35:58.822325+0000","last_unstale":"2024-03-18T01:35:58.823299+0000","last_undegraded":"2024-03-18T01:35:58.823299+0000","last_fullsized":"2024-03-18T01:35:58.823299+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:54:21.392764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.834419+0000","last_change":"2024-03-18T01:35:58.834419+0000","last_active":"2024-03-18T01:35:58.834419+0000","last_peered":"2024-03-18T01:35:58.834419+0000","last_clean":"2024-03-18T01:35:58.834419+0000","last_became_active":"2024-03-18T01:35:58.822930+0000","last_became_peered":"2024-03-18T01:35:58.822930+0000","last_unstale":"2024-03-18T01:35:58.834419+0000","last_undegraded":"2024-03-18T01:35:58.834419+0000","last_fullsized":"2024-03-18T01:35:58.834419+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:15.777787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.222003+0000","last_change":"2024-03-18T01:34:56.657786+0000","last_active":"2024-03-18T01:36:57.222003+0000","last_peered":"2024-03-18T01:36:57.222003+0000","last_clean":"2024-03-18T01:36:57.222003+0000","last_became_active":"2024-03-18T01:34:56.657432+0000","last_became_peered":"2024-03-18T01:34:56.657432+0000","last_unstale":"2024-03-18T01:36:57.222003+0000","last_undegraded":"2024-03-18T01:36:57.222003+0000","last_fullsized":"2024-03-18T01:36:57.222003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:13:39.498562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804963+0000","last_change":"2024-03-18T01:35:01.150495+0000","last_active":"2024-03-18T01:35:58.804963+0000","last_peered":"2024-03-18T01:35:58.804963+0000","last_clean":"2024-03-18T01:35:58.804963+0000","last_became_active":"2024-03-18T01:35:01.150033+0000","last_became_peered":"2024-03-18T01:35:01.150033+0000","last_unstale":"2024-03-18T01:35:58.804963+0000","last_undegraded":"2024-03-18T01:35:58.804963+0000","last_fullsized":"2024-03-18T01:35:58.804963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:38.433827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807841+0000","last_change":"2024-03-18T01:34:55.658820+0000","last_active":"2024-03-18T01:35:58.807841+0000","last_peered":"2024-03-18T01:35:58.807841+0000","last_clean":"2024-03-18T01:35:58.807841+0000","last_became_active":"2024-03-18T01:34:55.658446+0000","last_became_peered":"2024-03-18T01:34:55.658446+0000","last_unstale":"2024-03-18T01:35:58.807841+0000","last_undegraded":"2024-03-18T01:35:58.807841+0000","last_fullsized":"2024-03-18T01:35:58.807841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:19:21.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.199889+0000","last_change":"2024-03-18T01:34:56.652890+0000","last_active":"2024-03-18T01:36:57.199889+0000","last_peered":"2024-03-18T01:36:57.199889+0000","last_clean":"2024-03-18T01:36:57.199889+0000","last_became_active":"2024-03-18T01:34:56.652444+0000","last_became_peered":"2024-03-18T01:34:56.652444+0000","last_unstale":"2024-03-18T01:36:57.199889+0000","last_undegraded":"2024-03-18T01:36:57.199889+0000","last_fullsized":"2024-03-18T01:36:57.199889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:58:47.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812407+0000","last_change":"2024-03-18T01:35:01.154700+0000","last_active":"2024-03-18T01:35:58.812407+0000","last_peered":"2024-03-18T01:35:58.812407+0000","last_clean":"2024-03-18T01:35:58.812407+0000","last_became_active":"2024-03-18T01:35:01.148453+0000","last_became_peered":"2024-03-18T01:35:01.148453+0000","last_unstale":"2024-03-18T01:35:58.812407+0000","last_undegraded":"2024-03-18T01:35:58.812407+0000","last_fullsized":"2024-03-18T01:35:58.812407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:08:40.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":193,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:03.952949+0000","last_change":"2024-03-18T01:34:55.643861+0000","last_active":"2024-03-18T01:38:03.952949+0000","last_peered":"2024-03-18T01:38:03.952949+0000","last_clean":"2024-03-18T01:38:03.952949+0000","last_became_active":"2024-03-18T01:34:55.643352+0000","last_became_peered":"2024-03-18T01:34:55.643352+0000","last_unstale":"2024-03-18T01:38:03.952949+0000","last_undegraded":"2024-03-18T01:38:03.952949+0000","last_fullsized":"2024-03-18T01:38:03.952949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:56:53.685571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":159,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193499+0000","last_change":"2024-03-18T01:35:58.814042+0000","last_active":"2024-03-18T01:36:57.193499+0000","last_peered":"2024-03-18T01:36:57.193499+0000","last_clean":"2024-03-18T01:36:57.193499+0000","last_became_active":"2024-03-18T01:35:58.813622+0000","last_became_peered":"2024-03-18T01:35:58.813622+0000","last_unstale":"2024-03-18T01:36:57.193499+0000","last_undegraded":"2024-03-18T01:36:57.193499+0000","last_fullsized":"2024-03-18T01:36:57.193499+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:52.815608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.825880+0000","last_change":"2024-03-18T01:35:58.824798+0000","last_active":"2024-03-18T01:35:58.825880+0000","last_peered":"2024-03-18T01:35:58.825880+0000","last_clean":"2024-03-18T01:35:58.825880+0000","last_became_active":"2024-03-18T01:35:58.823278+0000","last_became_peered":"2024-03-18T01:35:58.823278+0000","last_unstale":"2024-03-18T01:35:58.825880+0000","last_undegraded":"2024-03-18T01:35:58.825880+0000","last_fullsized":"2024-03-18T01:35:58.825880+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:18:53.871541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801980+0000","last_change":"2024-03-18T01:34:55.658245+0000","last_active":"2024-03-18T01:35:58.801980+0000","last_peered":"2024-03-18T01:35:58.801980+0000","last_clean":"2024-03-18T01:35:58.801980+0000","last_became_active":"2024-03-18T01:34:55.657916+0000","last_became_peered":"2024-03-18T01:34:55.657916+0000","last_unstale":"2024-03-18T01:35:58.801980+0000","last_undegraded":"2024-03-18T01:35:58.801980+0000","last_fullsized":"2024-03-18T01:35:58.801980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:18:49.207721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192214+0000","last_change":"2024-03-18T01:34:56.648646+0000","last_active":"2024-03-18T01:36:57.192214+0000","last_peered":"2024-03-18T01:36:57.192214+0000","last_clean":"2024-03-18T01:36:57.192214+0000","last_became_active":"2024-03-18T01:34:56.648257+0000","last_became_peered":"2024-03-18T01:34:56.648257+0000","last_unstale":"2024-03-18T01:36:57.192214+0000","last_undegraded":"2024-03-18T01:36:57.192214+0000","last_fullsized":"2024-03-18T01:36:57.192214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:53.788445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807611+0000","last_change":"2024-03-18T01:35:01.136912+0000","last_active":"2024-03-18T01:35:58.807611+0000","last_peered":"2024-03-18T01:35:58.807611+0000","last_clean":"2024-03-18T01:35:58.807611+0000","last_became_active":"2024-03-18T01:35:01.136611+0000","last_became_peered":"2024-03-18T01:35:01.136611+0000","last_unstale":"2024-03-18T01:35:58.807611+0000","last_undegraded":"2024-03-18T01:35:58.807611+0000","last_fullsized":"2024-03-18T01:35:58.807611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:31.089605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802559+0000","last_change":"2024-03-18T01:34:55.648345+0000","last_active":"2024-03-18T01:35:58.802559+0000","last_peered":"2024-03-18T01:35:58.802559+0000","last_clean":"2024-03-18T01:35:58.802559+0000","last_became_active":"2024-03-18T01:34:55.647113+0000","last_became_peered":"2024-03-18T01:34:55.647113+0000","last_unstale":"2024-03-18T01:35:58.802559+0000","last_undegraded":"2024-03-18T01:35:58.802559+0000","last_fullsized":"2024-03-18T01:35:58.802559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:20.904129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190181+0000","last_change":"2024-03-18T01:34:56.649147+0000","last_active":"2024-03-18T01:36:57.190181+0000","last_peered":"2024-03-18T01:36:57.190181+0000","last_clean":"2024-03-18T01:36:57.190181+0000","last_became_active":"2024-03-18T01:34:56.648815+0000","last_became_peered":"2024-03-18T01:34:56.648815+0000","last_unstale":"2024-03-18T01:36:57.190181+0000","last_undegraded":"2024-03-18T01:36:57.190181+0000","last_fullsized":"2024-03-18T01:36:57.190181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:53.560679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802064+0000","last_change":"2024-03-18T01:35:01.156778+0000","last_active":"2024-03-18T01:35:58.802064+0000","last_peered":"2024-03-18T01:35:58.802064+0000","last_clean":"2024-03-18T01:35:58.802064+0000","last_became_active":"2024-03-18T01:35:01.156367+0000","last_became_peered":"2024-03-18T01:35:01.156367+0000","last_unstale":"2024-03-18T01:35:58.802064+0000","last_undegraded":"2024-03-18T01:35:58.802064+0000","last_fullsized":"2024-03-18T01:35:58.802064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:46.223915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810972+0000","last_change":"2024-03-18T01:34:55.658722+0000","last_active":"2024-03-18T01:35:58.810972+0000","last_peered":"2024-03-18T01:35:58.810972+0000","last_clean":"2024-03-18T01:35:58.810972+0000","last_became_active":"2024-03-18T01:34:55.658321+0000","last_became_peered":"2024-03-18T01:34:55.658321+0000","last_unstale":"2024-03-18T01:35:58.810972+0000","last_undegraded":"2024-03-18T01:35:58.810972+0000","last_fullsized":"2024-03-18T01:35:58.810972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:05.753226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.221076+0000","last_change":"2024-03-18T01:34:56.649415+0000","last_active":"2024-03-18T01:36:57.221076+0000","last_peered":"2024-03-18T01:36:57.221076+0000","last_clean":"2024-03-18T01:36:57.221076+0000","last_became_active":"2024-03-18T01:34:56.649019+0000","last_became_peered":"2024-03-18T01:34:56.649019+0000","last_unstale":"2024-03-18T01:36:57.221076+0000","last_undegraded":"2024-03-18T01:36:57.221076+0000","last_fullsized":"2024-03-18T01:36:57.221076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:28.473683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819934+0000","last_change":"2024-03-18T01:35:01.148937+0000","last_active":"2024-03-18T01:35:58.819934+0000","last_peered":"2024-03-18T01:35:58.819934+0000","last_clean":"2024-03-18T01:35:58.819934+0000","last_became_active":"2024-03-18T01:35:01.148415+0000","last_became_peered":"2024-03-18T01:35:01.148415+0000","last_unstale":"2024-03-18T01:35:58.819934+0000","last_undegraded":"2024-03-18T01:35:58.819934+0000","last_fullsized":"2024-03-18T01:35:58.819934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:45.926970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":185,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:59.870480+0000","last_change":"2024-03-18T01:34:55.653443+0000","last_active":"2024-03-18T01:37:59.870480+0000","last_peered":"2024-03-18T01:37:59.870480+0000","last_clean":"2024-03-18T01:37:59.870480+0000","last_became_active":"2024-03-18T01:34:55.653072+0000","last_became_peered":"2024-03-18T01:34:55.653072+0000","last_unstale":"2024-03-18T01:37:59.870480+0000","last_undegraded":"2024-03-18T01:37:59.870480+0000","last_fullsized":"2024-03-18T01:37:59.870480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:26.376636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202535+0000","last_change":"2024-03-18T01:35:58.811178+0000","last_active":"2024-03-18T01:36:57.202535+0000","last_peered":"2024-03-18T01:36:57.202535+0000","last_clean":"2024-03-18T01:36:57.202535+0000","last_became_active":"2024-03-18T01:35:58.810764+0000","last_became_peered":"2024-03-18T01:35:58.810764+0000","last_unstale":"2024-03-18T01:36:57.202535+0000","last_undegraded":"2024-03-18T01:36:57.202535+0000","last_fullsized":"2024-03-18T01:36:57.202535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:49.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819478+0000","last_change":"2024-03-18T01:35:01.152767+0000","last_active":"2024-03-18T01:35:58.819478+0000","last_peered":"2024-03-18T01:35:58.819478+0000","last_clean":"2024-03-18T01:35:58.819478+0000","last_became_active":"2024-03-18T01:35:01.150619+0000","last_became_peered":"2024-03-18T01:35:01.150619+0000","last_unstale":"2024-03-18T01:35:58.819478+0000","last_undegraded":"2024-03-18T01:35:58.819478+0000","last_fullsized":"2024-03-18T01:35:58.819478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:57:28.258251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191740+0000","last_change":"2024-03-18T01:34:56.650541+0000","last_active":"2024-03-18T01:36:57.191740+0000","last_peered":"2024-03-18T01:36:57.191740+0000","last_clean":"2024-03-18T01:36:57.191740+0000","last_became_active":"2024-03-18T01:34:56.650072+0000","last_became_peered":"2024-03-18T01:34:56.650072+0000","last_unstale":"2024-03-18T01:36:57.191740+0000","last_undegraded":"2024-03-18T01:36:57.191740+0000","last_fullsized":"2024-03-18T01:36:57.191740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:51.224883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":189,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:02.950644+0000","last_change":"2024-03-18T01:34:55.664297+0000","last_active":"2024-03-18T01:38:02.950644+0000","last_peered":"2024-03-18T01:38:02.950644+0000","last_clean":"2024-03-18T01:38:02.950644+0000","last_became_active":"2024-03-18T01:34:55.663938+0000","last_became_peered":"2024-03-18T01:34:55.663938+0000","last_unstale":"2024-03-18T01:38:02.950644+0000","last_undegraded":"2024-03-18T01:38:02.950644+0000","last_fullsized":"2024-03-18T01:38:02.950644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:51.771987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":155,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806717+0000","last_change":"2024-03-18T01:35:01.141736+0000","last_active":"2024-03-18T01:35:58.806717+0000","last_peered":"2024-03-18T01:35:58.806717+0000","last_clean":"2024-03-18T01:35:58.806717+0000","last_became_active":"2024-03-18T01:35:01.141381+0000","last_became_peered":"2024-03-18T01:35:01.141381+0000","last_unstale":"2024-03-18T01:35:58.806717+0000","last_undegraded":"2024-03-18T01:35:58.806717+0000","last_fullsized":"2024-03-18T01:35:58.806717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:25:04.190250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216313+0000","last_change":"2024-03-18T01:34:56.666439+0000","last_active":"2024-03-18T01:36:57.216313+0000","last_peered":"2024-03-18T01:36:57.216313+0000","last_clean":"2024-03-18T01:36:57.216313+0000","last_became_active":"2024-03-18T01:34:56.664556+0000","last_became_peered":"2024-03-18T01:34:56.664556+0000","last_unstale":"2024-03-18T01:36:57.216313+0000","last_undegraded":"2024-03-18T01:36:57.216313+0000","last_fullsized":"2024-03-18T01:36:57.216313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:08:48.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802167+0000","last_change":"2024-03-18T01:34:55.656484+0000","last_active":"2024-03-18T01:35:58.802167+0000","last_peered":"2024-03-18T01:35:58.802167+0000","last_clean":"2024-03-18T01:35:58.802167+0000","last_became_active":"2024-03-18T01:34:55.656125+0000","last_became_peered":"2024-03-18T01:34:55.656125+0000","last_unstale":"2024-03-18T01:35:58.802167+0000","last_undegraded":"2024-03-18T01:35:58.802167+0000","last_fullsized":"2024-03-18T01:35:58.802167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:58:25.944403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806027+0000","last_change":"2024-03-18T01:33:35.824651+0000","last_active":"2024-03-18T01:35:58.806027+0000","last_peered":"2024-03-18T01:35:58.806027+0000","last_clean":"2024-03-18T01:35:58.806027+0000","last_became_active":"2024-03-18T01:33:35.824106+0000","last_became_peered":"2024-03-18T01:33:35.824106+0000","last_unstale":"2024-03-18T01:35:58.806027+0000","last_undegraded":"2024-03-18T01:35:58.806027+0000","last_fullsized":"2024-03-18T01:35:58.806027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_clean_scrub_stamp":"2024-03-18T01:29:41.724134+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:06.422316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808524+0000","last_change":"2024-03-18T01:35:01.157735+0000","last_active":"2024-03-18T01:35:58.808524+0000","last_peered":"2024-03-18T01:35:58.808524+0000","last_clean":"2024-03-18T01:35:58.808524+0000","last_became_active":"2024-03-18T01:35:01.156954+0000","last_became_peered":"2024-03-18T01:35:01.156954+0000","last_unstale":"2024-03-18T01:35:58.808524+0000","last_undegraded":"2024-03-18T01:35:58.808524+0000","last_fullsized":"2024-03-18T01:35:58.808524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:00.106809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193743+0000","last_change":"2024-03-18T01:34:56.646808+0000","last_active":"2024-03-18T01:36:57.193743+0000","last_peered":"2024-03-18T01:36:57.193743+0000","last_clean":"2024-03-18T01:36:57.193743+0000","last_became_active":"2024-03-18T01:34:56.646437+0000","last_became_peered":"2024-03-18T01:34:56.646437+0000","last_unstale":"2024-03-18T01:36:57.193743+0000","last_undegraded":"2024-03-18T01:36:57.193743+0000","last_fullsized":"2024-03-18T01:36:57.193743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:46:09.703889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806136+0000","last_change":"2024-03-18T01:34:55.645365+0000","last_active":"2024-03-18T01:35:58.806136+0000","last_peered":"2024-03-18T01:35:58.806136+0000","last_clean":"2024-03-18T01:35:58.806136+0000","last_became_active":"2024-03-18T01:34:55.645018+0000","last_became_peered":"2024-03-18T01:34:55.645018+0000","last_unstale":"2024-03-18T01:35:58.806136+0000","last_undegraded":"2024-03-18T01:35:58.806136+0000","last_fullsized":"2024-03-18T01:35:58.806136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:08:31.869328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804962+0000","last_change":"2024-03-18T01:35:01.147294+0000","last_active":"2024-03-18T01:35:58.804962+0000","last_peered":"2024-03-18T01:35:58.804962+0000","last_clean":"2024-03-18T01:35:58.804962+0000","last_became_active":"2024-03-18T01:35:01.146858+0000","last_became_peered":"2024-03-18T01:35:01.146858+0000","last_unstale":"2024-03-18T01:35:58.804962+0000","last_undegraded":"2024-03-18T01:35:58.804962+0000","last_fullsized":"2024-03-18T01:35:58.804962+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:45:24.010746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802584+0000","last_change":"2024-03-18T01:34:55.658110+0000","last_active":"2024-03-18T01:35:58.802584+0000","last_peered":"2024-03-18T01:35:58.802584+0000","last_clean":"2024-03-18T01:35:58.802584+0000","last_became_active":"2024-03-18T01:34:55.657802+0000","last_became_peered":"2024-03-18T01:34:55.657802+0000","last_unstale":"2024-03-18T01:35:58.802584+0000","last_undegraded":"2024-03-18T01:35:58.802584+0000","last_fullsized":"2024-03-18T01:35:58.802584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:34.264115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205099+0000","last_change":"2024-03-18T01:34:56.656689+0000","last_active":"2024-03-18T01:36:57.205099+0000","last_peered":"2024-03-18T01:36:57.205099+0000","last_clean":"2024-03-18T01:36:57.205099+0000","last_became_active":"2024-03-18T01:34:56.655653+0000","last_became_peered":"2024-03-18T01:34:56.655653+0000","last_unstale":"2024-03-18T01:36:57.205099+0000","last_undegraded":"2024-03-18T01:36:57.205099+0000","last_fullsized":"2024-03-18T01:36:57.205099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:58.370904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816539+0000","last_change":"2024-03-18T01:35:01.137419+0000","last_active":"2024-03-18T01:35:58.816539+0000","last_peered":"2024-03-18T01:35:58.816539+0000","last_clean":"2024-03-18T01:35:58.816539+0000","last_became_active":"2024-03-18T01:35:01.137033+0000","last_became_peered":"2024-03-18T01:35:01.137033+0000","last_unstale":"2024-03-18T01:35:58.816539+0000","last_undegraded":"2024-03-18T01:35:58.816539+0000","last_fullsized":"2024-03-18T01:35:58.816539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:13:48.382329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:05.854415+0000","last_change":"2024-03-18T01:34:55.663134+0000","last_active":"2024-03-18T01:37:05.854415+0000","last_peered":"2024-03-18T01:37:05.854415+0000","last_clean":"2024-03-18T01:37:05.854415+0000","last_became_active":"2024-03-18T01:34:55.662083+0000","last_became_peered":"2024-03-18T01:34:55.662083+0000","last_unstale":"2024-03-18T01:37:05.854415+0000","last_undegraded":"2024-03-18T01:37:05.854415+0000","last_fullsized":"2024-03-18T01:37:05.854415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:01:09.186599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216678+0000","last_change":"2024-03-18T01:34:56.664761+0000","last_active":"2024-03-18T01:36:57.216678+0000","last_peered":"2024-03-18T01:36:57.216678+0000","last_clean":"2024-03-18T01:36:57.216678+0000","last_became_active":"2024-03-18T01:34:56.663657+0000","last_became_peered":"2024-03-18T01:34:56.663657+0000","last_unstale":"2024-03-18T01:36:57.216678+0000","last_undegraded":"2024-03-18T01:36:57.216678+0000","last_fullsized":"2024-03-18T01:36:57.216678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:26:53.075000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809995+0000","last_change":"2024-03-18T01:35:01.133002+0000","last_active":"2024-03-18T01:35:58.809995+0000","last_peered":"2024-03-18T01:35:58.809995+0000","last_clean":"2024-03-18T01:35:58.809995+0000","last_became_active":"2024-03-18T01:35:01.132542+0000","last_became_peered":"2024-03-18T01:35:01.132542+0000","last_unstale":"2024-03-18T01:35:58.809995+0000","last_undegraded":"2024-03-18T01:35:58.809995+0000","last_fullsized":"2024-03-18T01:35:58.809995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:23.823409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801972+0000","last_change":"2024-03-18T01:34:55.648924+0000","last_active":"2024-03-18T01:35:58.801972+0000","last_peered":"2024-03-18T01:35:58.801972+0000","last_clean":"2024-03-18T01:35:58.801972+0000","last_became_active":"2024-03-18T01:34:55.648548+0000","last_became_peered":"2024-03-18T01:34:55.648548+0000","last_unstale":"2024-03-18T01:35:58.801972+0000","last_undegraded":"2024-03-18T01:35:58.801972+0000","last_fullsized":"2024-03-18T01:35:58.801972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:26:38.916856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.221812+0000","last_change":"2024-03-18T01:34:56.663142+0000","last_active":"2024-03-18T01:36:57.221812+0000","last_peered":"2024-03-18T01:36:57.221812+0000","last_clean":"2024-03-18T01:36:57.221812+0000","last_became_active":"2024-03-18T01:34:56.660256+0000","last_became_peered":"2024-03-18T01:34:56.660256+0000","last_unstale":"2024-03-18T01:36:57.221812+0000","last_undegraded":"2024-03-18T01:36:57.221812+0000","last_fullsized":"2024-03-18T01:36:57.221812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:59.503277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801730+0000","last_change":"2024-03-18T01:35:01.136186+0000","last_active":"2024-03-18T01:35:58.801730+0000","last_peered":"2024-03-18T01:35:58.801730+0000","last_clean":"2024-03-18T01:35:58.801730+0000","last_became_active":"2024-03-18T01:35:01.135803+0000","last_became_peered":"2024-03-18T01:35:01.135803+0000","last_unstale":"2024-03-18T01:35:58.801730+0000","last_undegraded":"2024-03-18T01:35:58.801730+0000","last_fullsized":"2024-03-18T01:35:58.801730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:20.027448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806474+0000","last_change":"2024-03-18T01:34:55.663090+0000","last_active":"2024-03-18T01:35:58.806474+0000","last_peered":"2024-03-18T01:35:58.806474+0000","last_clean":"2024-03-18T01:35:58.806474+0000","last_became_active":"2024-03-18T01:34:55.662774+0000","last_became_peered":"2024-03-18T01:34:55.662774+0000","last_unstale":"2024-03-18T01:35:58.806474+0000","last_undegraded":"2024-03-18T01:35:58.806474+0000","last_fullsized":"2024-03-18T01:35:58.806474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:30:35.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.178629+0000","last_change":"2024-03-18T01:34:56.639509+0000","last_active":"2024-03-18T01:36:57.178629+0000","last_peered":"2024-03-18T01:36:57.178629+0000","last_clean":"2024-03-18T01:36:57.178629+0000","last_became_active":"2024-03-18T01:34:56.639144+0000","last_became_peered":"2024-03-18T01:34:56.639144+0000","last_unstale":"2024-03-18T01:36:57.178629+0000","last_undegraded":"2024-03-18T01:36:57.178629+0000","last_fullsized":"2024-03-18T01:36:57.178629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:12:36.220793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812801+0000","last_change":"2024-03-18T01:35:01.141762+0000","last_active":"2024-03-18T01:35:58.812801+0000","last_peered":"2024-03-18T01:35:58.812801+0000","last_clean":"2024-03-18T01:35:58.812801+0000","last_became_active":"2024-03-18T01:35:01.141219+0000","last_became_peered":"2024-03-18T01:35:01.141219+0000","last_unstale":"2024-03-18T01:35:58.812801+0000","last_undegraded":"2024-03-18T01:35:58.812801+0000","last_fullsized":"2024-03-18T01:35:58.812801+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:55.839120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.815926+0000","last_change":"2024-03-18T01:34:55.662530+0000","last_active":"2024-03-18T01:35:58.815926+0000","last_peered":"2024-03-18T01:35:58.815926+0000","last_clean":"2024-03-18T01:35:58.815926+0000","last_became_active":"2024-03-18T01:34:55.661178+0000","last_became_peered":"2024-03-18T01:34:55.661178+0000","last_unstale":"2024-03-18T01:35:58.815926+0000","last_undegraded":"2024-03-18T01:35:58.815926+0000","last_fullsized":"2024-03-18T01:35:58.815926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:47:16.678553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191228+0000","last_change":"2024-03-18T01:34:56.660936+0000","last_active":"2024-03-18T01:36:57.191228+0000","last_peered":"2024-03-18T01:36:57.191228+0000","last_clean":"2024-03-18T01:36:57.191228+0000","last_became_active":"2024-03-18T01:34:56.660513+0000","last_became_peered":"2024-03-18T01:34:56.660513+0000","last_unstale":"2024-03-18T01:36:57.191228+0000","last_undegraded":"2024-03-18T01:36:57.191228+0000","last_fullsized":"2024-03-18T01:36:57.191228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:35:21.037366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802056+0000","last_change":"2024-03-18T01:35:01.136914+0000","last_active":"2024-03-18T01:35:58.802056+0000","last_peered":"2024-03-18T01:35:58.802056+0000","last_clean":"2024-03-18T01:35:58.802056+0000","last_became_active":"2024-03-18T01:35:01.136480+0000","last_became_peered":"2024-03-18T01:35:01.136480+0000","last_unstale":"2024-03-18T01:35:58.802056+0000","last_undegraded":"2024-03-18T01:35:58.802056+0000","last_fullsized":"2024-03-18T01:35:58.802056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:49.262255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:05.892340+0000","last_change":"2024-03-18T01:34:55.654938+0000","last_active":"2024-03-18T01:37:05.892340+0000","last_peered":"2024-03-18T01:37:05.892340+0000","last_clean":"2024-03-18T01:37:05.892340+0000","last_became_active":"2024-03-18T01:34:55.654611+0000","last_became_peered":"2024-03-18T01:34:55.654611+0000","last_unstale":"2024-03-18T01:37:05.892340+0000","last_undegraded":"2024-03-18T01:37:05.892340+0000","last_fullsized":"2024-03-18T01:37:05.892340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:23.210205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220083+0000","last_change":"2024-03-18T01:34:56.650731+0000","last_active":"2024-03-18T01:36:57.220083+0000","last_peered":"2024-03-18T01:36:57.220083+0000","last_clean":"2024-03-18T01:36:57.220083+0000","last_became_active":"2024-03-18T01:34:56.650363+0000","last_became_peered":"2024-03-18T01:34:56.650363+0000","last_unstale":"2024-03-18T01:36:57.220083+0000","last_undegraded":"2024-03-18T01:36:57.220083+0000","last_fullsized":"2024-03-18T01:36:57.220083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:09:50.305019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801699+0000","last_change":"2024-03-18T01:35:01.135101+0000","last_active":"2024-03-18T01:35:58.801699+0000","last_peered":"2024-03-18T01:35:58.801699+0000","last_clean":"2024-03-18T01:35:58.801699+0000","last_became_active":"2024-03-18T01:35:01.134690+0000","last_became_peered":"2024-03-18T01:35:01.134690+0000","last_unstale":"2024-03-18T01:35:58.801699+0000","last_undegraded":"2024-03-18T01:35:58.801699+0000","last_fullsized":"2024-03-18T01:35:58.801699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:33.381629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802175+0000","last_change":"2024-03-18T01:34:55.648108+0000","last_active":"2024-03-18T01:35:58.802175+0000","last_peered":"2024-03-18T01:35:58.802175+0000","last_clean":"2024-03-18T01:35:58.802175+0000","last_became_active":"2024-03-18T01:34:55.647754+0000","last_became_peered":"2024-03-18T01:34:55.647754+0000","last_unstale":"2024-03-18T01:35:58.802175+0000","last_undegraded":"2024-03-18T01:35:58.802175+0000","last_fullsized":"2024-03-18T01:35:58.802175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:05:48.508015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203569+0000","last_change":"2024-03-18T01:34:56.651589+0000","last_active":"2024-03-18T01:36:57.203569+0000","last_peered":"2024-03-18T01:36:57.203569+0000","last_clean":"2024-03-18T01:36:57.203569+0000","last_became_active":"2024-03-18T01:34:56.650929+0000","last_became_peered":"2024-03-18T01:34:56.650929+0000","last_unstale":"2024-03-18T01:36:57.203569+0000","last_undegraded":"2024-03-18T01:36:57.203569+0000","last_fullsized":"2024-03-18T01:36:57.203569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:43.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806823+0000","last_change":"2024-03-18T01:35:01.148347+0000","last_active":"2024-03-18T01:35:58.806823+0000","last_peered":"2024-03-18T01:35:58.806823+0000","last_clean":"2024-03-18T01:35:58.806823+0000","last_became_active":"2024-03-18T01:35:01.147936+0000","last_became_peered":"2024-03-18T01:35:01.147936+0000","last_unstale":"2024-03-18T01:35:58.806823+0000","last_undegraded":"2024-03-18T01:35:58.806823+0000","last_fullsized":"2024-03-18T01:35:58.806823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:12.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":188,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:59.800787+0000","last_change":"2024-03-18T01:34:55.653625+0000","last_active":"2024-03-18T01:37:59.800787+0000","last_peered":"2024-03-18T01:37:59.800787+0000","last_clean":"2024-03-18T01:37:59.800787+0000","last_became_active":"2024-03-18T01:34:55.653186+0000","last_became_peered":"2024-03-18T01:34:55.653186+0000","last_unstale":"2024-03-18T01:37:59.800787+0000","last_undegraded":"2024-03-18T01:37:59.800787+0000","last_fullsized":"2024-03-18T01:37:59.800787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:10:51.193296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1480,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":154,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203868+0000","last_change":"2024-03-18T01:34:56.652703+0000","last_active":"2024-03-18T01:36:57.203868+0000","last_peered":"2024-03-18T01:36:57.203868+0000","last_clean":"2024-03-18T01:36:57.203868+0000","last_became_active":"2024-03-18T01:34:56.651768+0000","last_became_peered":"2024-03-18T01:34:56.651768+0000","last_unstale":"2024-03-18T01:36:57.203868+0000","last_undegraded":"2024-03-18T01:36:57.203868+0000","last_fullsized":"2024-03-18T01:36:57.203868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:59.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809369+0000","last_change":"2024-03-18T01:35:01.144900+0000","last_active":"2024-03-18T01:35:58.809369+0000","last_peered":"2024-03-18T01:35:58.809369+0000","last_clean":"2024-03-18T01:35:58.809369+0000","last_became_active":"2024-03-18T01:35:01.144431+0000","last_became_peered":"2024-03-18T01:35:01.144431+0000","last_unstale":"2024-03-18T01:35:58.809369+0000","last_undegraded":"2024-03-18T01:35:58.809369+0000","last_fullsized":"2024-03-18T01:35:58.809369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:36.199904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808782+0000","last_change":"2024-03-18T01:34:55.661259+0000","last_active":"2024-03-18T01:35:58.808782+0000","last_peered":"2024-03-18T01:35:58.808782+0000","last_clean":"2024-03-18T01:35:58.808782+0000","last_became_active":"2024-03-18T01:34:55.660964+0000","last_became_peered":"2024-03-18T01:34:55.660964+0000","last_unstale":"2024-03-18T01:35:58.808782+0000","last_undegraded":"2024-03-18T01:35:58.808782+0000","last_fullsized":"2024-03-18T01:35:58.808782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:14.723905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213011+0000","last_change":"2024-03-18T01:34:56.663075+0000","last_active":"2024-03-18T01:36:57.213011+0000","last_peered":"2024-03-18T01:36:57.213011+0000","last_clean":"2024-03-18T01:36:57.213011+0000","last_became_active":"2024-03-18T01:34:56.662458+0000","last_became_peered":"2024-03-18T01:34:56.662458+0000","last_unstale":"2024-03-18T01:36:57.213011+0000","last_undegraded":"2024-03-18T01:36:57.213011+0000","last_fullsized":"2024-03-18T01:36:57.213011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:31:31.837493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806874+0000","last_change":"2024-03-18T01:35:01.133317+0000","last_active":"2024-03-18T01:35:58.806874+0000","last_peered":"2024-03-18T01:35:58.806874+0000","last_clean":"2024-03-18T01:35:58.806874+0000","last_became_active":"2024-03-18T01:35:01.131798+0000","last_became_peered":"2024-03-18T01:35:01.131798+0000","last_unstale":"2024-03-18T01:35:58.806874+0000","last_undegraded":"2024-03-18T01:35:58.806874+0000","last_fullsized":"2024-03-18T01:35:58.806874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:55.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819510+0000","last_change":"2024-03-18T01:34:55.660887+0000","last_active":"2024-03-18T01:35:58.819510+0000","last_peered":"2024-03-18T01:35:58.819510+0000","last_clean":"2024-03-18T01:35:58.819510+0000","last_became_active":"2024-03-18T01:34:55.660524+0000","last_became_peered":"2024-03-18T01:34:55.660524+0000","last_unstale":"2024-03-18T01:35:58.819510+0000","last_undegraded":"2024-03-18T01:35:58.819510+0000","last_fullsized":"2024-03-18T01:35:58.819510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:24.816951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192183+0000","last_change":"2024-03-18T01:34:56.654107+0000","last_active":"2024-03-18T01:36:57.192183+0000","last_peered":"2024-03-18T01:36:57.192183+0000","last_clean":"2024-03-18T01:36:57.192183+0000","last_became_active":"2024-03-18T01:34:56.653742+0000","last_became_peered":"2024-03-18T01:34:56.653742+0000","last_unstale":"2024-03-18T01:36:57.192183+0000","last_undegraded":"2024-03-18T01:36:57.192183+0000","last_fullsized":"2024-03-18T01:36:57.192183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:58:46.093589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191185+0000","last_change":"2024-03-18T01:34:56.647818+0000","last_active":"2024-03-18T01:36:57.191185+0000","last_peered":"2024-03-18T01:36:57.191185+0000","last_clean":"2024-03-18T01:36:57.191185+0000","last_became_active":"2024-03-18T01:34:56.647174+0000","last_became_peered":"2024-03-18T01:34:56.647174+0000","last_unstale":"2024-03-18T01:36:57.191185+0000","last_undegraded":"2024-03-18T01:36:57.191185+0000","last_fullsized":"2024-03-18T01:36:57.191185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:55.656083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191353+0000","last_change":"2024-03-18T01:34:56.643221+0000","last_active":"2024-03-18T01:36:57.191353+0000","last_peered":"2024-03-18T01:36:57.191353+0000","last_clean":"2024-03-18T01:36:57.191353+0000","last_became_active":"2024-03-18T01:34:56.642871+0000","last_became_peered":"2024-03-18T01:34:56.642871+0000","last_unstale":"2024-03-18T01:36:57.191353+0000","last_undegraded":"2024-03-18T01:36:57.191353+0000","last_fullsized":"2024-03-18T01:36:57.191353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:23:25.106060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213467+0000","last_change":"2024-03-18T01:34:56.656227+0000","last_active":"2024-03-18T01:36:57.213467+0000","last_peered":"2024-03-18T01:36:57.213467+0000","last_clean":"2024-03-18T01:36:57.213467+0000","last_became_active":"2024-03-18T01:34:56.655859+0000","last_became_peered":"2024-03-18T01:34:56.655859+0000","last_unstale":"2024-03-18T01:36:57.213467+0000","last_undegraded":"2024-03-18T01:36:57.213467+0000","last_fullsized":"2024-03-18T01:36:57.213467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:51:54.424342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.215195+0000","last_change":"2024-03-18T01:34:56.664655+0000","last_active":"2024-03-18T01:36:57.215195+0000","last_peered":"2024-03-18T01:36:57.215195+0000","last_clean":"2024-03-18T01:36:57.215195+0000","last_became_active":"2024-03-18T01:34:56.662526+0000","last_became_peered":"2024-03-18T01:34:56.662526+0000","last_unstale":"2024-03-18T01:36:57.215195+0000","last_undegraded":"2024-03-18T01:36:57.215195+0000","last_fullsized":"2024-03-18T01:36:57.215195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:45:06.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214796+0000","last_change":"2024-03-18T01:34:56.657045+0000","last_active":"2024-03-18T01:36:57.214796+0000","last_peered":"2024-03-18T01:36:57.214796+0000","last_clean":"2024-03-18T01:36:57.214796+0000","last_became_active":"2024-03-18T01:34:56.656339+0000","last_became_peered":"2024-03-18T01:34:56.656339+0000","last_unstale":"2024-03-18T01:36:57.214796+0000","last_undegraded":"2024-03-18T01:36:57.214796+0000","last_fullsized":"2024-03-18T01:36:57.214796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:41.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.184826+0000","last_change":"2024-03-18T01:34:56.647626+0000","last_active":"2024-03-18T01:36:57.184826+0000","last_peered":"2024-03-18T01:36:57.184826+0000","last_clean":"2024-03-18T01:36:57.184826+0000","last_became_active":"2024-03-18T01:34:56.647216+0000","last_became_peered":"2024-03-18T01:34:56.647216+0000","last_unstale":"2024-03-18T01:36:57.184826+0000","last_undegraded":"2024-03-18T01:36:57.184826+0000","last_fullsized":"2024-03-18T01:36:57.184826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:12:33.330925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.180192+0000","last_change":"2024-03-18T01:34:56.645091+0000","last_active":"2024-03-18T01:36:57.180192+0000","last_peered":"2024-03-18T01:36:57.180192+0000","last_clean":"2024-03-18T01:36:57.180192+0000","last_became_active":"2024-03-18T01:34:56.644725+0000","last_became_peered":"2024-03-18T01:34:56.644725+0000","last_unstale":"2024-03-18T01:36:57.180192+0000","last_undegraded":"2024-03-18T01:36:57.180192+0000","last_fullsized":"2024-03-18T01:36:57.180192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211414+0000","last_change":"2024-03-18T01:34:56.651282+0000","last_active":"2024-03-18T01:36:57.211414+0000","last_peered":"2024-03-18T01:36:57.211414+0000","last_clean":"2024-03-18T01:36:57.211414+0000","last_became_active":"2024-03-18T01:34:56.650872+0000","last_became_peered":"2024-03-18T01:34:56.650872+0000","last_unstale":"2024-03-18T01:36:57.211414+0000","last_undegraded":"2024-03-18T01:36:57.211414+0000","last_fullsized":"2024-03-18T01:36:57.211414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:14:45.106848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211139+0000","last_change":"2024-03-18T01:34:56.651347+0000","last_active":"2024-03-18T01:36:57.211139+0000","last_peered":"2024-03-18T01:36:57.211139+0000","last_clean":"2024-03-18T01:36:57.211139+0000","last_became_active":"2024-03-18T01:34:56.650935+0000","last_became_peered":"2024-03-18T01:34:56.650935+0000","last_unstale":"2024-03-18T01:36:57.211139+0000","last_undegraded":"2024-03-18T01:36:57.211139+0000","last_fullsized":"2024-03-18T01:36:57.211139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:41.849883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211642+0000","last_change":"2024-03-18T01:34:56.667333+0000","last_active":"2024-03-18T01:36:57.211642+0000","last_peered":"2024-03-18T01:36:57.211642+0000","last_clean":"2024-03-18T01:36:57.211642+0000","last_became_active":"2024-03-18T01:34:56.666845+0000","last_became_peered":"2024-03-18T01:34:56.666845+0000","last_unstale":"2024-03-18T01:36:57.211642+0000","last_undegraded":"2024-03-18T01:36:57.211642+0000","last_fullsized":"2024-03-18T01:36:57.211642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:49:23.813455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.217436+0000","last_change":"2024-03-18T01:34:56.662677+0000","last_active":"2024-03-18T01:36:57.217436+0000","last_peered":"2024-03-18T01:36:57.217436+0000","last_clean":"2024-03-18T01:36:57.217436+0000","last_became_active":"2024-03-18T01:34:56.662306+0000","last_became_peered":"2024-03-18T01:34:56.662306+0000","last_unstale":"2024-03-18T01:36:57.217436+0000","last_undegraded":"2024-03-18T01:36:57.217436+0000","last_fullsized":"2024-03-18T01:36:57.217436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:53:56.868984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216916+0000","last_change":"2024-03-18T01:34:56.657721+0000","last_active":"2024-03-18T01:36:57.216916+0000","last_peered":"2024-03-18T01:36:57.216916+0000","last_clean":"2024-03-18T01:36:57.216916+0000","last_became_active":"2024-03-18T01:34:56.657072+0000","last_became_peered":"2024-03-18T01:34:56.657072+0000","last_unstale":"2024-03-18T01:36:57.216916+0000","last_undegraded":"2024-03-18T01:36:57.216916+0000","last_fullsized":"2024-03-18T01:36:57.216916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:21.120391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190992+0000","last_change":"2024-03-18T01:34:56.647327+0000","last_active":"2024-03-18T01:36:57.190992+0000","last_peered":"2024-03-18T01:36:57.190992+0000","last_clean":"2024-03-18T01:36:57.190992+0000","last_became_active":"2024-03-18T01:34:56.646910+0000","last_became_peered":"2024-03-18T01:34:56.646910+0000","last_unstale":"2024-03-18T01:36:57.190992+0000","last_undegraded":"2024-03-18T01:36:57.190992+0000","last_fullsized":"2024-03-18T01:36:57.190992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:37:25.970272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74209,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1562,"num_read_kb":861,"num_write":257,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":222627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383732,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":502393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645368,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1155726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808893,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070533,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":501159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332170,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1098379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495696,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":564842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":560990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182496,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920859,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":501473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757335,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1093429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:38:07.039 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-18T01:38:07.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T01:38:07.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:06 smithi203 ceph-mon[36345]: pgmap v415: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:07.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:06 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1997917221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:07.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 -- 172.21.15.7:0/741789389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104103a00 msgr2=0x7f5104103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 --2- 172.21.15.7:0/741789389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104103a00 0x7f5104103e20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009900 tx=0x7f50f402bc50 comp rx=0 tx=0).stop 2024-03-18T01:38:07.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 -- 172.21.15.7:0/741789389 shutdown_connections 2024-03-18T01:38:07.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 --2- 172.21.15.7:0/741789389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51041055b0 0x7f510410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 --2- 172.21.15.7:0/741789389 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5104104bf0 0x7f5104105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 --2- 172.21.15.7:0/741789389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104103a00 0x7f5104103e20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 -- 172.21.15.7:0/741789389 >> 172.21.15.7:0/741789389 conn(0x7f51040fef90 msgr2=0x7f51041013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:07.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 -- 172.21.15.7:0/741789389 shutdown_connections 2024-03-18T01:38:07.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.321+0000 7f5109a00640 1 -- 172.21.15.7:0/741789389 wait complete. 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.322+0000 7f5109a00640 1 Processor -- start 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5109a00640 1 -- start start 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5109a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5104103a00 0x7f5104199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5109a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104104bf0 0x7f510419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5109a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51041055b0 0x7f51041a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5109a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f510410f650 con 0x7f51041055b0 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5109a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f510410f4d0 con 0x7f5104104bf0 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5109a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f510410f7d0 con 0x7f5104103a00 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5102a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5104103a00 0x7f5104199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:07.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5102a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5104103a00 0x7f5104199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44586/0 (socket says 172.21.15.7:44586) 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5102000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104104bf0 0x7f510419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5102a00640 1 -- 172.21.15.7:0/1642806084 learned_addr learned my addr 172.21.15.7:0/1642806084 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.323+0000 7f5103400640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51041055b0 0x7f51041a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f5102a00640 1 -- 172.21.15.7:0/1642806084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104104bf0 msgr2=0x7f510419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f5102a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104104bf0 0x7f510419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f5102a00640 1 -- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51041055b0 msgr2=0x7f51041a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f5102a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51041055b0 0x7f51041a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f5102a00640 1 -- 172.21.15.7:0/1642806084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51041a19d0 con 0x7f5104103a00 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f5102000640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104104bf0 0x7f510419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f5102a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5104103a00 0x7f5104199db0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50f4000c00 tx=0x7f50f402c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f50fb400640 1 -- 172.21.15.7:0/1642806084 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4042020 con 0x7f5104103a00 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f50fb400640 1 -- 172.21.15.7:0/1642806084 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f4004450 con 0x7f5104103a00 2024-03-18T01:38:07.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51041a1c60 con 0x7f5104103a00 2024-03-18T01:38:07.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.324+0000 7f50fb400640 1 -- 172.21.15.7:0/1642806084 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f4003df0 con 0x7f5104103a00 2024-03-18T01:38:07.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.325+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51041a21b0 con 0x7f5104103a00 2024-03-18T01:38:07.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.326+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f510410adf0 con 0x7f5104103a00 2024-03-18T01:38:07.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.327+0000 7f50fb400640 1 -- 172.21.15.7:0/1642806084 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f50f4045020 con 0x7f5104103a00 2024-03-18T01:38:07.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.329+0000 7f50fb400640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50e80a2880 0x7f50e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:07.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.329+0000 7f50fb400640 1 -- 172.21.15.7:0/1642806084 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f50f40f0d10 con 0x7f5104103a00 2024-03-18T01:38:07.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.329+0000 7f5102000640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50e80a2880 0x7f50e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:07.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.330+0000 7f5102000640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50e80a2880 0x7f50e80a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f510419b2c0 tx=0x7f50f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:07.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.333+0000 7f50fb400640 1 -- 172.21.15.7:0/1642806084 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f50f40f6020 con 0x7f5104103a00 2024-03-18T01:38:07.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.547+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 --> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5104061c20 con 0x7f50e80a2880 2024-03-18T01:38:07.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.576+0000 7f50fb400640 1 -- 172.21.15.7:0/1642806084 <== mgr.14152 v2:172.21.15.7:6800/2743163723 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340539 (secure 0 0 0) 0x7f5104061c20 con 0x7f50e80a2880 2024-03-18T01:38:07.581 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:07.583 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-03-18T01:38:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50e80a2880 msgr2=0x7f50e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50e80a2880 0x7f50e80a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f510419b2c0 tx=0x7f50f0005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5104103a00 msgr2=0x7f5104199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5104103a00 0x7f5104199db0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50f4000c00 tx=0x7f50f402c660 comp rx=0 tx=0).stop 2024-03-18T01:38:07.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 shutdown_connections 2024-03-18T01:38:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f50e80a2880 0x7f50e80a4d40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f51041055b0 0x7f51041a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5104104bf0 0x7f510419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 --2- 172.21.15.7:0/1642806084 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5104103a00 0x7f5104199db0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 >> 172.21.15.7:0/1642806084 conn(0x7f51040fef90 msgr2=0x7f5104107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 shutdown_connections 2024-03-18T01:38:07.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.582+0000 7f5109a00640 1 -- 172.21.15.7:0/1642806084 wait complete. 2024-03-18T01:38:07.604 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-03-18T01:38:07.255546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664811,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1703,"num_read_kb":975,"num_write":440,"num_write_kb":3094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1181386,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":860,"ondisk_log_size":860,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":367564,"kb_used_data":14252,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124574260,"statfs":{"total":1151940427776,"available":1151564042240,"internally_reserved":0,"allocated":14594048,"data_stored":7980239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":61,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004810"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202699+0000","last_change":"2024-03-18T01:34:56.661330+0000","last_active":"2024-03-18T01:36:57.202699+0000","last_peered":"2024-03-18T01:36:57.202699+0000","last_clean":"2024-03-18T01:36:57.202699+0000","last_became_active":"2024-03-18T01:34:56.660949+0000","last_became_peered":"2024-03-18T01:34:56.660949+0000","last_unstale":"2024-03-18T01:36:57.202699+0000","last_undegraded":"2024-03-18T01:36:57.202699+0000","last_fullsized":"2024-03-18T01:36:57.202699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:20:40.962487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.209926+0000","last_change":"2024-03-18T01:34:56.652417+0000","last_active":"2024-03-18T01:36:57.209926+0000","last_peered":"2024-03-18T01:36:57.209926+0000","last_clean":"2024-03-18T01:36:57.209926+0000","last_became_active":"2024-03-18T01:34:56.651851+0000","last_became_peered":"2024-03-18T01:34:56.651851+0000","last_unstale":"2024-03-18T01:36:57.209926+0000","last_undegraded":"2024-03-18T01:36:57.209926+0000","last_fullsized":"2024-03-18T01:36:57.209926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:14:24.172177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214561+0000","last_change":"2024-03-18T01:34:56.647657+0000","last_active":"2024-03-18T01:36:57.214561+0000","last_peered":"2024-03-18T01:36:57.214561+0000","last_clean":"2024-03-18T01:36:57.214561+0000","last_became_active":"2024-03-18T01:34:56.647241+0000","last_became_peered":"2024-03-18T01:34:56.647241+0000","last_unstale":"2024-03-18T01:36:57.214561+0000","last_undegraded":"2024-03-18T01:36:57.214561+0000","last_fullsized":"2024-03-18T01:36:57.214561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:01.706351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.200510+0000","last_change":"2024-03-18T01:35:58.815017+0000","last_active":"2024-03-18T01:36:57.200510+0000","last_peered":"2024-03-18T01:36:57.200510+0000","last_clean":"2024-03-18T01:36:57.200510+0000","last_became_active":"2024-03-18T01:35:58.814140+0000","last_became_peered":"2024-03-18T01:35:58.814140+0000","last_unstale":"2024-03-18T01:36:57.200510+0000","last_undegraded":"2024-03-18T01:36:57.200510+0000","last_fullsized":"2024-03-18T01:36:57.200510+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:13:21.960119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.212414+0000","last_change":"2024-03-18T01:34:56.643992+0000","last_active":"2024-03-18T01:36:57.212414+0000","last_peered":"2024-03-18T01:36:57.212414+0000","last_clean":"2024-03-18T01:36:57.212414+0000","last_became_active":"2024-03-18T01:34:56.643652+0000","last_became_peered":"2024-03-18T01:34:56.643652+0000","last_unstale":"2024-03-18T01:36:57.212414+0000","last_undegraded":"2024-03-18T01:36:57.212414+0000","last_fullsized":"2024-03-18T01:36:57.212414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:12.005357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202984+0000","last_change":"2024-03-18T01:34:56.649955+0000","last_active":"2024-03-18T01:36:57.202984+0000","last_peered":"2024-03-18T01:36:57.202984+0000","last_clean":"2024-03-18T01:36:57.202984+0000","last_became_active":"2024-03-18T01:34:56.649609+0000","last_became_peered":"2024-03-18T01:34:56.649609+0000","last_unstale":"2024-03-18T01:36:57.202984+0000","last_undegraded":"2024-03-18T01:36:57.202984+0000","last_fullsized":"2024-03-18T01:36:57.202984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:34:28.113733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193161+0000","last_change":"2024-03-18T01:34:56.647561+0000","last_active":"2024-03-18T01:36:57.193161+0000","last_peered":"2024-03-18T01:36:57.193161+0000","last_clean":"2024-03-18T01:36:57.193161+0000","last_became_active":"2024-03-18T01:34:56.647040+0000","last_became_peered":"2024-03-18T01:34:56.647040+0000","last_unstale":"2024-03-18T01:36:57.193161+0000","last_undegraded":"2024-03-18T01:36:57.193161+0000","last_fullsized":"2024-03-18T01:36:57.193161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:42.433751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220346+0000","last_change":"2024-03-18T01:34:56.652073+0000","last_active":"2024-03-18T01:36:57.220346+0000","last_peered":"2024-03-18T01:36:57.220346+0000","last_clean":"2024-03-18T01:36:57.220346+0000","last_became_active":"2024-03-18T01:34:56.651748+0000","last_became_peered":"2024-03-18T01:34:56.651748+0000","last_unstale":"2024-03-18T01:36:57.220346+0000","last_undegraded":"2024-03-18T01:36:57.220346+0000","last_fullsized":"2024-03-18T01:36:57.220346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:33.148823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.200974+0000","last_change":"2024-03-18T01:34:56.645782+0000","last_active":"2024-03-18T01:36:57.200974+0000","last_peered":"2024-03-18T01:36:57.200974+0000","last_clean":"2024-03-18T01:36:57.200974+0000","last_became_active":"2024-03-18T01:34:56.645439+0000","last_became_peered":"2024-03-18T01:34:56.645439+0000","last_unstale":"2024-03-18T01:36:57.200974+0000","last_undegraded":"2024-03-18T01:36:57.200974+0000","last_fullsized":"2024-03-18T01:36:57.200974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:40.242348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.188909+0000","last_change":"2024-03-18T01:34:56.639697+0000","last_active":"2024-03-18T01:36:57.188909+0000","last_peered":"2024-03-18T01:36:57.188909+0000","last_clean":"2024-03-18T01:36:57.188909+0000","last_became_active":"2024-03-18T01:34:56.639339+0000","last_became_peered":"2024-03-18T01:34:56.639339+0000","last_unstale":"2024-03-18T01:36:57.188909+0000","last_undegraded":"2024-03-18T01:36:57.188909+0000","last_fullsized":"2024-03-18T01:36:57.188909+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:25:22.741437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.204096+0000","last_change":"2024-03-18T01:34:56.655341+0000","last_active":"2024-03-18T01:36:57.204096+0000","last_peered":"2024-03-18T01:36:57.204096+0000","last_clean":"2024-03-18T01:36:57.204096+0000","last_became_active":"2024-03-18T01:34:56.655015+0000","last_became_peered":"2024-03-18T01:34:56.655015+0000","last_unstale":"2024-03-18T01:36:57.204096+0000","last_undegraded":"2024-03-18T01:36:57.204096+0000","last_fullsized":"2024-03-18T01:36:57.204096+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:50.766000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.185565+0000","last_change":"2024-03-18T01:34:56.648399+0000","last_active":"2024-03-18T01:36:57.185565+0000","last_peered":"2024-03-18T01:36:57.185565+0000","last_clean":"2024-03-18T01:36:57.185565+0000","last_became_active":"2024-03-18T01:34:56.648042+0000","last_became_peered":"2024-03-18T01:34:56.648042+0000","last_unstale":"2024-03-18T01:36:57.185565+0000","last_undegraded":"2024-03-18T01:36:57.185565+0000","last_fullsized":"2024-03-18T01:36:57.185565+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:43:04.972230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211790+0000","last_change":"2024-03-18T01:34:56.653465+0000","last_active":"2024-03-18T01:36:57.211790+0000","last_peered":"2024-03-18T01:36:57.211790+0000","last_clean":"2024-03-18T01:36:57.211790+0000","last_became_active":"2024-03-18T01:34:56.651183+0000","last_became_peered":"2024-03-18T01:34:56.651183+0000","last_unstale":"2024-03-18T01:36:57.211790+0000","last_undegraded":"2024-03-18T01:36:57.211790+0000","last_fullsized":"2024-03-18T01:36:57.211790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:57.983330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211583+0000","last_change":"2024-03-18T01:34:56.658845+0000","last_active":"2024-03-18T01:36:57.211583+0000","last_peered":"2024-03-18T01:36:57.211583+0000","last_clean":"2024-03-18T01:36:57.211583+0000","last_became_active":"2024-03-18T01:34:56.658435+0000","last_became_peered":"2024-03-18T01:34:56.658435+0000","last_unstale":"2024-03-18T01:36:57.211583+0000","last_undegraded":"2024-03-18T01:36:57.211583+0000","last_fullsized":"2024-03-18T01:36:57.211583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:30:02.176325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211816+0000","last_change":"2024-03-18T01:34:56.660752+0000","last_active":"2024-03-18T01:36:57.211816+0000","last_peered":"2024-03-18T01:36:57.211816+0000","last_clean":"2024-03-18T01:36:57.211816+0000","last_became_active":"2024-03-18T01:34:56.660378+0000","last_became_peered":"2024-03-18T01:34:56.660378+0000","last_unstale":"2024-03-18T01:36:57.211816+0000","last_undegraded":"2024-03-18T01:36:57.211816+0000","last_fullsized":"2024-03-18T01:36:57.211816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:13.143481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214270+0000","last_change":"2024-03-18T01:34:56.656589+0000","last_active":"2024-03-18T01:36:57.214270+0000","last_peered":"2024-03-18T01:36:57.214270+0000","last_clean":"2024-03-18T01:36:57.214270+0000","last_became_active":"2024-03-18T01:34:56.656184+0000","last_became_peered":"2024-03-18T01:34:56.656184+0000","last_unstale":"2024-03-18T01:36:57.214270+0000","last_undegraded":"2024-03-18T01:36:57.214270+0000","last_fullsized":"2024-03-18T01:36:57.214270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:56.399889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202791+0000","last_change":"2024-03-18T01:34:56.661327+0000","last_active":"2024-03-18T01:36:57.202791+0000","last_peered":"2024-03-18T01:36:57.202791+0000","last_clean":"2024-03-18T01:36:57.202791+0000","last_became_active":"2024-03-18T01:34:56.660767+0000","last_became_peered":"2024-03-18T01:34:56.660767+0000","last_unstale":"2024-03-18T01:36:57.202791+0000","last_undegraded":"2024-03-18T01:36:57.202791+0000","last_fullsized":"2024-03-18T01:36:57.202791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:07:59.066028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203762+0000","last_change":"2024-03-18T01:34:56.650149+0000","last_active":"2024-03-18T01:36:57.203762+0000","last_peered":"2024-03-18T01:36:57.203762+0000","last_clean":"2024-03-18T01:36:57.203762+0000","last_became_active":"2024-03-18T01:34:56.649780+0000","last_became_peered":"2024-03-18T01:34:56.649780+0000","last_unstale":"2024-03-18T01:36:57.203762+0000","last_undegraded":"2024-03-18T01:36:57.203762+0000","last_fullsized":"2024-03-18T01:36:57.203762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:41.605563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.188792+0000","last_change":"2024-03-18T01:34:56.653919+0000","last_active":"2024-03-18T01:36:57.188792+0000","last_peered":"2024-03-18T01:36:57.188792+0000","last_clean":"2024-03-18T01:36:57.188792+0000","last_became_active":"2024-03-18T01:34:56.653512+0000","last_became_peered":"2024-03-18T01:34:56.653512+0000","last_unstale":"2024-03-18T01:36:57.188792+0000","last_undegraded":"2024-03-18T01:36:57.188792+0000","last_fullsized":"2024-03-18T01:36:57.188792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:35:27.419518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810162+0000","last_change":"2024-03-18T01:35:01.149999+0000","last_active":"2024-03-18T01:35:58.810162+0000","last_peered":"2024-03-18T01:35:58.810162+0000","last_clean":"2024-03-18T01:35:58.810162+0000","last_became_active":"2024-03-18T01:35:01.149603+0000","last_became_peered":"2024-03-18T01:35:01.149603+0000","last_unstale":"2024-03-18T01:35:58.810162+0000","last_undegraded":"2024-03-18T01:35:58.810162+0000","last_fullsized":"2024-03-18T01:35:58.810162+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:37.619517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":197,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:03.876246+0000","last_change":"2024-03-18T01:34:55.650404+0000","last_active":"2024-03-18T01:38:03.876246+0000","last_peered":"2024-03-18T01:38:03.876246+0000","last_clean":"2024-03-18T01:38:03.876246+0000","last_became_active":"2024-03-18T01:34:55.650068+0000","last_became_peered":"2024-03-18T01:34:55.650068+0000","last_unstale":"2024-03-18T01:38:03.876246+0000","last_undegraded":"2024-03-18T01:38:03.876246+0000","last_fullsized":"2024-03-18T01:38:03.876246+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:21:11.396529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":156,"num_read_kb":156,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211353+0000","last_change":"2024-03-18T01:34:56.654265+0000","last_active":"2024-03-18T01:36:57.211353+0000","last_peered":"2024-03-18T01:36:57.211353+0000","last_clean":"2024-03-18T01:36:57.211353+0000","last_became_active":"2024-03-18T01:34:56.653886+0000","last_became_peered":"2024-03-18T01:34:56.653886+0000","last_unstale":"2024-03-18T01:36:57.211353+0000","last_undegraded":"2024-03-18T01:36:57.211353+0000","last_fullsized":"2024-03-18T01:36:57.211353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:01:44.069966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801492+0000","last_change":"2024-03-18T01:35:01.155838+0000","last_active":"2024-03-18T01:35:58.801492+0000","last_peered":"2024-03-18T01:35:58.801492+0000","last_clean":"2024-03-18T01:35:58.801492+0000","last_became_active":"2024-03-18T01:35:01.155391+0000","last_became_peered":"2024-03-18T01:35:01.155391+0000","last_unstale":"2024-03-18T01:35:58.801492+0000","last_undegraded":"2024-03-18T01:35:58.801492+0000","last_fullsized":"2024-03-18T01:35:58.801492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:31.885767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.821989+0000","last_change":"2024-03-18T01:34:55.658830+0000","last_active":"2024-03-18T01:35:58.821989+0000","last_peered":"2024-03-18T01:35:58.821989+0000","last_clean":"2024-03-18T01:35:58.821989+0000","last_became_active":"2024-03-18T01:34:55.658395+0000","last_became_peered":"2024-03-18T01:34:55.658395+0000","last_unstale":"2024-03-18T01:35:58.821989+0000","last_undegraded":"2024-03-18T01:35:58.821989+0000","last_fullsized":"2024-03-18T01:35:58.821989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:07.169424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.215807+0000","last_change":"2024-03-18T01:34:56.656874+0000","last_active":"2024-03-18T01:36:57.215807+0000","last_peered":"2024-03-18T01:36:57.215807+0000","last_clean":"2024-03-18T01:36:57.215807+0000","last_became_active":"2024-03-18T01:34:56.656525+0000","last_became_peered":"2024-03-18T01:34:56.656525+0000","last_unstale":"2024-03-18T01:36:57.215807+0000","last_undegraded":"2024-03-18T01:36:57.215807+0000","last_fullsized":"2024-03-18T01:36:57.215807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:35:19.363586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.822875+0000","last_change":"2024-03-18T01:35:58.822392+0000","last_active":"2024-03-18T01:35:58.822875+0000","last_peered":"2024-03-18T01:35:58.822875+0000","last_clean":"2024-03-18T01:35:58.822875+0000","last_became_active":"2024-03-18T01:35:58.821792+0000","last_became_peered":"2024-03-18T01:35:58.821792+0000","last_unstale":"2024-03-18T01:35:58.822875+0000","last_undegraded":"2024-03-18T01:35:58.822875+0000","last_fullsized":"2024-03-18T01:35:58.822875+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:43:26.345066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802737+0000","last_change":"2024-03-18T01:34:55.648218+0000","last_active":"2024-03-18T01:35:58.802737+0000","last_peered":"2024-03-18T01:35:58.802737+0000","last_clean":"2024-03-18T01:35:58.802737+0000","last_became_active":"2024-03-18T01:34:55.647779+0000","last_became_peered":"2024-03-18T01:34:55.647779+0000","last_unstale":"2024-03-18T01:35:58.802737+0000","last_undegraded":"2024-03-18T01:35:58.802737+0000","last_fullsized":"2024-03-18T01:35:58.802737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:22:08.136973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205385+0000","last_change":"2024-03-18T01:34:56.649900+0000","last_active":"2024-03-18T01:36:57.205385+0000","last_peered":"2024-03-18T01:36:57.205385+0000","last_clean":"2024-03-18T01:36:57.205385+0000","last_became_active":"2024-03-18T01:34:56.649564+0000","last_became_peered":"2024-03-18T01:34:56.649564+0000","last_unstale":"2024-03-18T01:36:57.205385+0000","last_undegraded":"2024-03-18T01:36:57.205385+0000","last_fullsized":"2024-03-18T01:36:57.205385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:29.861861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801607+0000","last_change":"2024-03-18T01:35:01.147062+0000","last_active":"2024-03-18T01:35:58.801607+0000","last_peered":"2024-03-18T01:35:58.801607+0000","last_clean":"2024-03-18T01:35:58.801607+0000","last_became_active":"2024-03-18T01:35:01.146666+0000","last_became_peered":"2024-03-18T01:35:01.146666+0000","last_unstale":"2024-03-18T01:35:58.801607+0000","last_undegraded":"2024-03-18T01:35:58.801607+0000","last_fullsized":"2024-03-18T01:35:58.801607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:15:21.433263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.817437+0000","last_change":"2024-03-18T01:35:58.816601+0000","last_active":"2024-03-18T01:35:58.817437+0000","last_peered":"2024-03-18T01:35:58.817437+0000","last_clean":"2024-03-18T01:35:58.817437+0000","last_became_active":"2024-03-18T01:35:58.808678+0000","last_became_peered":"2024-03-18T01:35:58.808678+0000","last_unstale":"2024-03-18T01:35:58.817437+0000","last_undegraded":"2024-03-18T01:35:58.817437+0000","last_fullsized":"2024-03-18T01:35:58.817437+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:37:42.998856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192461+0000","last_change":"2024-03-18T01:34:56.650449+0000","last_active":"2024-03-18T01:36:57.192461+0000","last_peered":"2024-03-18T01:36:57.192461+0000","last_clean":"2024-03-18T01:36:57.192461+0000","last_became_active":"2024-03-18T01:34:56.650103+0000","last_became_peered":"2024-03-18T01:34:56.650103+0000","last_unstale":"2024-03-18T01:36:57.192461+0000","last_undegraded":"2024-03-18T01:36:57.192461+0000","last_fullsized":"2024-03-18T01:36:57.192461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:07:55.003592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809597+0000","last_change":"2024-03-18T01:35:01.132227+0000","last_active":"2024-03-18T01:35:58.809597+0000","last_peered":"2024-03-18T01:35:58.809597+0000","last_clean":"2024-03-18T01:35:58.809597+0000","last_became_active":"2024-03-18T01:35:01.131767+0000","last_became_peered":"2024-03-18T01:35:01.131767+0000","last_unstale":"2024-03-18T01:35:58.809597+0000","last_undegraded":"2024-03-18T01:35:58.809597+0000","last_fullsized":"2024-03-18T01:35:58.809597+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:54.335482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":189,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:02.803783+0000","last_change":"2024-03-18T01:34:55.649277+0000","last_active":"2024-03-18T01:38:02.803783+0000","last_peered":"2024-03-18T01:38:02.803783+0000","last_clean":"2024-03-18T01:38:02.803783+0000","last_became_active":"2024-03-18T01:34:55.648917+0000","last_became_peered":"2024-03-18T01:34:55.648917+0000","last_unstale":"2024-03-18T01:38:02.803783+0000","last_undegraded":"2024-03-18T01:38:02.803783+0000","last_fullsized":"2024-03-18T01:38:02.803783+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:00.117647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":156,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193004+0000","last_change":"2024-03-18T01:34:56.648394+0000","last_active":"2024-03-18T01:36:57.193004+0000","last_peered":"2024-03-18T01:36:57.193004+0000","last_clean":"2024-03-18T01:36:57.193004+0000","last_became_active":"2024-03-18T01:34:56.648043+0000","last_became_peered":"2024-03-18T01:34:56.648043+0000","last_unstale":"2024-03-18T01:36:57.193004+0000","last_undegraded":"2024-03-18T01:36:57.193004+0000","last_fullsized":"2024-03-18T01:36:57.193004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:54:52.531851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809510+0000","last_change":"2024-03-18T01:35:01.151346+0000","last_active":"2024-03-18T01:35:58.809510+0000","last_peered":"2024-03-18T01:35:58.809510+0000","last_clean":"2024-03-18T01:35:58.809510+0000","last_became_active":"2024-03-18T01:35:01.150916+0000","last_became_peered":"2024-03-18T01:35:01.150916+0000","last_unstale":"2024-03-18T01:35:58.809510+0000","last_undegraded":"2024-03-18T01:35:58.809510+0000","last_fullsized":"2024-03-18T01:35:58.809510+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:44:36.000983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804000+0000","last_change":"2024-03-18T01:34:55.648498+0000","last_active":"2024-03-18T01:35:58.804000+0000","last_peered":"2024-03-18T01:35:58.804000+0000","last_clean":"2024-03-18T01:35:58.804000+0000","last_became_active":"2024-03-18T01:34:55.648152+0000","last_became_peered":"2024-03-18T01:34:55.648152+0000","last_unstale":"2024-03-18T01:35:58.804000+0000","last_undegraded":"2024-03-18T01:35:58.804000+0000","last_fullsized":"2024-03-18T01:35:58.804000+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:19:33.138620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211371+0000","last_change":"2024-03-18T01:34:56.651420+0000","last_active":"2024-03-18T01:36:57.211371+0000","last_peered":"2024-03-18T01:36:57.211371+0000","last_clean":"2024-03-18T01:36:57.211371+0000","last_became_active":"2024-03-18T01:34:56.650909+0000","last_became_peered":"2024-03-18T01:34:56.650909+0000","last_unstale":"2024-03-18T01:36:57.211371+0000","last_undegraded":"2024-03-18T01:36:57.211371+0000","last_fullsized":"2024-03-18T01:36:57.211371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:43:40.548970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804705+0000","last_change":"2024-03-18T01:35:01.157759+0000","last_active":"2024-03-18T01:35:58.804705+0000","last_peered":"2024-03-18T01:35:58.804705+0000","last_clean":"2024-03-18T01:35:58.804705+0000","last_became_active":"2024-03-18T01:35:01.157288+0000","last_became_peered":"2024-03-18T01:35:01.157288+0000","last_unstale":"2024-03-18T01:35:58.804705+0000","last_undegraded":"2024-03-18T01:35:58.804705+0000","last_fullsized":"2024-03-18T01:35:58.804705+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:05:41.272727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.818781+0000","last_change":"2024-03-18T01:34:55.650388+0000","last_active":"2024-03-18T01:35:58.818781+0000","last_peered":"2024-03-18T01:35:58.818781+0000","last_clean":"2024-03-18T01:35:58.818781+0000","last_became_active":"2024-03-18T01:34:55.650055+0000","last_became_peered":"2024-03-18T01:34:55.650055+0000","last_unstale":"2024-03-18T01:35:58.818781+0000","last_undegraded":"2024-03-18T01:35:58.818781+0000","last_fullsized":"2024-03-18T01:35:58.818781+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:29:40.758561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220432+0000","last_change":"2024-03-18T01:34:56.658634+0000","last_active":"2024-03-18T01:36:57.220432+0000","last_peered":"2024-03-18T01:36:57.220432+0000","last_clean":"2024-03-18T01:36:57.220432+0000","last_became_active":"2024-03-18T01:34:56.657196+0000","last_became_peered":"2024-03-18T01:34:56.657196+0000","last_unstale":"2024-03-18T01:36:57.220432+0000","last_undegraded":"2024-03-18T01:36:57.220432+0000","last_fullsized":"2024-03-18T01:36:57.220432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:12:36.265949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811261+0000","last_change":"2024-03-18T01:35:01.150431+0000","last_active":"2024-03-18T01:35:58.811261+0000","last_peered":"2024-03-18T01:35:58.811261+0000","last_clean":"2024-03-18T01:35:58.811261+0000","last_became_active":"2024-03-18T01:35:01.149981+0000","last_became_peered":"2024-03-18T01:35:01.149981+0000","last_unstale":"2024-03-18T01:35:58.811261+0000","last_undegraded":"2024-03-18T01:35:58.811261+0000","last_fullsized":"2024-03-18T01:35:58.811261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:34:55.371860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807035+0000","last_change":"2024-03-18T01:34:55.649346+0000","last_active":"2024-03-18T01:35:58.807035+0000","last_peered":"2024-03-18T01:35:58.807035+0000","last_clean":"2024-03-18T01:35:58.807035+0000","last_became_active":"2024-03-18T01:34:55.648958+0000","last_became_peered":"2024-03-18T01:34:55.648958+0000","last_unstale":"2024-03-18T01:35:58.807035+0000","last_undegraded":"2024-03-18T01:35:58.807035+0000","last_fullsized":"2024-03-18T01:35:58.807035+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:53.788081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.219078+0000","last_change":"2024-03-18T01:34:56.653525+0000","last_active":"2024-03-18T01:36:57.219078+0000","last_peered":"2024-03-18T01:36:57.219078+0000","last_clean":"2024-03-18T01:36:57.219078+0000","last_became_active":"2024-03-18T01:34:56.653218+0000","last_became_peered":"2024-03-18T01:34:56.653218+0000","last_unstale":"2024-03-18T01:36:57.219078+0000","last_undegraded":"2024-03-18T01:36:57.219078+0000","last_fullsized":"2024-03-18T01:36:57.219078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:32:37.479782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816359+0000","last_change":"2024-03-18T01:35:01.135505+0000","last_active":"2024-03-18T01:35:58.816359+0000","last_peered":"2024-03-18T01:35:58.816359+0000","last_clean":"2024-03-18T01:35:58.816359+0000","last_became_active":"2024-03-18T01:35:01.135055+0000","last_became_peered":"2024-03-18T01:35:01.135055+0000","last_unstale":"2024-03-18T01:35:58.816359+0000","last_undegraded":"2024-03-18T01:35:58.816359+0000","last_fullsized":"2024-03-18T01:35:58.816359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:52:35.069313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":337,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:01.802027+0000","last_change":"2024-03-18T01:34:55.659389+0000","last_active":"2024-03-18T01:38:01.802027+0000","last_peered":"2024-03-18T01:38:01.802027+0000","last_clean":"2024-03-18T01:38:01.802027+0000","last_became_active":"2024-03-18T01:34:55.659090+0000","last_became_peered":"2024-03-18T01:34:55.659090+0000","last_unstale":"2024-03-18T01:38:01.802027+0000","last_undegraded":"2024-03-18T01:38:01.802027+0000","last_fullsized":"2024-03-18T01:38:01.802027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:55:51.731788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":304,"num_read_kb":304,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.206644+0000","last_change":"2024-03-18T01:34:56.650411+0000","last_active":"2024-03-18T01:36:57.206644+0000","last_peered":"2024-03-18T01:36:57.206644+0000","last_clean":"2024-03-18T01:36:57.206644+0000","last_became_active":"2024-03-18T01:34:56.650036+0000","last_became_peered":"2024-03-18T01:34:56.650036+0000","last_unstale":"2024-03-18T01:36:57.206644+0000","last_undegraded":"2024-03-18T01:36:57.206644+0000","last_fullsized":"2024-03-18T01:36:57.206644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:35:59.912207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.811599+0000","last_change":"2024-03-18T01:35:58.811137+0000","last_active":"2024-03-18T01:35:58.811599+0000","last_peered":"2024-03-18T01:35:58.811599+0000","last_clean":"2024-03-18T01:35:58.811599+0000","last_became_active":"2024-03-18T01:35:58.810398+0000","last_became_peered":"2024-03-18T01:35:58.810398+0000","last_unstale":"2024-03-18T01:35:58.811599+0000","last_undegraded":"2024-03-18T01:35:58.811599+0000","last_fullsized":"2024-03-18T01:35:58.811599+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:08.271404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,2,4],"acting":[10,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807107+0000","last_change":"2024-03-18T01:34:55.647879+0000","last_active":"2024-03-18T01:35:58.807107+0000","last_peered":"2024-03-18T01:35:58.807107+0000","last_clean":"2024-03-18T01:35:58.807107+0000","last_became_active":"2024-03-18T01:34:55.647549+0000","last_became_peered":"2024-03-18T01:34:55.647549+0000","last_unstale":"2024-03-18T01:35:58.807107+0000","last_undegraded":"2024-03-18T01:35:58.807107+0000","last_fullsized":"2024-03-18T01:35:58.807107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:12.261729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.210366+0000","last_change":"2024-03-18T01:35:58.813429+0000","last_active":"2024-03-18T01:36:57.210366+0000","last_peered":"2024-03-18T01:36:57.210366+0000","last_clean":"2024-03-18T01:36:57.210366+0000","last_became_active":"2024-03-18T01:35:58.812659+0000","last_became_peered":"2024-03-18T01:35:58.812659+0000","last_unstale":"2024-03-18T01:36:57.210366+0000","last_undegraded":"2024-03-18T01:36:57.210366+0000","last_fullsized":"2024-03-18T01:36:57.210366+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:22:01.273649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802250+0000","last_change":"2024-03-18T01:35:01.153448+0000","last_active":"2024-03-18T01:35:58.802250+0000","last_peered":"2024-03-18T01:35:58.802250+0000","last_clean":"2024-03-18T01:35:58.802250+0000","last_became_active":"2024-03-18T01:35:01.152593+0000","last_became_peered":"2024-03-18T01:35:01.152593+0000","last_unstale":"2024-03-18T01:35:58.802250+0000","last_undegraded":"2024-03-18T01:35:58.802250+0000","last_fullsized":"2024-03-18T01:35:58.802250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:46.363662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808590+0000","last_change":"2024-03-18T01:34:55.655079+0000","last_active":"2024-03-18T01:35:58.808590+0000","last_peered":"2024-03-18T01:35:58.808590+0000","last_clean":"2024-03-18T01:35:58.808590+0000","last_became_active":"2024-03-18T01:34:55.654702+0000","last_became_peered":"2024-03-18T01:34:55.654702+0000","last_unstale":"2024-03-18T01:35:58.808590+0000","last_undegraded":"2024-03-18T01:35:58.808590+0000","last_fullsized":"2024-03-18T01:35:58.808590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:50:46.524614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205015+0000","last_change":"2024-03-18T01:34:56.644441+0000","last_active":"2024-03-18T01:36:57.205015+0000","last_peered":"2024-03-18T01:36:57.205015+0000","last_clean":"2024-03-18T01:36:57.205015+0000","last_became_active":"2024-03-18T01:34:56.643924+0000","last_became_peered":"2024-03-18T01:34:56.643924+0000","last_unstale":"2024-03-18T01:36:57.205015+0000","last_undegraded":"2024-03-18T01:36:57.205015+0000","last_fullsized":"2024-03-18T01:36:57.205015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:44.260300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810329+0000","last_change":"2024-03-18T01:35:01.150022+0000","last_active":"2024-03-18T01:35:58.810329+0000","last_peered":"2024-03-18T01:35:58.810329+0000","last_clean":"2024-03-18T01:35:58.810329+0000","last_became_active":"2024-03-18T01:35:01.149510+0000","last_became_peered":"2024-03-18T01:35:01.149510+0000","last_unstale":"2024-03-18T01:35:58.810329+0000","last_undegraded":"2024-03-18T01:35:58.810329+0000","last_fullsized":"2024-03-18T01:35:58.810329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:40:36.140068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'38","reported_seq":378,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:01.802481+0000","last_change":"2024-03-18T01:34:55.642397+0000","last_active":"2024-03-18T01:38:01.802481+0000","last_peered":"2024-03-18T01:38:01.802481+0000","last_clean":"2024-03-18T01:38:01.802481+0000","last_became_active":"2024-03-18T01:34:55.642046+0000","last_became_peered":"2024-03-18T01:34:55.642046+0000","last_unstale":"2024-03-18T01:38:01.802481+0000","last_undegraded":"2024-03-18T01:38:01.802481+0000","last_fullsized":"2024-03-18T01:38:01.802481+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:48:31.631414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66945,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":312,"num_read_kb":222,"num_write":40,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.182441+0000","last_change":"2024-03-18T01:34:56.644388+0000","last_active":"2024-03-18T01:36:57.182441+0000","last_peered":"2024-03-18T01:36:57.182441+0000","last_clean":"2024-03-18T01:36:57.182441+0000","last_became_active":"2024-03-18T01:34:56.644014+0000","last_became_peered":"2024-03-18T01:34:56.644014+0000","last_unstale":"2024-03-18T01:36:57.182441+0000","last_undegraded":"2024-03-18T01:36:57.182441+0000","last_fullsized":"2024-03-18T01:36:57.182441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:06:18.193600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.823299+0000","last_change":"2024-03-18T01:35:58.822872+0000","last_active":"2024-03-18T01:35:58.823299+0000","last_peered":"2024-03-18T01:35:58.823299+0000","last_clean":"2024-03-18T01:35:58.823299+0000","last_became_active":"2024-03-18T01:35:58.822325+0000","last_became_peered":"2024-03-18T01:35:58.822325+0000","last_unstale":"2024-03-18T01:35:58.823299+0000","last_undegraded":"2024-03-18T01:35:58.823299+0000","last_fullsized":"2024-03-18T01:35:58.823299+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:54:21.392764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.834419+0000","last_change":"2024-03-18T01:35:58.834419+0000","last_active":"2024-03-18T01:35:58.834419+0000","last_peered":"2024-03-18T01:35:58.834419+0000","last_clean":"2024-03-18T01:35:58.834419+0000","last_became_active":"2024-03-18T01:35:58.822930+0000","last_became_peered":"2024-03-18T01:35:58.822930+0000","last_unstale":"2024-03-18T01:35:58.834419+0000","last_undegraded":"2024-03-18T01:35:58.834419+0000","last_fullsized":"2024-03-18T01:35:58.834419+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:15.777787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.222003+0000","last_change":"2024-03-18T01:34:56.657786+0000","last_active":"2024-03-18T01:36:57.222003+0000","last_peered":"2024-03-18T01:36:57.222003+0000","last_clean":"2024-03-18T01:36:57.222003+0000","last_became_active":"2024-03-18T01:34:56.657432+0000","last_became_peered":"2024-03-18T01:34:56.657432+0000","last_unstale":"2024-03-18T01:36:57.222003+0000","last_undegraded":"2024-03-18T01:36:57.222003+0000","last_fullsized":"2024-03-18T01:36:57.222003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:13:39.498562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804963+0000","last_change":"2024-03-18T01:35:01.150495+0000","last_active":"2024-03-18T01:35:58.804963+0000","last_peered":"2024-03-18T01:35:58.804963+0000","last_clean":"2024-03-18T01:35:58.804963+0000","last_became_active":"2024-03-18T01:35:01.150033+0000","last_became_peered":"2024-03-18T01:35:01.150033+0000","last_unstale":"2024-03-18T01:35:58.804963+0000","last_undegraded":"2024-03-18T01:35:58.804963+0000","last_fullsized":"2024-03-18T01:35:58.804963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:38.433827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807841+0000","last_change":"2024-03-18T01:34:55.658820+0000","last_active":"2024-03-18T01:35:58.807841+0000","last_peered":"2024-03-18T01:35:58.807841+0000","last_clean":"2024-03-18T01:35:58.807841+0000","last_became_active":"2024-03-18T01:34:55.658446+0000","last_became_peered":"2024-03-18T01:34:55.658446+0000","last_unstale":"2024-03-18T01:35:58.807841+0000","last_undegraded":"2024-03-18T01:35:58.807841+0000","last_fullsized":"2024-03-18T01:35:58.807841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:19:21.759294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.199889+0000","last_change":"2024-03-18T01:34:56.652890+0000","last_active":"2024-03-18T01:36:57.199889+0000","last_peered":"2024-03-18T01:36:57.199889+0000","last_clean":"2024-03-18T01:36:57.199889+0000","last_became_active":"2024-03-18T01:34:56.652444+0000","last_became_peered":"2024-03-18T01:34:56.652444+0000","last_unstale":"2024-03-18T01:36:57.199889+0000","last_undegraded":"2024-03-18T01:36:57.199889+0000","last_fullsized":"2024-03-18T01:36:57.199889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:58:47.065907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812407+0000","last_change":"2024-03-18T01:35:01.154700+0000","last_active":"2024-03-18T01:35:58.812407+0000","last_peered":"2024-03-18T01:35:58.812407+0000","last_clean":"2024-03-18T01:35:58.812407+0000","last_became_active":"2024-03-18T01:35:01.148453+0000","last_became_peered":"2024-03-18T01:35:01.148453+0000","last_unstale":"2024-03-18T01:35:58.812407+0000","last_undegraded":"2024-03-18T01:35:58.812407+0000","last_fullsized":"2024-03-18T01:35:58.812407+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:08:40.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":193,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:03.952949+0000","last_change":"2024-03-18T01:34:55.643861+0000","last_active":"2024-03-18T01:38:03.952949+0000","last_peered":"2024-03-18T01:38:03.952949+0000","last_clean":"2024-03-18T01:38:03.952949+0000","last_became_active":"2024-03-18T01:34:55.643352+0000","last_became_peered":"2024-03-18T01:34:55.643352+0000","last_unstale":"2024-03-18T01:38:03.952949+0000","last_undegraded":"2024-03-18T01:38:03.952949+0000","last_fullsized":"2024-03-18T01:38:03.952949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:56:53.685571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":159,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193499+0000","last_change":"2024-03-18T01:35:58.814042+0000","last_active":"2024-03-18T01:36:57.193499+0000","last_peered":"2024-03-18T01:36:57.193499+0000","last_clean":"2024-03-18T01:36:57.193499+0000","last_became_active":"2024-03-18T01:35:58.813622+0000","last_became_peered":"2024-03-18T01:35:58.813622+0000","last_unstale":"2024-03-18T01:36:57.193499+0000","last_undegraded":"2024-03-18T01:36:57.193499+0000","last_fullsized":"2024-03-18T01:36:57.193499+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:52.815608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.825880+0000","last_change":"2024-03-18T01:35:58.824798+0000","last_active":"2024-03-18T01:35:58.825880+0000","last_peered":"2024-03-18T01:35:58.825880+0000","last_clean":"2024-03-18T01:35:58.825880+0000","last_became_active":"2024-03-18T01:35:58.823278+0000","last_became_peered":"2024-03-18T01:35:58.823278+0000","last_unstale":"2024-03-18T01:35:58.825880+0000","last_undegraded":"2024-03-18T01:35:58.825880+0000","last_fullsized":"2024-03-18T01:35:58.825880+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:18:53.871541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801980+0000","last_change":"2024-03-18T01:34:55.658245+0000","last_active":"2024-03-18T01:35:58.801980+0000","last_peered":"2024-03-18T01:35:58.801980+0000","last_clean":"2024-03-18T01:35:58.801980+0000","last_became_active":"2024-03-18T01:34:55.657916+0000","last_became_peered":"2024-03-18T01:34:55.657916+0000","last_unstale":"2024-03-18T01:35:58.801980+0000","last_undegraded":"2024-03-18T01:35:58.801980+0000","last_fullsized":"2024-03-18T01:35:58.801980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:18:49.207721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192214+0000","last_change":"2024-03-18T01:34:56.648646+0000","last_active":"2024-03-18T01:36:57.192214+0000","last_peered":"2024-03-18T01:36:57.192214+0000","last_clean":"2024-03-18T01:36:57.192214+0000","last_became_active":"2024-03-18T01:34:56.648257+0000","last_became_peered":"2024-03-18T01:34:56.648257+0000","last_unstale":"2024-03-18T01:36:57.192214+0000","last_undegraded":"2024-03-18T01:36:57.192214+0000","last_fullsized":"2024-03-18T01:36:57.192214+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:40:53.788445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.807611+0000","last_change":"2024-03-18T01:35:01.136912+0000","last_active":"2024-03-18T01:35:58.807611+0000","last_peered":"2024-03-18T01:35:58.807611+0000","last_clean":"2024-03-18T01:35:58.807611+0000","last_became_active":"2024-03-18T01:35:01.136611+0000","last_became_peered":"2024-03-18T01:35:01.136611+0000","last_unstale":"2024-03-18T01:35:58.807611+0000","last_undegraded":"2024-03-18T01:35:58.807611+0000","last_fullsized":"2024-03-18T01:35:58.807611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:19:31.089605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802559+0000","last_change":"2024-03-18T01:34:55.648345+0000","last_active":"2024-03-18T01:35:58.802559+0000","last_peered":"2024-03-18T01:35:58.802559+0000","last_clean":"2024-03-18T01:35:58.802559+0000","last_became_active":"2024-03-18T01:34:55.647113+0000","last_became_peered":"2024-03-18T01:34:55.647113+0000","last_unstale":"2024-03-18T01:35:58.802559+0000","last_undegraded":"2024-03-18T01:35:58.802559+0000","last_fullsized":"2024-03-18T01:35:58.802559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:42:20.904129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190181+0000","last_change":"2024-03-18T01:34:56.649147+0000","last_active":"2024-03-18T01:36:57.190181+0000","last_peered":"2024-03-18T01:36:57.190181+0000","last_clean":"2024-03-18T01:36:57.190181+0000","last_became_active":"2024-03-18T01:34:56.648815+0000","last_became_peered":"2024-03-18T01:34:56.648815+0000","last_unstale":"2024-03-18T01:36:57.190181+0000","last_undegraded":"2024-03-18T01:36:57.190181+0000","last_fullsized":"2024-03-18T01:36:57.190181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:30:53.560679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802064+0000","last_change":"2024-03-18T01:35:01.156778+0000","last_active":"2024-03-18T01:35:58.802064+0000","last_peered":"2024-03-18T01:35:58.802064+0000","last_clean":"2024-03-18T01:35:58.802064+0000","last_became_active":"2024-03-18T01:35:01.156367+0000","last_became_peered":"2024-03-18T01:35:01.156367+0000","last_unstale":"2024-03-18T01:35:58.802064+0000","last_undegraded":"2024-03-18T01:35:58.802064+0000","last_fullsized":"2024-03-18T01:35:58.802064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:42:46.223915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.810972+0000","last_change":"2024-03-18T01:34:55.658722+0000","last_active":"2024-03-18T01:35:58.810972+0000","last_peered":"2024-03-18T01:35:58.810972+0000","last_clean":"2024-03-18T01:35:58.810972+0000","last_became_active":"2024-03-18T01:34:55.658321+0000","last_became_peered":"2024-03-18T01:34:55.658321+0000","last_unstale":"2024-03-18T01:35:58.810972+0000","last_undegraded":"2024-03-18T01:35:58.810972+0000","last_fullsized":"2024-03-18T01:35:58.810972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:15:05.753226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.221076+0000","last_change":"2024-03-18T01:34:56.649415+0000","last_active":"2024-03-18T01:36:57.221076+0000","last_peered":"2024-03-18T01:36:57.221076+0000","last_clean":"2024-03-18T01:36:57.221076+0000","last_became_active":"2024-03-18T01:34:56.649019+0000","last_became_peered":"2024-03-18T01:34:56.649019+0000","last_unstale":"2024-03-18T01:36:57.221076+0000","last_undegraded":"2024-03-18T01:36:57.221076+0000","last_fullsized":"2024-03-18T01:36:57.221076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:28.473683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819934+0000","last_change":"2024-03-18T01:35:01.148937+0000","last_active":"2024-03-18T01:35:58.819934+0000","last_peered":"2024-03-18T01:35:58.819934+0000","last_clean":"2024-03-18T01:35:58.819934+0000","last_became_active":"2024-03-18T01:35:01.148415+0000","last_became_peered":"2024-03-18T01:35:01.148415+0000","last_unstale":"2024-03-18T01:35:58.819934+0000","last_undegraded":"2024-03-18T01:35:58.819934+0000","last_fullsized":"2024-03-18T01:35:58.819934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:45.926970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":190,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:04.877699+0000","last_change":"2024-03-18T01:34:55.653443+0000","last_active":"2024-03-18T01:38:04.877699+0000","last_peered":"2024-03-18T01:38:04.877699+0000","last_clean":"2024-03-18T01:38:04.877699+0000","last_became_active":"2024-03-18T01:34:55.653072+0000","last_became_peered":"2024-03-18T01:34:55.653072+0000","last_unstale":"2024-03-18T01:38:04.877699+0000","last_undegraded":"2024-03-18T01:38:04.877699+0000","last_fullsized":"2024-03-18T01:38:04.877699+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:26.376636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":157,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.202535+0000","last_change":"2024-03-18T01:35:58.811178+0000","last_active":"2024-03-18T01:36:57.202535+0000","last_peered":"2024-03-18T01:36:57.202535+0000","last_clean":"2024-03-18T01:36:57.202535+0000","last_became_active":"2024-03-18T01:35:58.810764+0000","last_became_peered":"2024-03-18T01:35:58.810764+0000","last_unstale":"2024-03-18T01:36:57.202535+0000","last_undegraded":"2024-03-18T01:36:57.202535+0000","last_fullsized":"2024-03-18T01:36:57.202535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:49.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819478+0000","last_change":"2024-03-18T01:35:01.152767+0000","last_active":"2024-03-18T01:35:58.819478+0000","last_peered":"2024-03-18T01:35:58.819478+0000","last_clean":"2024-03-18T01:35:58.819478+0000","last_became_active":"2024-03-18T01:35:01.150619+0000","last_became_peered":"2024-03-18T01:35:01.150619+0000","last_unstale":"2024-03-18T01:35:58.819478+0000","last_undegraded":"2024-03-18T01:35:58.819478+0000","last_fullsized":"2024-03-18T01:35:58.819478+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:57:28.258251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191740+0000","last_change":"2024-03-18T01:34:56.650541+0000","last_active":"2024-03-18T01:36:57.191740+0000","last_peered":"2024-03-18T01:36:57.191740+0000","last_clean":"2024-03-18T01:36:57.191740+0000","last_became_active":"2024-03-18T01:34:56.650072+0000","last_became_peered":"2024-03-18T01:34:56.650072+0000","last_unstale":"2024-03-18T01:36:57.191740+0000","last_undegraded":"2024-03-18T01:36:57.191740+0000","last_fullsized":"2024-03-18T01:36:57.191740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:33:51.224883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":189,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:02.950644+0000","last_change":"2024-03-18T01:34:55.664297+0000","last_active":"2024-03-18T01:38:02.950644+0000","last_peered":"2024-03-18T01:38:02.950644+0000","last_clean":"2024-03-18T01:38:02.950644+0000","last_became_active":"2024-03-18T01:34:55.663938+0000","last_became_peered":"2024-03-18T01:34:55.663938+0000","last_unstale":"2024-03-18T01:38:02.950644+0000","last_undegraded":"2024-03-18T01:38:02.950644+0000","last_fullsized":"2024-03-18T01:38:02.950644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:05:51.771987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":155,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806717+0000","last_change":"2024-03-18T01:35:01.141736+0000","last_active":"2024-03-18T01:35:58.806717+0000","last_peered":"2024-03-18T01:35:58.806717+0000","last_clean":"2024-03-18T01:35:58.806717+0000","last_became_active":"2024-03-18T01:35:01.141381+0000","last_became_peered":"2024-03-18T01:35:01.141381+0000","last_unstale":"2024-03-18T01:35:58.806717+0000","last_undegraded":"2024-03-18T01:35:58.806717+0000","last_fullsized":"2024-03-18T01:35:58.806717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:25:04.190250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216313+0000","last_change":"2024-03-18T01:34:56.666439+0000","last_active":"2024-03-18T01:36:57.216313+0000","last_peered":"2024-03-18T01:36:57.216313+0000","last_clean":"2024-03-18T01:36:57.216313+0000","last_became_active":"2024-03-18T01:34:56.664556+0000","last_became_peered":"2024-03-18T01:34:56.664556+0000","last_unstale":"2024-03-18T01:36:57.216313+0000","last_undegraded":"2024-03-18T01:36:57.216313+0000","last_fullsized":"2024-03-18T01:36:57.216313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:08:48.827008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802167+0000","last_change":"2024-03-18T01:34:55.656484+0000","last_active":"2024-03-18T01:35:58.802167+0000","last_peered":"2024-03-18T01:35:58.802167+0000","last_clean":"2024-03-18T01:35:58.802167+0000","last_became_active":"2024-03-18T01:34:55.656125+0000","last_became_peered":"2024-03-18T01:34:55.656125+0000","last_unstale":"2024-03-18T01:35:58.802167+0000","last_undegraded":"2024-03-18T01:35:58.802167+0000","last_fullsized":"2024-03-18T01:35:58.802167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:58:25.944403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":350,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806027+0000","last_change":"2024-03-18T01:33:35.824651+0000","last_active":"2024-03-18T01:35:58.806027+0000","last_peered":"2024-03-18T01:35:58.806027+0000","last_clean":"2024-03-18T01:35:58.806027+0000","last_became_active":"2024-03-18T01:33:35.824106+0000","last_became_peered":"2024-03-18T01:33:35.824106+0000","last_unstale":"2024-03-18T01:35:58.806027+0000","last_undegraded":"2024-03-18T01:35:58.806027+0000","last_fullsized":"2024-03-18T01:35:58.806027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:29:41.724134+0000","last_clean_scrub_stamp":"2024-03-18T01:29:41.724134+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:20:06.422316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808524+0000","last_change":"2024-03-18T01:35:01.157735+0000","last_active":"2024-03-18T01:35:58.808524+0000","last_peered":"2024-03-18T01:35:58.808524+0000","last_clean":"2024-03-18T01:35:58.808524+0000","last_became_active":"2024-03-18T01:35:01.156954+0000","last_became_peered":"2024-03-18T01:35:01.156954+0000","last_unstale":"2024-03-18T01:35:58.808524+0000","last_undegraded":"2024-03-18T01:35:58.808524+0000","last_fullsized":"2024-03-18T01:35:58.808524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:00.106809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.193743+0000","last_change":"2024-03-18T01:34:56.646808+0000","last_active":"2024-03-18T01:36:57.193743+0000","last_peered":"2024-03-18T01:36:57.193743+0000","last_clean":"2024-03-18T01:36:57.193743+0000","last_became_active":"2024-03-18T01:34:56.646437+0000","last_became_peered":"2024-03-18T01:34:56.646437+0000","last_unstale":"2024-03-18T01:36:57.193743+0000","last_undegraded":"2024-03-18T01:36:57.193743+0000","last_fullsized":"2024-03-18T01:36:57.193743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:46:09.703889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806136+0000","last_change":"2024-03-18T01:34:55.645365+0000","last_active":"2024-03-18T01:35:58.806136+0000","last_peered":"2024-03-18T01:35:58.806136+0000","last_clean":"2024-03-18T01:35:58.806136+0000","last_became_active":"2024-03-18T01:34:55.645018+0000","last_became_peered":"2024-03-18T01:34:55.645018+0000","last_unstale":"2024-03-18T01:35:58.806136+0000","last_undegraded":"2024-03-18T01:35:58.806136+0000","last_fullsized":"2024-03-18T01:35:58.806136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:08:31.869328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.804962+0000","last_change":"2024-03-18T01:35:01.147294+0000","last_active":"2024-03-18T01:35:58.804962+0000","last_peered":"2024-03-18T01:35:58.804962+0000","last_clean":"2024-03-18T01:35:58.804962+0000","last_became_active":"2024-03-18T01:35:01.146858+0000","last_became_peered":"2024-03-18T01:35:01.146858+0000","last_unstale":"2024-03-18T01:35:58.804962+0000","last_undegraded":"2024-03-18T01:35:58.804962+0000","last_fullsized":"2024-03-18T01:35:58.804962+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:45:24.010746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802584+0000","last_change":"2024-03-18T01:34:55.658110+0000","last_active":"2024-03-18T01:35:58.802584+0000","last_peered":"2024-03-18T01:35:58.802584+0000","last_clean":"2024-03-18T01:35:58.802584+0000","last_became_active":"2024-03-18T01:34:55.657802+0000","last_became_peered":"2024-03-18T01:34:55.657802+0000","last_unstale":"2024-03-18T01:35:58.802584+0000","last_undegraded":"2024-03-18T01:35:58.802584+0000","last_fullsized":"2024-03-18T01:35:58.802584+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:34.264115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.205099+0000","last_change":"2024-03-18T01:34:56.656689+0000","last_active":"2024-03-18T01:36:57.205099+0000","last_peered":"2024-03-18T01:36:57.205099+0000","last_clean":"2024-03-18T01:36:57.205099+0000","last_became_active":"2024-03-18T01:34:56.655653+0000","last_became_peered":"2024-03-18T01:34:56.655653+0000","last_unstale":"2024-03-18T01:36:57.205099+0000","last_undegraded":"2024-03-18T01:36:57.205099+0000","last_fullsized":"2024-03-18T01:36:57.205099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:58.370904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.816539+0000","last_change":"2024-03-18T01:35:01.137419+0000","last_active":"2024-03-18T01:35:58.816539+0000","last_peered":"2024-03-18T01:35:58.816539+0000","last_clean":"2024-03-18T01:35:58.816539+0000","last_became_active":"2024-03-18T01:35:01.137033+0000","last_became_peered":"2024-03-18T01:35:01.137033+0000","last_unstale":"2024-03-18T01:35:58.816539+0000","last_undegraded":"2024-03-18T01:35:58.816539+0000","last_fullsized":"2024-03-18T01:35:58.816539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:13:48.382329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:05.854415+0000","last_change":"2024-03-18T01:34:55.663134+0000","last_active":"2024-03-18T01:37:05.854415+0000","last_peered":"2024-03-18T01:37:05.854415+0000","last_clean":"2024-03-18T01:37:05.854415+0000","last_became_active":"2024-03-18T01:34:55.662083+0000","last_became_peered":"2024-03-18T01:34:55.662083+0000","last_unstale":"2024-03-18T01:37:05.854415+0000","last_undegraded":"2024-03-18T01:37:05.854415+0000","last_fullsized":"2024-03-18T01:37:05.854415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:01:09.186599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216678+0000","last_change":"2024-03-18T01:34:56.664761+0000","last_active":"2024-03-18T01:36:57.216678+0000","last_peered":"2024-03-18T01:36:57.216678+0000","last_clean":"2024-03-18T01:36:57.216678+0000","last_became_active":"2024-03-18T01:34:56.663657+0000","last_became_peered":"2024-03-18T01:34:56.663657+0000","last_unstale":"2024-03-18T01:36:57.216678+0000","last_undegraded":"2024-03-18T01:36:57.216678+0000","last_fullsized":"2024-03-18T01:36:57.216678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:26:53.075000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809995+0000","last_change":"2024-03-18T01:35:01.133002+0000","last_active":"2024-03-18T01:35:58.809995+0000","last_peered":"2024-03-18T01:35:58.809995+0000","last_clean":"2024-03-18T01:35:58.809995+0000","last_became_active":"2024-03-18T01:35:01.132542+0000","last_became_peered":"2024-03-18T01:35:01.132542+0000","last_unstale":"2024-03-18T01:35:58.809995+0000","last_undegraded":"2024-03-18T01:35:58.809995+0000","last_fullsized":"2024-03-18T01:35:58.809995+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:51:23.823409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801972+0000","last_change":"2024-03-18T01:34:55.648924+0000","last_active":"2024-03-18T01:35:58.801972+0000","last_peered":"2024-03-18T01:35:58.801972+0000","last_clean":"2024-03-18T01:35:58.801972+0000","last_became_active":"2024-03-18T01:34:55.648548+0000","last_became_peered":"2024-03-18T01:34:55.648548+0000","last_unstale":"2024-03-18T01:35:58.801972+0000","last_undegraded":"2024-03-18T01:35:58.801972+0000","last_fullsized":"2024-03-18T01:35:58.801972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:26:38.916856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.221812+0000","last_change":"2024-03-18T01:34:56.663142+0000","last_active":"2024-03-18T01:36:57.221812+0000","last_peered":"2024-03-18T01:36:57.221812+0000","last_clean":"2024-03-18T01:36:57.221812+0000","last_became_active":"2024-03-18T01:34:56.660256+0000","last_became_peered":"2024-03-18T01:34:56.660256+0000","last_unstale":"2024-03-18T01:36:57.221812+0000","last_undegraded":"2024-03-18T01:36:57.221812+0000","last_fullsized":"2024-03-18T01:36:57.221812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:36:59.503277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801730+0000","last_change":"2024-03-18T01:35:01.136186+0000","last_active":"2024-03-18T01:35:58.801730+0000","last_peered":"2024-03-18T01:35:58.801730+0000","last_clean":"2024-03-18T01:35:58.801730+0000","last_became_active":"2024-03-18T01:35:01.135803+0000","last_became_peered":"2024-03-18T01:35:01.135803+0000","last_unstale":"2024-03-18T01:35:58.801730+0000","last_undegraded":"2024-03-18T01:35:58.801730+0000","last_fullsized":"2024-03-18T01:35:58.801730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:20.027448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806474+0000","last_change":"2024-03-18T01:34:55.663090+0000","last_active":"2024-03-18T01:35:58.806474+0000","last_peered":"2024-03-18T01:35:58.806474+0000","last_clean":"2024-03-18T01:35:58.806474+0000","last_became_active":"2024-03-18T01:34:55.662774+0000","last_became_peered":"2024-03-18T01:34:55.662774+0000","last_unstale":"2024-03-18T01:35:58.806474+0000","last_undegraded":"2024-03-18T01:35:58.806474+0000","last_fullsized":"2024-03-18T01:35:58.806474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:30:35.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.178629+0000","last_change":"2024-03-18T01:34:56.639509+0000","last_active":"2024-03-18T01:36:57.178629+0000","last_peered":"2024-03-18T01:36:57.178629+0000","last_clean":"2024-03-18T01:36:57.178629+0000","last_became_active":"2024-03-18T01:34:56.639144+0000","last_became_peered":"2024-03-18T01:34:56.639144+0000","last_unstale":"2024-03-18T01:36:57.178629+0000","last_undegraded":"2024-03-18T01:36:57.178629+0000","last_fullsized":"2024-03-18T01:36:57.178629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:12:36.220793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.812801+0000","last_change":"2024-03-18T01:35:01.141762+0000","last_active":"2024-03-18T01:35:58.812801+0000","last_peered":"2024-03-18T01:35:58.812801+0000","last_clean":"2024-03-18T01:35:58.812801+0000","last_became_active":"2024-03-18T01:35:01.141219+0000","last_became_peered":"2024-03-18T01:35:01.141219+0000","last_unstale":"2024-03-18T01:35:58.812801+0000","last_undegraded":"2024-03-18T01:35:58.812801+0000","last_fullsized":"2024-03-18T01:35:58.812801+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:55.839120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.815926+0000","last_change":"2024-03-18T01:34:55.662530+0000","last_active":"2024-03-18T01:35:58.815926+0000","last_peered":"2024-03-18T01:35:58.815926+0000","last_clean":"2024-03-18T01:35:58.815926+0000","last_became_active":"2024-03-18T01:34:55.661178+0000","last_became_peered":"2024-03-18T01:34:55.661178+0000","last_unstale":"2024-03-18T01:35:58.815926+0000","last_undegraded":"2024-03-18T01:35:58.815926+0000","last_fullsized":"2024-03-18T01:35:58.815926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:47:16.678553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191228+0000","last_change":"2024-03-18T01:34:56.660936+0000","last_active":"2024-03-18T01:36:57.191228+0000","last_peered":"2024-03-18T01:36:57.191228+0000","last_clean":"2024-03-18T01:36:57.191228+0000","last_became_active":"2024-03-18T01:34:56.660513+0000","last_became_peered":"2024-03-18T01:34:56.660513+0000","last_unstale":"2024-03-18T01:36:57.191228+0000","last_undegraded":"2024-03-18T01:36:57.191228+0000","last_fullsized":"2024-03-18T01:36:57.191228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:35:21.037366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802056+0000","last_change":"2024-03-18T01:35:01.136914+0000","last_active":"2024-03-18T01:35:58.802056+0000","last_peered":"2024-03-18T01:35:58.802056+0000","last_clean":"2024-03-18T01:35:58.802056+0000","last_became_active":"2024-03-18T01:35:01.136480+0000","last_became_peered":"2024-03-18T01:35:01.136480+0000","last_unstale":"2024-03-18T01:35:58.802056+0000","last_undegraded":"2024-03-18T01:35:58.802056+0000","last_fullsized":"2024-03-18T01:35:58.802056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:49.262255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:37:05.892340+0000","last_change":"2024-03-18T01:34:55.654938+0000","last_active":"2024-03-18T01:37:05.892340+0000","last_peered":"2024-03-18T01:37:05.892340+0000","last_clean":"2024-03-18T01:37:05.892340+0000","last_became_active":"2024-03-18T01:34:55.654611+0000","last_became_peered":"2024-03-18T01:34:55.654611+0000","last_unstale":"2024-03-18T01:37:05.892340+0000","last_undegraded":"2024-03-18T01:37:05.892340+0000","last_fullsized":"2024-03-18T01:37:05.892340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:04:23.210205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.220083+0000","last_change":"2024-03-18T01:34:56.650731+0000","last_active":"2024-03-18T01:36:57.220083+0000","last_peered":"2024-03-18T01:36:57.220083+0000","last_clean":"2024-03-18T01:36:57.220083+0000","last_became_active":"2024-03-18T01:34:56.650363+0000","last_became_peered":"2024-03-18T01:34:56.650363+0000","last_unstale":"2024-03-18T01:36:57.220083+0000","last_undegraded":"2024-03-18T01:36:57.220083+0000","last_fullsized":"2024-03-18T01:36:57.220083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:09:50.305019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.801699+0000","last_change":"2024-03-18T01:35:01.135101+0000","last_active":"2024-03-18T01:35:58.801699+0000","last_peered":"2024-03-18T01:35:58.801699+0000","last_clean":"2024-03-18T01:35:58.801699+0000","last_became_active":"2024-03-18T01:35:01.134690+0000","last_became_peered":"2024-03-18T01:35:01.134690+0000","last_unstale":"2024-03-18T01:35:58.801699+0000","last_undegraded":"2024-03-18T01:35:58.801699+0000","last_fullsized":"2024-03-18T01:35:58.801699+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:33.381629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.802175+0000","last_change":"2024-03-18T01:34:55.648108+0000","last_active":"2024-03-18T01:35:58.802175+0000","last_peered":"2024-03-18T01:35:58.802175+0000","last_clean":"2024-03-18T01:35:58.802175+0000","last_became_active":"2024-03-18T01:34:55.647754+0000","last_became_peered":"2024-03-18T01:34:55.647754+0000","last_unstale":"2024-03-18T01:35:58.802175+0000","last_undegraded":"2024-03-18T01:35:58.802175+0000","last_fullsized":"2024-03-18T01:35:58.802175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:05:48.508015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203569+0000","last_change":"2024-03-18T01:34:56.651589+0000","last_active":"2024-03-18T01:36:57.203569+0000","last_peered":"2024-03-18T01:36:57.203569+0000","last_clean":"2024-03-18T01:36:57.203569+0000","last_became_active":"2024-03-18T01:34:56.650929+0000","last_became_peered":"2024-03-18T01:34:56.650929+0000","last_unstale":"2024-03-18T01:36:57.203569+0000","last_undegraded":"2024-03-18T01:36:57.203569+0000","last_fullsized":"2024-03-18T01:36:57.203569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:16:43.974882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806823+0000","last_change":"2024-03-18T01:35:01.148347+0000","last_active":"2024-03-18T01:35:58.806823+0000","last_peered":"2024-03-18T01:35:58.806823+0000","last_clean":"2024-03-18T01:35:58.806823+0000","last_became_active":"2024-03-18T01:35:01.147936+0000","last_became_peered":"2024-03-18T01:35:01.147936+0000","last_unstale":"2024-03-18T01:35:58.806823+0000","last_undegraded":"2024-03-18T01:35:58.806823+0000","last_fullsized":"2024-03-18T01:35:58.806823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:12.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'6","reported_seq":193,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:38:04.806161+0000","last_change":"2024-03-18T01:34:55.653625+0000","last_active":"2024-03-18T01:38:04.806161+0000","last_peered":"2024-03-18T01:38:04.806161+0000","last_clean":"2024-03-18T01:38:04.806161+0000","last_became_active":"2024-03-18T01:34:55.653186+0000","last_became_peered":"2024-03-18T01:34:55.653186+0000","last_unstale":"2024-03-18T01:38:04.806161+0000","last_undegraded":"2024-03-18T01:38:04.806161+0000","last_fullsized":"2024-03-18T01:38:04.806161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:10:51.193296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1480,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":159,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.203868+0000","last_change":"2024-03-18T01:34:56.652703+0000","last_active":"2024-03-18T01:36:57.203868+0000","last_peered":"2024-03-18T01:36:57.203868+0000","last_clean":"2024-03-18T01:36:57.203868+0000","last_became_active":"2024-03-18T01:34:56.651768+0000","last_became_peered":"2024-03-18T01:34:56.651768+0000","last_unstale":"2024-03-18T01:36:57.203868+0000","last_undegraded":"2024-03-18T01:36:57.203868+0000","last_fullsized":"2024-03-18T01:36:57.203868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:59.748758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.809369+0000","last_change":"2024-03-18T01:35:01.144900+0000","last_active":"2024-03-18T01:35:58.809369+0000","last_peered":"2024-03-18T01:35:58.809369+0000","last_clean":"2024-03-18T01:35:58.809369+0000","last_became_active":"2024-03-18T01:35:01.144431+0000","last_became_peered":"2024-03-18T01:35:01.144431+0000","last_unstale":"2024-03-18T01:35:58.809369+0000","last_undegraded":"2024-03-18T01:35:58.809369+0000","last_fullsized":"2024-03-18T01:35:58.809369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:36.199904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.808782+0000","last_change":"2024-03-18T01:34:55.661259+0000","last_active":"2024-03-18T01:35:58.808782+0000","last_peered":"2024-03-18T01:35:58.808782+0000","last_clean":"2024-03-18T01:35:58.808782+0000","last_became_active":"2024-03-18T01:34:55.660964+0000","last_became_peered":"2024-03-18T01:34:55.660964+0000","last_unstale":"2024-03-18T01:35:58.808782+0000","last_undegraded":"2024-03-18T01:35:58.808782+0000","last_fullsized":"2024-03-18T01:35:58.808782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:21:14.723905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213011+0000","last_change":"2024-03-18T01:34:56.663075+0000","last_active":"2024-03-18T01:36:57.213011+0000","last_peered":"2024-03-18T01:36:57.213011+0000","last_clean":"2024-03-18T01:36:57.213011+0000","last_became_active":"2024-03-18T01:34:56.662458+0000","last_became_peered":"2024-03-18T01:34:56.662458+0000","last_unstale":"2024-03-18T01:36:57.213011+0000","last_undegraded":"2024-03-18T01:36:57.213011+0000","last_fullsized":"2024-03-18T01:36:57.213011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:31:31.837493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.806874+0000","last_change":"2024-03-18T01:35:01.133317+0000","last_active":"2024-03-18T01:35:58.806874+0000","last_peered":"2024-03-18T01:35:58.806874+0000","last_clean":"2024-03-18T01:35:58.806874+0000","last_became_active":"2024-03-18T01:35:01.131798+0000","last_became_peered":"2024-03-18T01:35:01.131798+0000","last_unstale":"2024-03-18T01:35:58.806874+0000","last_undegraded":"2024-03-18T01:35:58.806874+0000","last_fullsized":"2024-03-18T01:35:58.806874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:35:00.088536+0000","last_clean_scrub_stamp":"2024-03-18T01:35:00.088536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:55.141519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:35:58.819510+0000","last_change":"2024-03-18T01:34:55.660887+0000","last_active":"2024-03-18T01:35:58.819510+0000","last_peered":"2024-03-18T01:35:58.819510+0000","last_clean":"2024-03-18T01:35:58.819510+0000","last_became_active":"2024-03-18T01:34:55.660524+0000","last_became_peered":"2024-03-18T01:34:55.660524+0000","last_unstale":"2024-03-18T01:35:58.819510+0000","last_undegraded":"2024-03-18T01:35:58.819510+0000","last_fullsized":"2024-03-18T01:35:58.819510+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:54.589991+0000","last_clean_scrub_stamp":"2024-03-18T01:34:54.589991+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:02:24.816951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.192183+0000","last_change":"2024-03-18T01:34:56.654107+0000","last_active":"2024-03-18T01:36:57.192183+0000","last_peered":"2024-03-18T01:36:57.192183+0000","last_clean":"2024-03-18T01:36:57.192183+0000","last_became_active":"2024-03-18T01:34:56.653742+0000","last_became_peered":"2024-03-18T01:34:56.653742+0000","last_unstale":"2024-03-18T01:36:57.192183+0000","last_undegraded":"2024-03-18T01:36:57.192183+0000","last_fullsized":"2024-03-18T01:36:57.192183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:58:46.093589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191185+0000","last_change":"2024-03-18T01:34:56.647818+0000","last_active":"2024-03-18T01:36:57.191185+0000","last_peered":"2024-03-18T01:36:57.191185+0000","last_clean":"2024-03-18T01:36:57.191185+0000","last_became_active":"2024-03-18T01:34:56.647174+0000","last_became_peered":"2024-03-18T01:34:56.647174+0000","last_unstale":"2024-03-18T01:36:57.191185+0000","last_undegraded":"2024-03-18T01:36:57.191185+0000","last_fullsized":"2024-03-18T01:36:57.191185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:55.656083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.191353+0000","last_change":"2024-03-18T01:34:56.643221+0000","last_active":"2024-03-18T01:36:57.191353+0000","last_peered":"2024-03-18T01:36:57.191353+0000","last_clean":"2024-03-18T01:36:57.191353+0000","last_became_active":"2024-03-18T01:34:56.642871+0000","last_became_peered":"2024-03-18T01:34:56.642871+0000","last_unstale":"2024-03-18T01:36:57.191353+0000","last_undegraded":"2024-03-18T01:36:57.191353+0000","last_fullsized":"2024-03-18T01:36:57.191353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:23:25.106060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.213467+0000","last_change":"2024-03-18T01:34:56.656227+0000","last_active":"2024-03-18T01:36:57.213467+0000","last_peered":"2024-03-18T01:36:57.213467+0000","last_clean":"2024-03-18T01:36:57.213467+0000","last_became_active":"2024-03-18T01:34:56.655859+0000","last_became_peered":"2024-03-18T01:34:56.655859+0000","last_unstale":"2024-03-18T01:36:57.213467+0000","last_undegraded":"2024-03-18T01:36:57.213467+0000","last_fullsized":"2024-03-18T01:36:57.213467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:51:54.424342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.215195+0000","last_change":"2024-03-18T01:34:56.664655+0000","last_active":"2024-03-18T01:36:57.215195+0000","last_peered":"2024-03-18T01:36:57.215195+0000","last_clean":"2024-03-18T01:36:57.215195+0000","last_became_active":"2024-03-18T01:34:56.662526+0000","last_became_peered":"2024-03-18T01:34:56.662526+0000","last_unstale":"2024-03-18T01:36:57.215195+0000","last_undegraded":"2024-03-18T01:36:57.215195+0000","last_fullsized":"2024-03-18T01:36:57.215195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:45:06.158576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.214796+0000","last_change":"2024-03-18T01:34:56.657045+0000","last_active":"2024-03-18T01:36:57.214796+0000","last_peered":"2024-03-18T01:36:57.214796+0000","last_clean":"2024-03-18T01:36:57.214796+0000","last_became_active":"2024-03-18T01:34:56.656339+0000","last_became_peered":"2024-03-18T01:34:56.656339+0000","last_unstale":"2024-03-18T01:36:57.214796+0000","last_undegraded":"2024-03-18T01:36:57.214796+0000","last_fullsized":"2024-03-18T01:36:57.214796+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:41.412462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.184826+0000","last_change":"2024-03-18T01:34:56.647626+0000","last_active":"2024-03-18T01:36:57.184826+0000","last_peered":"2024-03-18T01:36:57.184826+0000","last_clean":"2024-03-18T01:36:57.184826+0000","last_became_active":"2024-03-18T01:34:56.647216+0000","last_became_peered":"2024-03-18T01:34:56.647216+0000","last_unstale":"2024-03-18T01:36:57.184826+0000","last_undegraded":"2024-03-18T01:36:57.184826+0000","last_fullsized":"2024-03-18T01:36:57.184826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:12:33.330925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.180192+0000","last_change":"2024-03-18T01:34:56.645091+0000","last_active":"2024-03-18T01:36:57.180192+0000","last_peered":"2024-03-18T01:36:57.180192+0000","last_clean":"2024-03-18T01:36:57.180192+0000","last_became_active":"2024-03-18T01:34:56.644725+0000","last_became_peered":"2024-03-18T01:34:56.644725+0000","last_unstale":"2024-03-18T01:36:57.180192+0000","last_undegraded":"2024-03-18T01:36:57.180192+0000","last_fullsized":"2024-03-18T01:36:57.180192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:17.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211414+0000","last_change":"2024-03-18T01:34:56.651282+0000","last_active":"2024-03-18T01:36:57.211414+0000","last_peered":"2024-03-18T01:36:57.211414+0000","last_clean":"2024-03-18T01:36:57.211414+0000","last_became_active":"2024-03-18T01:34:56.650872+0000","last_became_peered":"2024-03-18T01:34:56.650872+0000","last_unstale":"2024-03-18T01:36:57.211414+0000","last_undegraded":"2024-03-18T01:36:57.211414+0000","last_fullsized":"2024-03-18T01:36:57.211414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T13:14:45.106848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211139+0000","last_change":"2024-03-18T01:34:56.651347+0000","last_active":"2024-03-18T01:36:57.211139+0000","last_peered":"2024-03-18T01:36:57.211139+0000","last_clean":"2024-03-18T01:36:57.211139+0000","last_became_active":"2024-03-18T01:34:56.650935+0000","last_became_peered":"2024-03-18T01:34:56.650935+0000","last_unstale":"2024-03-18T01:36:57.211139+0000","last_undegraded":"2024-03-18T01:36:57.211139+0000","last_fullsized":"2024-03-18T01:36:57.211139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:38:41.849883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.211642+0000","last_change":"2024-03-18T01:34:56.667333+0000","last_active":"2024-03-18T01:36:57.211642+0000","last_peered":"2024-03-18T01:36:57.211642+0000","last_clean":"2024-03-18T01:36:57.211642+0000","last_became_active":"2024-03-18T01:34:56.666845+0000","last_became_peered":"2024-03-18T01:34:56.666845+0000","last_unstale":"2024-03-18T01:36:57.211642+0000","last_undegraded":"2024-03-18T01:36:57.211642+0000","last_fullsized":"2024-03-18T01:36:57.211642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:49:23.813455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.217436+0000","last_change":"2024-03-18T01:34:56.662677+0000","last_active":"2024-03-18T01:36:57.217436+0000","last_peered":"2024-03-18T01:36:57.217436+0000","last_clean":"2024-03-18T01:36:57.217436+0000","last_became_active":"2024-03-18T01:34:56.662306+0000","last_became_peered":"2024-03-18T01:34:56.662306+0000","last_unstale":"2024-03-18T01:36:57.217436+0000","last_undegraded":"2024-03-18T01:36:57.217436+0000","last_fullsized":"2024-03-18T01:36:57.217436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:53:56.868984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.216916+0000","last_change":"2024-03-18T01:34:56.657721+0000","last_active":"2024-03-18T01:36:57.216916+0000","last_peered":"2024-03-18T01:36:57.216916+0000","last_clean":"2024-03-18T01:36:57.216916+0000","last_became_active":"2024-03-18T01:34:56.657072+0000","last_became_peered":"2024-03-18T01:34:56.657072+0000","last_unstale":"2024-03-18T01:36:57.216916+0000","last_undegraded":"2024-03-18T01:36:57.216916+0000","last_fullsized":"2024-03-18T01:36:57.216916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:21.120391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T01:36:57.190992+0000","last_change":"2024-03-18T01:34:56.647327+0000","last_active":"2024-03-18T01:36:57.190992+0000","last_peered":"2024-03-18T01:36:57.190992+0000","last_clean":"2024-03-18T01:36:57.190992+0000","last_became_active":"2024-03-18T01:34:56.646910+0000","last_became_peered":"2024-03-18T01:34:56.646910+0000","last_unstale":"2024-03-18T01:36:57.190992+0000","last_undegraded":"2024-03-18T01:36:57.190992+0000","last_fullsized":"2024-03-18T01:36:57.190992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T01:34:55.619454+0000","last_clean_scrub_stamp":"2024-03-18T01:34:55.619454+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:37:25.970272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74209,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1577,"num_read_kb":866,"num_write":257,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":650,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":222627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383732,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":502393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645368,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1155726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808894,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":505459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070534,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1048576,"data_stored":501159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332171,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1098379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495696,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":496867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1105920,"data_stored":564842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":498633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":565304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182496,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":496575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920859,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":501473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757335,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1093429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T01:38:07.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:38:07.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.872+0000 7fe5fb200640 1 -- 172.21.15.7:0/498918549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 msgr2=0x7fe5fc101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.872+0000 7fe5fb200640 1 --2- 172.21.15.7:0/498918549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 0x7fe5fc101a40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec009ca0 tx=0x7fe5ec02c100 comp rx=0 tx=0).stop 2024-03-18T01:38:07.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.872+0000 7fe5fb200640 1 -- 172.21.15.7:0/498918549 shutdown_connections 2024-03-18T01:38:07.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.872+0000 7fe5fb200640 1 --2- 172.21.15.7:0/498918549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5fc1031d0 0x7fe5fc109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.872+0000 7fe5fb200640 1 --2- 172.21.15.7:0/498918549 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5fc102810 0x7fe5fc102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.872+0000 7fe5fb200640 1 --2- 172.21.15.7:0/498918549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 0x7fe5fc101a40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.873+0000 7fe5fb200640 1 -- 172.21.15.7:0/498918549 >> 172.21.15.7:0/498918549 conn(0x7fe5fc0fcb90 msgr2=0x7fe5fc0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:07.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.873+0000 7fe5fb200640 1 -- 172.21.15.7:0/498918549 shutdown_connections 2024-03-18T01:38:07.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.873+0000 7fe5fb200640 1 -- 172.21.15.7:0/498918549 wait complete. 2024-03-18T01:38:07.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.874+0000 7fe5fb200640 1 Processor -- start 2024-03-18T01:38:07.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.874+0000 7fe5fb200640 1 -- start start 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.874+0000 7fe5fb200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 0x7fe5fc199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5fb200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5fc102810 0x7fe5fc199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5fb200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5fc1031d0 0x7fe5fc1a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5fb200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc10d290 con 0x7fe5fc102810 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5fb200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc10d110 con 0x7fe5fc1031d0 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5fb200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc10d410 con 0x7fe5fc101620 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 0x7fe5fc199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5fc102810 0x7fe5fc199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5fa800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5fc1031d0 0x7fe5fc1a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 0x7fe5fc199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44610/0 (socket says 172.21.15.7:44610) 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5fa800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5fc1031d0 0x7fe5fc1a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:42348/0 (socket says 172.21.15.7:42348) 2024-03-18T01:38:07.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9e00640 1 -- 172.21.15.7:0/4196878941 learned_addr learned my addr 172.21.15.7:0/4196878941 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:07.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5fc102810 0x7fe5fc199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60054/0 (socket says 172.21.15.7:60054) 2024-03-18T01:38:07.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9e00640 1 -- 172.21.15.7:0/4196878941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5fc1031d0 msgr2=0x7fe5fc1a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9e00640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5fc1031d0 0x7fe5fc1a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9e00640 1 -- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5fc102810 msgr2=0x7fe5fc199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:07.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.875+0000 7fe5f9e00640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5fc102810 0x7fe5fc199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:07.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.876+0000 7fe5f9e00640 1 -- 172.21.15.7:0/4196878941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5fc1a1680 con 0x7fe5fc101620 2024-03-18T01:38:07.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.876+0000 7fe5fa800640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5fc1031d0 0x7fe5fc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:07.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.876+0000 7fe5f9e00640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 0x7fe5fc199a60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0324c0 tx=0x7fe5ec035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.876+0000 7fe5f2a00640 1 -- 172.21.15.7:0/4196878941 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec039040 con 0x7fe5fc101620 2024-03-18T01:38:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.876+0000 7fe5f2a00640 1 -- 172.21.15.7:0/4196878941 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5ec044070 con 0x7fe5fc101620 2024-03-18T01:38:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.876+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5fc1a1910 con 0x7fe5fc101620 2024-03-18T01:38:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.876+0000 7fe5f2a00640 1 -- 172.21.15.7:0/4196878941 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec0044a0 con 0x7fe5fc101620 2024-03-18T01:38:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.876+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5fc1a1e50 con 0x7fe5fc101620 2024-03-18T01:38:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.878+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5fc1a2080 con 0x7fe5fc101620 2024-03-18T01:38:07.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.879+0000 7fe5f2a00640 1 -- 172.21.15.7:0/4196878941 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fe5ec004640 con 0x7fe5fc101620 2024-03-18T01:38:07.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.881+0000 7fe5f2a00640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe5e00a2920 0x7fe5e00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:07.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.881+0000 7fe5f2a00640 1 -- 172.21.15.7:0/4196878941 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe5ec0f1340 con 0x7fe5fc101620 2024-03-18T01:38:07.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.881+0000 7fe5f9400640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe5e00a2920 0x7fe5e00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:07.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.882+0000 7fe5f9400640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe5e00a2920 0x7fe5e00a4de0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc19af70 tx=0x7fe5e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:07.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:07.889+0000 7fe5f2a00640 1 -- 172.21.15.7:0/4196878941 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe5ec0be020 con 0x7fe5fc101620 2024-03-18T01:38:08.018 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:07 smithi046 ceph-mon[36108]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:07 smithi046 ceph-mon[36108]: from='client.25538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:08.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:07 smithi007 ceph-mon[31996]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:08.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:07 smithi007 ceph-mon[31996]: from='client.25538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:08.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.092+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5fc1a26c0 con 0x7fe5fc101620 2024-03-18T01:38:08.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.094+0000 7fe5f2a00640 1 -- 172.21.15.7:0/4196878941 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7fe5ec0bd920 con 0x7fe5fc101620 2024-03-18T01:38:08.094 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:08.094 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:38:08.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe5e00a2920 msgr2=0x7fe5e00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe5e00a2920 0x7fe5e00a4de0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc19af70 tx=0x7fe5e8005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 msgr2=0x7fe5fc199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 0x7fe5fc199a60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec0324c0 tx=0x7fe5ec035d90 comp rx=0 tx=0).stop 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 shutdown_connections 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5fc1031d0 0x7fe5fc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe5e00a2920 0x7fe5e00a4de0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe5fc102810 0x7fe5fc199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 --2- 172.21.15.7:0/4196878941 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5fc101620 0x7fe5fc199a60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.096+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 >> 172.21.15.7:0/4196878941 conn(0x7fe5fc0fcb90 msgr2=0x7fe5fc073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.097+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 shutdown_connections 2024-03-18T01:38:08.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.097+0000 7fe5fb200640 1 -- 172.21.15.7:0/4196878941 wait complete. 2024-03-18T01:38:08.127 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:38:08.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:38:08.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:07 smithi203 ceph-mon[36345]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:08.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:07 smithi203 ceph-mon[36345]: from='client.25538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 -- 172.21.15.7:0/2117980391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b8c104c30 msgr2=0x7f6b8c107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 --2- 172.21.15.7:0/2117980391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b8c104c30 0x7f6b8c107020 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009900 tx=0x7f6b7802bc30 comp rx=0 tx=0).stop 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 -- 172.21.15.7:0/2117980391 shutdown_connections 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 --2- 172.21.15.7:0/2117980391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b8c107560 0x7f6b8c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 --2- 172.21.15.7:0/2117980391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b8c104c30 0x7f6b8c107020 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 --2- 172.21.15.7:0/2117980391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b8c102300 0x7f6b8c1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 -- 172.21.15.7:0/2117980391 >> 172.21.15.7:0/2117980391 conn(0x7f6b8c0fbc70 msgr2=0x7f6b8c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 -- 172.21.15.7:0/2117980391 shutdown_connections 2024-03-18T01:38:08.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.369+0000 7f6b94400640 1 -- 172.21.15.7:0/2117980391 wait complete. 2024-03-18T01:38:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.370+0000 7f6b94400640 1 Processor -- start 2024-03-18T01:38:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b94400640 1 -- start start 2024-03-18T01:38:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b94400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b8c102300 0x7f6b8c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b94400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b8c104c30 0x7f6b8c19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b94400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b8c107560 0x7f6b8c1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:08.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b94400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c10d4d0 con 0x7f6b8c104c30 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b94400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c10d350 con 0x7f6b8c107560 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b94400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c10d650 con 0x7f6b8c102300 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b91a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b8c102300 0x7f6b8c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b91000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b8c104c30 0x7f6b8c19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b91a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b8c102300 0x7f6b8c199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44630/0 (socket says 172.21.15.7:44630) 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b91a00640 1 -- 172.21.15.7:0/3124760939 learned_addr learned my addr 172.21.15.7:0/3124760939 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.371+0000 7f6b92400640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b8c107560 0x7f6b8c1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.372+0000 7f6b91a00640 1 -- 172.21.15.7:0/3124760939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b8c107560 msgr2=0x7f6b8c1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.372+0000 7f6b91a00640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b8c107560 0x7f6b8c1a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.372+0000 7f6b91a00640 1 -- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b8c104c30 msgr2=0x7f6b8c19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.372+0000 7f6b91a00640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b8c104c30 0x7f6b8c19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.372+0000 7f6b91a00640 1 -- 172.21.15.7:0/3124760939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c1a1980 con 0x7f6b8c102300 2024-03-18T01:38:08.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.372+0000 7f6b91000640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b8c104c30 0x7f6b8c19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.372+0000 7f6b91a00640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b8c102300 0x7f6b8c199d60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00ebc0 tx=0x7f6b7c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.373+0000 7f6b8a000640 1 -- 172.21.15.7:0/3124760939 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c010630 con 0x7f6b8c102300 2024-03-18T01:38:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.373+0000 7f6b8a000640 1 -- 172.21.15.7:0/3124760939 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b7c010cb0 con 0x7f6b8c102300 2024-03-18T01:38:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.373+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8c1a1c70 con 0x7f6b8c102300 2024-03-18T01:38:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.373+0000 7f6b8a000640 1 -- 172.21.15.7:0/3124760939 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c018bd0 con 0x7f6b8c102300 2024-03-18T01:38:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.373+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8c1a2190 con 0x7f6b8c102300 2024-03-18T01:38:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.374+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b8c109050 con 0x7f6b8c102300 2024-03-18T01:38:08.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.376+0000 7f6b8a000640 1 -- 172.21.15.7:0/3124760939 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f6b7c0107d0 con 0x7f6b8c102300 2024-03-18T01:38:08.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.378+0000 7f6b8a000640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6b700a2880 0x7f6b700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:08.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.378+0000 7f6b8a000640 1 -- 172.21.15.7:0/3124760939 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6b7c0cf360 con 0x7f6b8c102300 2024-03-18T01:38:08.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.378+0000 7f6b91000640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6b700a2880 0x7f6b700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:08.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.379+0000 7f6b91000640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6b700a2880 0x7f6b700a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b7802c170 tx=0x7f6b7802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:08.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.380+0000 7f6b8a000640 1 -- 172.21.15.7:0/3124760939 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b7c09b9f0 con 0x7f6b8c102300 2024-03-18T01:38:08.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.605+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6b8c02d420 con 0x7f6b8c102300 2024-03-18T01:38:08.606 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.606+0000 7f6b8a000640 1 -- 172.21.15.7:0/3124760939 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f6b7c09fa90 con 0x7f6b8c102300 2024-03-18T01:38:08.606 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:38:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.609+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6b700a2880 msgr2=0x7f6b700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.609+0000 7f6b94400640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6b700a2880 0x7f6b700a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b7802c170 tx=0x7f6b7802df90 comp rx=0 tx=0).stop 2024-03-18T01:38:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.609+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b8c102300 msgr2=0x7f6b8c199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b8c102300 0x7f6b8c199d60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00ebc0 tx=0x7f6b7c00c6a0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 shutdown_connections 2024-03-18T01:38:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b8c107560 0x7f6b8c1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6b700a2880 0x7f6b700a4d40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6b8c104c30 0x7f6b8c19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 --2- 172.21.15.7:0/3124760939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b8c102300 0x7f6b8c199d60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 >> 172.21.15.7:0/3124760939 conn(0x7f6b8c0fbc70 msgr2=0x7f6b8c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 shutdown_connections 2024-03-18T01:38:08.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.610+0000 7f6b94400640 1 -- 172.21.15.7:0/3124760939 wait complete. 2024-03-18T01:38:08.632 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:38:08.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 -- 172.21.15.7:0/2343533889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884107580 msgr2=0x7f5884109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 --2- 172.21.15.7:0/2343533889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884107580 0x7f5884109970 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f587c009ca0 tx=0x7f587c02c100 comp rx=0 tx=0).stop 2024-03-18T01:38:08.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 -- 172.21.15.7:0/2343533889 shutdown_connections 2024-03-18T01:38:08.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 --2- 172.21.15.7:0/2343533889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884107580 0x7f5884109970 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 --2- 172.21.15.7:0/2343533889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5884104c50 0x7f5884107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 --2- 172.21.15.7:0/2343533889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5884102320 0x7f5884104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 -- 172.21.15.7:0/2343533889 >> 172.21.15.7:0/2343533889 conn(0x7f58840fbc70 msgr2=0x7f58840fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 -- 172.21.15.7:0/2343533889 shutdown_connections 2024-03-18T01:38:08.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.857+0000 7f588a200640 1 -- 172.21.15.7:0/2343533889 wait complete. 2024-03-18T01:38:08.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.858+0000 7f588a200640 1 Processor -- start 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.858+0000 7f588a200640 1 -- start start 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f588a200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5884102320 0x7f5884101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f588a200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884104c50 0x7f58841002d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f588a200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5884107580 0x7f5884100810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f588a200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f588410d4a0 con 0x7f5884102320 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f588a200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f588410d320 con 0x7f5884107580 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f588a200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f588410d620 con 0x7f5884104c50 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5882a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884104c50 0x7f58841002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5882a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884104c50 0x7f58841002d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44640/0 (socket says 172.21.15.7:44640) 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5882a00640 1 -- 172.21.15.7:0/4259720618 learned_addr learned my addr 172.21.15.7:0/4259720618 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5883e00640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5884107580 0x7f5884100810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5883400640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5884102320 0x7f5884101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:08.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5882a00640 1 -- 172.21.15.7:0/4259720618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5884107580 msgr2=0x7f5884100810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5882a00640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5884107580 0x7f5884100810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5882a00640 1 -- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5884102320 msgr2=0x7f5884101c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:08.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5882a00640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5884102320 0x7f5884101c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:08.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.859+0000 7f5882a00640 1 -- 172.21.15.7:0/4259720618 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58841010a0 con 0x7f5884104c50 2024-03-18T01:38:08.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.860+0000 7f5883e00640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5884107580 0x7f5884100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:08.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.860+0000 7f5882a00640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884104c50 0x7f58841002d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f587000e9c0 tx=0x7f587000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.860+0000 7f587be00640 1 -- 172.21.15.7:0/4259720618 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587000cd80 con 0x7f5884104c50 2024-03-18T01:38:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.860+0000 7f587be00640 1 -- 172.21.15.7:0/4259720618 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5870005500 con 0x7f5884104c50 2024-03-18T01:38:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.860+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58841a1c30 con 0x7f5884104c50 2024-03-18T01:38:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.860+0000 7f587be00640 1 -- 172.21.15.7:0/4259720618 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5870010690 con 0x7f5884104c50 2024-03-18T01:38:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.860+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58841a2180 con 0x7f5884104c50 2024-03-18T01:38:08.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.861+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58840fd760 con 0x7f5884104c50 2024-03-18T01:38:08.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.864+0000 7f587be00640 1 -- 172.21.15.7:0/4259720618 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f58700046c0 con 0x7f5884104c50 2024-03-18T01:38:08.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.865+0000 7f587be00640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58680a2800 0x7f58680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:08.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.865+0000 7f587be00640 1 -- 172.21.15.7:0/4259720618 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f58700cf250 con 0x7f5884104c50 2024-03-18T01:38:08.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.865+0000 7f5883400640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58680a2800 0x7f58680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:08.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.866+0000 7f5883400640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58680a2800 0x7f58680a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5874009bd0 tx=0x7f5874005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:08.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:08.873+0000 7f587be00640 1 -- 172.21.15.7:0/4259720618 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f587009bf30 con 0x7f5884104c50 2024-03-18T01:38:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:08 smithi046 ceph-mon[36108]: pgmap v416: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:08 smithi046 ceph-mon[36108]: from='client.25537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:08 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/4196878941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:08 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3124760939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:08 smithi007 ceph-mon[31996]: pgmap v416: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:08 smithi007 ceph-mon[31996]: from='client.25537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:08 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4196878941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:09.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:08 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3124760939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:09.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.078+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f58840623e0 con 0x7f5884104c50 2024-03-18T01:38:09.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.079+0000 7f587be00640 1 -- 172.21.15.7:0/4259720618 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f587009b830 con 0x7f5884104c50 2024-03-18T01:38:09.079 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:38:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58680a2800 msgr2=0x7f58680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58680a2800 0x7f58680a4cc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5874009bd0 tx=0x7f5874005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884104c50 msgr2=0x7f58841002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884104c50 0x7f58841002d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f587000e9c0 tx=0x7f587000ee80 comp rx=0 tx=0).stop 2024-03-18T01:38:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 shutdown_connections 2024-03-18T01:38:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5884107580 0x7f5884100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5884104c50 0x7f58841002d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f58680a2800 0x7f58680a4cc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 --2- 172.21.15.7:0/4259720618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5884102320 0x7f5884101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.081+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 >> 172.21.15.7:0/4259720618 conn(0x7f58840fbc70 msgr2=0x7f5884102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.082+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 shutdown_connections 2024-03-18T01:38:09.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.082+0000 7f588a200640 1 -- 172.21.15.7:0/4259720618 wait complete. 2024-03-18T01:38:09.102 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:38:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:08 smithi203 ceph-mon[36345]: pgmap v416: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:08 smithi203 ceph-mon[36345]: from='client.25537 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T01:38:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:08 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4196878941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:08 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3124760939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:38:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.323+0000 7f9327200640 1 -- 172.21.15.7:0/1707059890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 msgr2=0x7f9328101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.323+0000 7f9327200640 1 --2- 172.21.15.7:0/1707059890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 0x7f9328101bc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9310009900 tx=0x7f931002bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.323+0000 7f9327200640 1 -- 172.21.15.7:0/1707059890 shutdown_connections 2024-03-18T01:38:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.323+0000 7f9327200640 1 --2- 172.21.15.7:0/1707059890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9328102100 0x7f9328108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.323+0000 7f9327200640 1 --2- 172.21.15.7:0/1707059890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 0x7f9328101bc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.323+0000 7f9327200640 1 --2- 172.21.15.7:0/1707059890 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9328100550 0x7f9328100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.323+0000 7f9327200640 1 -- 172.21.15.7:0/1707059890 >> 172.21.15.7:0/1707059890 conn(0x7f93280fbac0 msgr2=0x7f93280fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.323+0000 7f9327200640 1 -- 172.21.15.7:0/1707059890 shutdown_connections 2024-03-18T01:38:09.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.324+0000 7f9327200640 1 -- 172.21.15.7:0/1707059890 wait complete. 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.324+0000 7f9327200640 1 Processor -- start 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9327200640 1 -- start start 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9327200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9328100550 0x7f9328199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9327200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 0x7f932819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9327200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9328102100 0x7f93281a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9327200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f932810c230 con 0x7f9328100550 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9327200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f932810c0b0 con 0x7f9328102100 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9327200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f932810c3b0 con 0x7f9328101740 2024-03-18T01:38:09.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f931fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 0x7f932819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9326800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9328102100 0x7f93281a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f9325e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9328100550 0x7f9328199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f931fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 0x7f932819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44662/0 (socket says 172.21.15.7:44662) 2024-03-18T01:38:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.325+0000 7f931fe00640 1 -- 172.21.15.7:0/2054693807 learned_addr learned my addr 172.21.15.7:0/2054693807 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f9326800640 1 -- 172.21.15.7:0/2054693807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 msgr2=0x7f932819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f9326800640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 0x7f932819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f9326800640 1 -- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9328100550 msgr2=0x7f9328199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f9326800640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9328100550 0x7f9328199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f9326800640 1 -- 172.21.15.7:0/2054693807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93281a17f0 con 0x7f9328102100 2024-03-18T01:38:09.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f931fe00640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 0x7f932819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f9326800640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9328102100 0x7f93281a1180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f932000ec50 tx=0x7f932000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f931ea00640 1 -- 172.21.15.7:0/2054693807 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9320003b70 con 0x7f9328102100 2024-03-18T01:38:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93281a1ae0 con 0x7f9328102100 2024-03-18T01:38:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f931ea00640 1 -- 172.21.15.7:0/2054693807 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9320003d10 con 0x7f9328102100 2024-03-18T01:38:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.326+0000 7f931ea00640 1 -- 172.21.15.7:0/2054693807 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9320010890 con 0x7f9328102100 2024-03-18T01:38:09.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.327+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93281a1fe0 con 0x7f9328102100 2024-03-18T01:38:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.328+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93281a22d0 con 0x7f9328102100 2024-03-18T01:38:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.329+0000 7f931ea00640 1 -- 172.21.15.7:0/2054693807 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9320003fe0 con 0x7f9328102100 2024-03-18T01:38:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.331+0000 7f931ea00640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f930c0a2800 0x7f930c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.331+0000 7f931ea00640 1 -- 172.21.15.7:0/2054693807 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f93200cf3a0 con 0x7f9328102100 2024-03-18T01:38:09.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.331+0000 7f9325e00640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f930c0a2800 0x7f930c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:09.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.332+0000 7f9325e00640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f930c0a2800 0x7f930c0a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f93281015a0 tx=0x7f9318005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:09.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.335+0000 7f931ea00640 1 -- 172.21.15.7:0/2054693807 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f932009c080 con 0x7f9328102100 2024-03-18T01:38:09.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.538+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f93281a24a0 con 0x7f9328102100 2024-03-18T01:38:09.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.539+0000 7f931ea00640 1 -- 172.21.15.7:0/2054693807 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f932009b980 con 0x7f9328102100 2024-03-18T01:38:09.539 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:38:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f930c0a2800 msgr2=0x7f930c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f930c0a2800 0x7f930c0a4cc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f93281015a0 tx=0x7f9318005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9328102100 msgr2=0x7f93281a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9328102100 0x7f93281a1180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f932000ec50 tx=0x7f932000c6a0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 shutdown_connections 2024-03-18T01:38:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9328102100 0x7f93281a1180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9328101740 0x7f932819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f930c0a2800 0x7f930c0a4cc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 --2- 172.21.15.7:0/2054693807 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9328100550 0x7f9328199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.541+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 >> 172.21.15.7:0/2054693807 conn(0x7f93280fbac0 msgr2=0x7f93280fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.542+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 shutdown_connections 2024-03-18T01:38:09.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.542+0000 7f9327200640 1 -- 172.21.15.7:0/2054693807 wait complete. 2024-03-18T01:38:09.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 -- 172.21.15.7:0/3561346924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d01033e0 msgr2=0x7f84d0109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 --2- 172.21.15.7:0/3561346924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d01033e0 0x7f84d0109c70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009ca0 tx=0x7f84c802c100 comp rx=0 tx=0).stop 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 -- 172.21.15.7:0/3561346924 shutdown_connections 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 --2- 172.21.15.7:0/3561346924 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d01033e0 0x7f84d0109c70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 --2- 172.21.15.7:0/3561346924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d0102a20 0x7f84d0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 --2- 172.21.15.7:0/3561346924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d0101830 0x7f84d0101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 -- 172.21.15.7:0/3561346924 >> 172.21.15.7:0/3561346924 conn(0x7f84d00fcda0 msgr2=0x7f84d00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 -- 172.21.15.7:0/3561346924 shutdown_connections 2024-03-18T01:38:09.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.796+0000 7f84d7800640 1 -- 172.21.15.7:0/3561346924 wait complete. 2024-03-18T01:38:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.797+0000 7f84d7800640 1 Processor -- start 2024-03-18T01:38:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d7800640 1 -- start start 2024-03-18T01:38:09.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d7800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d0101830 0x7f84d0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d7800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d0102a20 0x7f84d0198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d7800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d01033e0 0x7f84d019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d7800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f84d010d4b0 con 0x7f84d01033e0 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d7800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84d010d330 con 0x7f84d0102a20 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d7800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f84d010d630 con 0x7f84d0101830 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d4e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d0101830 0x7f84d0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d5800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d01033e0 0x7f84d019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84c7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d0102a20 0x7f84d0198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d4e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d0101830 0x7f84d0197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44678/0 (socket says 172.21.15.7:44678) 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.798+0000 7f84d4e00640 1 -- 172.21.15.7:0/3485861353 learned_addr learned my addr 172.21.15.7:0/3485861353 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84c7e00640 1 -- 172.21.15.7:0/3485861353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d0101830 msgr2=0x7f84d0197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84c7e00640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d0101830 0x7f84d0197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84c7e00640 1 -- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d01033e0 msgr2=0x7f84d019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84c7e00640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d01033e0 0x7f84d019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84c7e00640 1 -- 172.21.15.7:0/3485861353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d019f6f0 con 0x7f84d0102a20 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84c7e00640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d0102a20 0x7f84d0198010 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00e9f0 tx=0x7f84bc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84ce000640 1 -- 172.21.15.7:0/3485861353 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc00ccc0 con 0x7f84d0102a20 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d019f9e0 con 0x7f84d0102a20 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84ce000640 1 -- 172.21.15.7:0/3485861353 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84bc005020 con 0x7f84d0102a20 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84ce000640 1 -- 172.21.15.7:0/3485861353 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0049e0 con 0x7f84d0102a20 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.799+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d019fee0 con 0x7f84d0102a20 2024-03-18T01:38:09.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.801+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d0108af0 con 0x7f84d0102a20 2024-03-18T01:38:09.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.803+0000 7f84ce000640 1 -- 172.21.15.7:0/3485861353 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f84bc005560 con 0x7f84d0102a20 2024-03-18T01:38:09.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.804+0000 7f84ce000640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84b40a2800 0x7f84b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:09.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.805+0000 7f84ce000640 1 -- 172.21.15.7:0/3485861353 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f84bc0cedc0 con 0x7f84d0102a20 2024-03-18T01:38:09.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.805+0000 7f84d4e00640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84b40a2800 0x7f84b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:09.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.806+0000 7f84d4e00640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84b40a2800 0x7f84b40a4cc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f84c00058c0 tx=0x7f84c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:09.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:09.808+0000 7f84ce000640 1 -- 172.21.15.7:0/3485861353 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f84bc0051d0 con 0x7f84d0102a20 2024-03-18T01:38:10.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.030+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f84d00623e0 con 0x7f84d0102a20 2024-03-18T01:38:10.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.030+0000 7f84ce000640 1 -- 172.21.15.7:0/3485861353 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f84bc09baa0 con 0x7f84d0102a20 2024-03-18T01:38:10.031 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84b40a2800 msgr2=0x7f84b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84b40a2800 0x7f84b40a4cc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f84c00058c0 tx=0x7f84c0005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d0102a20 msgr2=0x7f84d0198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d0102a20 0x7f84d0198010 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00e9f0 tx=0x7f84bc00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 shutdown_connections 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84b40a2800 0x7f84b40a4cc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84d01033e0 0x7f84d019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d0102a20 0x7f84d0198010 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 --2- 172.21.15.7:0/3485861353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84d0101830 0x7f84d0197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.033+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 >> 172.21.15.7:0/3485861353 conn(0x7f84d00fcda0 msgr2=0x7f84d0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.034+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 shutdown_connections 2024-03-18T01:38:10.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.034+0000 7f84d7800640 1 -- 172.21.15.7:0/3485861353 wait complete. 2024-03-18T01:38:10.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:09 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4259720618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:10.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:09 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2054693807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:10.050 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:09 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4259720618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:10.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:09 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2054693807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:10.053 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:38:10.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:09 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/4259720618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:38:10.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:09 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2054693807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:38:10.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.270+0000 7f27d5e00640 1 -- 172.21.15.7:0/697651708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d00ffee0 msgr2=0x7f27d0100300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.270+0000 7f27d5e00640 1 --2- 172.21.15.7:0/697651708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d00ffee0 0x7f27d0100300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009900 tx=0x7f27c402bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:10.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.271+0000 7f27d5e00640 1 -- 172.21.15.7:0/697651708 shutdown_connections 2024-03-18T01:38:10.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.271+0000 7f27d5e00640 1 --2- 172.21.15.7:0/697651708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27d01011e0 0x7f27d0101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.271+0000 7f27d5e00640 1 --2- 172.21.15.7:0/697651708 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27d0100840 0x7f27d0100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.271+0000 7f27d5e00640 1 --2- 172.21.15.7:0/697651708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d00ffee0 0x7f27d0100300 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.271+0000 7f27d5e00640 1 -- 172.21.15.7:0/697651708 >> 172.21.15.7:0/697651708 conn(0x7f27d00fbaa0 msgr2=0x7f27d00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:10.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.271+0000 7f27d5e00640 1 -- 172.21.15.7:0/697651708 shutdown_connections 2024-03-18T01:38:10.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.271+0000 7f27d5e00640 1 -- 172.21.15.7:0/697651708 wait complete. 2024-03-18T01:38:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.272+0000 7f27d5e00640 1 Processor -- start 2024-03-18T01:38:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.272+0000 7f27d5e00640 1 -- start start 2024-03-18T01:38:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.272+0000 7f27d5e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27d00ffee0 0x7f27d01999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.272+0000 7f27d5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27d0100840 0x7f27d0199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.272+0000 7f27d5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d01011e0 0x7f27d01a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.272+0000 7f27d5e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f27d010f4a0 con 0x7f27d00ffee0 2024-03-18T01:38:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.272+0000 7f27d5e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27d010f320 con 0x7f27d01011e0 2024-03-18T01:38:10.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.272+0000 7f27d5e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f27d010f620 con 0x7f27d0100840 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27d0100840 0x7f27d0199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27d4a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27d00ffee0 0x7f27d01999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27d0100840 0x7f27d0199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44692/0 (socket says 172.21.15.7:44692) 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 -- 172.21.15.7:0/3967797987 learned_addr learned my addr 172.21.15.7:0/3967797987 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27d5400640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d01011e0 0x7f27d01a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 -- 172.21.15.7:0/3967797987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d01011e0 msgr2=0x7f27d01a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d01011e0 0x7f27d01a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 -- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27d00ffee0 msgr2=0x7f27d01999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27d00ffee0 0x7f27d01999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 -- 172.21.15.7:0/3967797987 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d01a1610 con 0x7f27d0100840 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27d5400640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d01011e0 0x7f27d01a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.273+0000 7f27cfe00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27d0100840 0x7f27d0199f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00e9f0 tx=0x7f27bc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.274+0000 7f27cd600640 1 -- 172.21.15.7:0/3967797987 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00cd80 con 0x7f27d0100840 2024-03-18T01:38:10.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.274+0000 7f27cd600640 1 -- 172.21.15.7:0/3967797987 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27bc010910 con 0x7f27d0100840 2024-03-18T01:38:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.274+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d01a1900 con 0x7f27d0100840 2024-03-18T01:38:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.274+0000 7f27cd600640 1 -- 172.21.15.7:0/3967797987 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc018990 con 0x7f27d0100840 2024-03-18T01:38:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.274+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d01a1e50 con 0x7f27d0100840 2024-03-18T01:38:10.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.275+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27d010aca0 con 0x7f27d0100840 2024-03-18T01:38:10.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.277+0000 7f27cd600640 1 -- 172.21.15.7:0/3967797987 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f27bc010ac0 con 0x7f27d0100840 2024-03-18T01:38:10.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.279+0000 7f27cd600640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27b80a28a0 0x7f27b80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:10.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.279+0000 7f27cd600640 1 -- 172.21.15.7:0/3967797987 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f27bc0cf150 con 0x7f27d0100840 2024-03-18T01:38:10.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.279+0000 7f27d4a00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27b80a28a0 0x7f27b80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:10.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.280+0000 7f27d4a00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27b80a28a0 0x7f27b80a4d60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f27c402c250 tx=0x7f27c402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:10.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.284+0000 7f27cd600640 1 -- 172.21.15.7:0/3967797987 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27bc09b7e0 con 0x7f27d0100840 2024-03-18T01:38:10.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.534+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27d002d3d0 con 0x7f27d0100840 2024-03-18T01:38:10.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.535+0000 7f27cd600640 1 -- 172.21.15.7:0/3967797987 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11918 (secure 0 0 0) 0x7f27bc09f880 con 0x7f27d0100840 2024-03-18T01:38:10.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:10.536 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:37:59.950482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24562,24596,24542,24544,24523]},"id":1}]} 2024-03-18T01:38:10.536 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.538+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27b80a28a0 msgr2=0x7f27b80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.538+0000 7f27d5e00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27b80a28a0 0x7f27b80a4d60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f27c402c250 tx=0x7f27c402cf90 comp rx=0 tx=0).stop 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.538+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27d0100840 msgr2=0x7f27d0199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27d0100840 0x7f27d0199f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00e9f0 tx=0x7f27bc00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 shutdown_connections 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27d01011e0 0x7f27d01a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27d0100840 0x7f27d0199f30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f27b80a28a0 0x7f27b80a4d60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 --2- 172.21.15.7:0/3967797987 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f27d00ffee0 0x7f27d01999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 >> 172.21.15.7:0/3967797987 conn(0x7f27d00fbaa0 msgr2=0x7f27d00fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:10.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 shutdown_connections 2024-03-18T01:38:10.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.539+0000 7f27d5e00640 1 -- 172.21.15.7:0/3967797987 wait complete. 2024-03-18T01:38:10.557 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:38:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 -- 172.21.15.7:0/3023955415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7641011c0 msgr2=0x7fb764101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 --2- 172.21.15.7:0/3023955415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7641011c0 0x7fb764101640 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009ca0 tx=0x7fb75c02c100 comp rx=0 tx=0).stop 2024-03-18T01:38:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 -- 172.21.15.7:0/3023955415 shutdown_connections 2024-03-18T01:38:10.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 --2- 172.21.15.7:0/3023955415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7641011c0 0x7fb764101640 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 --2- 172.21.15.7:0/3023955415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb764100820 0x7fb764100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 --2- 172.21.15.7:0/3023955415 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb7640ffec0 0x7fb7641002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 -- 172.21.15.7:0/3023955415 >> 172.21.15.7:0/3023955415 conn(0x7fb7640fbaa0 msgr2=0x7fb7640fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 -- 172.21.15.7:0/3023955415 shutdown_connections 2024-03-18T01:38:10.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.769+0000 7fb76ae00640 1 -- 172.21.15.7:0/3023955415 wait complete. 2024-03-18T01:38:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.771+0000 7fb76ae00640 1 Processor -- start 2024-03-18T01:38:10.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.771+0000 7fb76ae00640 1 -- start start 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb76ae00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7640ffec0 0x7fb7641999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb76ae00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb764100820 0x7fb764199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb76ae00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7641011c0 0x7fb7641a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb76ae00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb76410f460 con 0x7fb764100820 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb76ae00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb76410f2e0 con 0x7fb7640ffec0 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb76ae00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb76410f5e0 con 0x7fb7641011c0 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb769a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7640ffec0 0x7fb7641999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb769a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7640ffec0 0x7fb7641999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:42456/0 (socket says 172.21.15.7:42456) 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb769a00640 1 -- 172.21.15.7:0/688076484 learned_addr learned my addr 172.21.15.7:0/688076484 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:10.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.772+0000 7fb76a400640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7641011c0 0x7fb7641a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb769000640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb764100820 0x7fb764199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb769a00640 1 -- 172.21.15.7:0/688076484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7641011c0 msgr2=0x7fb7641a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb769a00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7641011c0 0x7fb7641a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb769a00640 1 -- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb764100820 msgr2=0x7fb764199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb769a00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb764100820 0x7fb764199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb769a00640 1 -- 172.21.15.7:0/688076484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7641a15d0 con 0x7fb7640ffec0 2024-03-18T01:38:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb769a00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7640ffec0 0x7fb7641999b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb75800e9f0 tx=0x7fb75800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb762000640 1 -- 172.21.15.7:0/688076484 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75800cd80 con 0x7fb7640ffec0 2024-03-18T01:38:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7641a18c0 con 0x7fb7640ffec0 2024-03-18T01:38:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.773+0000 7fb762000640 1 -- 172.21.15.7:0/688076484 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb758010910 con 0x7fb7640ffec0 2024-03-18T01:38:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.774+0000 7fb762000640 1 -- 172.21.15.7:0/688076484 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7580188f0 con 0x7fb7640ffec0 2024-03-18T01:38:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.774+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7641a1e60 con 0x7fb7640ffec0 2024-03-18T01:38:10.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.775+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb76410ac10 con 0x7fb7640ffec0 2024-03-18T01:38:10.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.777+0000 7fb762000640 1 -- 172.21.15.7:0/688076484 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fb758010ac0 con 0x7fb7640ffec0 2024-03-18T01:38:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.779+0000 7fb762000640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb74c0a2920 0x7fb74c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.779+0000 7fb762000640 1 -- 172.21.15.7:0/688076484 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb7580cf1b0 con 0x7fb7640ffec0 2024-03-18T01:38:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.779+0000 7fb769000640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb74c0a2920 0x7fb74c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.780+0000 7fb769000640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb74c0a2920 0x7fb74c0a4de0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb76419aec0 tx=0x7fb750005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:10.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.782+0000 7fb762000640 1 -- 172.21.15.7:0/688076484 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb75809bef0 con 0x7fb7640ffec0 2024-03-18T01:38:10.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.990+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb76402d1b0 con 0x7fb7640ffec0 2024-03-18T01:38:10.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.992+0000 7fb762000640 1 -- 172.21.15.7:0/688076484 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7fb75809b7f0 con 0x7fb7640ffec0 2024-03-18T01:38:10.993 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:10.993 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:38:10.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb74c0a2920 msgr2=0x7fb74c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb74c0a2920 0x7fb74c0a4de0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb76419aec0 tx=0x7fb750005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7640ffec0 msgr2=0x7fb7641999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7640ffec0 0x7fb7641999b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb75800e9f0 tx=0x7fb75800eeb0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 shutdown_connections 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7641011c0 0x7fb7641a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb74c0a2920 0x7fb74c0a4de0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb764100820 0x7fb764199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 --2- 172.21.15.7:0/688076484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7640ffec0 0x7fb7641999b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 >> 172.21.15.7:0/688076484 conn(0x7fb7640fbaa0 msgr2=0x7fb7640fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 shutdown_connections 2024-03-18T01:38:10.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:10.995+0000 7fb76ae00640 1 -- 172.21.15.7:0/688076484 wait complete. 2024-03-18T01:38:11.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:10 smithi007 ceph-mon[31996]: pgmap v417: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T01:38:11.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3485861353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:11.012 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:10 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3967797987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:11.017 DEBUG:tasks.fwd_scrub:Yielding 2024-03-18T01:38:11.017 INFO:teuthology.run_tasks:Running task workunit... 2024-03-18T01:38:11.031 INFO:tasks.workunit:Pulling workunits from ref c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-18T01:38:11.031 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T01:38:11.031 INFO:tasks.workunit:timeout=3h 2024-03-18T01:38:11.031 INFO:tasks.workunit:cleanup=True 2024-03-18T01:38:11.032 DEBUG:teuthology.orchestra.run.smithi007:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T01:38:11.032 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-18T01:38:11.032 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T01:38:11.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:38:11.032 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-18 01:35:31.441311021 +0000 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-18 01:35:31.441311021 +0000 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-18 01:37:58.769699608 +0000 2024-03-18T01:38:11.081 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-18 01:35:31.441311021 +0000 2024-03-18T01:38:11.081 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T01:38:11.082 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T01:38:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:10 smithi203 ceph-mon[36345]: pgmap v417: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T01:38:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:10 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3485861353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:10 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3967797987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:11.209 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-18T01:38:11.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 -- 172.21.15.7:0/3112589666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0103840 msgr2=0x7fa0d0103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/3112589666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0103840 0x7fa0d0103c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0009900 tx=0x7fa0c002bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 -- 172.21.15.7:0/3112589666 shutdown_connections 2024-03-18T01:38:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/3112589666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d01053f0 0x7fa0d010bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/3112589666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0d0104a30 0x7fa0d0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/3112589666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0103840 0x7fa0d0103c60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 -- 172.21.15.7:0/3112589666 >> 172.21.15.7:0/3112589666 conn(0x7fa0d00fedb0 msgr2=0x7fa0d01011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 -- 172.21.15.7:0/3112589666 shutdown_connections 2024-03-18T01:38:11.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.253+0000 7fa0d4a00640 1 -- 172.21.15.7:0/3112589666 wait complete. 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.254+0000 7fa0d4a00640 1 Processor -- start 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.254+0000 7fa0d4a00640 1 -- start start 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0d4a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0103840 0x7fa0d0199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0d4a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0104a30 0x7fa0d019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0d4a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0d01053f0 0x7fa0d01a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0d4a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010f500 con 0x7fa0d0103840 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0d4a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010f380 con 0x7fa0d0104a30 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0d4a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d010f680 con 0x7fa0d01053f0 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0ce800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0103840 0x7fa0d0199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0104a30 0x7fa0d019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0ce800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0103840 0x7fa0d0199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60176/0 (socket says 172.21.15.7:60176) 2024-03-18T01:38:11.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0ce800640 1 -- 172.21.15.7:0/1895814976 learned_addr learned my addr 172.21.15.7:0/1895814976 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0cf200640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0d01053f0 0x7fa0d01a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0cde00640 1 -- 172.21.15.7:0/1895814976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0d01053f0 msgr2=0x7fa0d01a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0cde00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0d01053f0 0x7fa0d01a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0cde00640 1 -- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0103840 msgr2=0x7fa0d0199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0cde00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0103840 0x7fa0d0199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.255+0000 7fa0cde00640 1 -- 172.21.15.7:0/1895814976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d01a17c0 con 0x7fa0d0104a30 2024-03-18T01:38:11.257 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T01:38:11.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.256+0000 7fa0cde00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0104a30 0x7fa0d019a0e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00e9f0 tx=0x7fa0bc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.256+0000 7fa0c7400640 1 -- 172.21.15.7:0/1895814976 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc00ccc0 con 0x7fa0d0104a30 2024-03-18T01:38:11.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.256+0000 7fa0c7400640 1 -- 172.21.15.7:0/1895814976 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0bc005020 con 0x7fa0d0104a30 2024-03-18T01:38:11.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.256+0000 7fa0d4a00640 1 -- 172.21.15.7:0/1895814976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d01a1ab0 con 0x7fa0d0104a30 2024-03-18T01:38:11.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.256+0000 7fa0c7400640 1 -- 172.21.15.7:0/1895814976 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0049e0 con 0x7fa0d0104a30 2024-03-18T01:38:11.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.256+0000 7fa0d4a00640 1 -- 172.21.15.7:0/1895814976 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d01a1fb0 con 0x7fa0d0104a30 2024-03-18T01:38:11.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0d81c1740 1 Processor -- start 2024-03-18T01:38:11.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0d81c1740 1 -- start start 2024-03-18T01:38:11.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0c7400640 1 -- 172.21.15.7:0/1895814976 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fa0bc005560 con 0x7fa0d0104a30 2024-03-18T01:38:11.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04dbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04e3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5614b043ef80 con 0x5614b04e14e0 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5614b03e7ad0 con 0x5614b04dc370 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.260+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5614b043b020 con 0x5614b04db980 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cf200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04dbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0ce800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cf200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04dbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44726/0 (socket says 172.21.15.7:44726) 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cf200640 1 -- 172.21.15.7:0/2659637334 learned_addr learned my addr 172.21.15.7:0/2659637334 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cf200640 1 -- 172.21.15.7:0/2659637334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 msgr2=0x5614b04e0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cf200640 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04e0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cf200640 1 -- 172.21.15.7:0/2659637334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 msgr2=0x5614b04e3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cde00640 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cf200640 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04e3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cf200640 1 -- 172.21.15.7:0/2659637334 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614b03ff6f0 con 0x5614b04db980 2024-03-18T01:38:11.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.261+0000 7fa0cde00640 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:11.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.262+0000 7fa0cf200640 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04dbda0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8010370 tx=0x7fa0c8010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96ad2aa12ae15d49 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.262+0000 7fa0c7400640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0b40a2800 0x7fa0b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.262+0000 7fa0bb400640 1 -- 172.21.15.7:0/2659637334 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8016070 con 0x5614b04db980 2024-03-18T01:38:11.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.262+0000 7fa0c7400640 1 -- 172.21.15.7:0/1895814976 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa0bc0cedd0 con 0x7fa0d0104a30 2024-03-18T01:38:11.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.262+0000 7fa0bb400640 1 -- 172.21.15.7:0/2659637334 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0c8004d20 con 0x5614b04db980 2024-03-18T01:38:11.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.262+0000 7fa0bb400640 1 -- 172.21.15.7:0/2659637334 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8005480 con 0x5614b04db980 2024-03-18T01:38:11.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.262+0000 7fa0ce800640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0b40a2800 0x7fa0b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2659637334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 msgr2=0x5614b04dbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04dbda0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8010370 tx=0x7fa0c8010830 comp rx=0 tx=0).stop 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0ce800640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0b40a2800 0x7fa0b40a4cc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0005ab0 tx=0x7fa0c002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2659637334 shutdown_connections 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04e0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2659637334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04dbda0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2659637334 >> 172.21.15.7:0/2659637334 conn(0x5614b04a3930 msgr2=0x5614b04d72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2659637334 shutdown_connections 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.263+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2659637334 wait complete. 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.264+0000 7fa0d81c1740 1 -- start start 2024-03-18T01:38:11.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.264+0000 7fa0d81c1740 1 Processor -- start 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04db980 0x5614b04e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04e6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04e14e0 0x5614b04ed270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5614b04e4490 con 0x5614b04db980 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5614b03faf30 con 0x5614b04dc370 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5614b04e4610 con 0x5614b04e14e0 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0d81c1740 10 client.0 _ll_register_callbacks cb 0x5614b03e8bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0ce800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04db980 0x5614b04e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0cf200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04e14e0 0x5614b04ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04e6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0ce800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04db980 0x5614b04e5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60202/0 (socket says 172.21.15.7:60202) 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0ce800640 1 -- 172.21.15.7:0/2888213826 learned_addr learned my addr 172.21.15.7:0/2888213826 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0cf200640 1 -- 172.21.15.7:0/2888213826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 msgr2=0x5614b04e6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0cf200640 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04e6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0cf200640 1 -- 172.21.15.7:0/2888213826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04db980 msgr2=0x5614b04e5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0cf200640 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04db980 0x5614b04e5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.265+0000 7fa0cf200640 1 -- 172.21.15.7:0/2888213826 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614b021b7e0 con 0x5614b04e14e0 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.266+0000 7fa0cf200640 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04e14e0 0x5614b04ed270 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8010d30 tx=0x7fa0c8004c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a75c5dfea4f3cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.266+0000 7fa0afe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.266+0000 7fa0afe00640 1 -- 172.21.15.7:0/2888213826 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8029060 con 0x5614b04e14e0 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.266+0000 7fa0afe00640 1 -- 172.21.15.7:0/2888213826 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa0c8003cc0 con 0x5614b04e14e0 2024-03-18T01:38:11.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.266+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2888213826 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0c8004d20 con 0x5614b04e14e0 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.266+0000 7fa0afe00640 1 -- 172.21.15.7:0/2888213826 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa0c8030050 con 0x5614b04e14e0 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.266+0000 7fa0d81c1740 10 client.25591 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.266+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2888213826 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5614b027bf70 con 0x5614b04e14e0 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0afe00640 1 -- 172.21.15.7:0/2888213826 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0c8004d20 con 0x5614b04e14e0 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0afe00640 1 -- 172.21.15.7:0/2888213826 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa0c800f040 con 0x5614b04e14e0 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0d81c1740 10 client.25591 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0d81c1740 10 client.25591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0d81c1740 20 client.25591 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5614b04f17d0 0x5614b04f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0d81c1740 4 client.25591 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2888213826 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614b044c4c0 con 0x5614b04f17d0 2024-03-18T01:38:11.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.267+0000 7fa0ce800640 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5614b04f17d0 0x5614b04f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.268+0000 7fa0ce800640 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5614b04f17d0 0x5614b04f3bc0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98af5186d9680e2f server_cookie=2b8cd252265571ac in_seq=0 out_seq=0 2024-03-18T01:38:11.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.268+0000 7fa0afe00640 10 client.25591 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:11.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:10 smithi046 ceph-mon[36108]: pgmap v417: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T01:38:11.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:10 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3485861353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:38:11.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:10 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3967797987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:11.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.268+0000 7fa0afe00640 1 -- 172.21.15.7:0/2888213826 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5614b044c4c0 con 0x5614b04f17d0 2024-03-18T01:38:11.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.268+0000 7fa0afe00640 10 client.25591 handle_command_reply: tid=0 2024-03-18T01:38:11.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 client.25591 shutdown 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2888213826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04e14e0 msgr2=0x5614b04ed270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04e14e0 0x5614b04ed270 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8010d30 tx=0x7fa0c8004c70 comp rx=0 tx=0).stop 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2888213826 shutdown_connections 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5614b04f17d0 0x5614b04f3bc0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04e14e0 0x5614b04ed270 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04e6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0afe00640 0 client.25591 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2888213826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04db980 0x5614b04e5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2888213826 >> 172.21.15.7:0/2888213826 conn(0x5614b04a3930 msgr2=0x5614b04d6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2888213826 shutdown_connections 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2888213826 wait complete. 2024-03-18T01:38:11.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.269+0000 7fa0d81c1740 20 client.25591 trim_cache size 0 max 16384 2024-03-18T01:38:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.296+0000 7fa0d81c1740 1 Processor -- start 2024-03-18T01:38:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.296+0000 7fa0d81c1740 1 -- start start 2024-03-18T01:38:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.296+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04db980 0x5614b04dd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.296+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04dc370 0x5614b04ddc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.296+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ed230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d01ac380 con 0x5614b04e14e0 2024-03-18T01:38:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d01adec0 con 0x5614b04db980 2024-03-18T01:38:11.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d01aca50 con 0x5614b04dc370 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04dc370 0x5614b04ddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04dc370 0x5614b04ddc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44738/0 (socket says 172.21.15.7:44738) 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cde00640 1 -- 172.21.15.7:0/635791888 learned_addr learned my addr 172.21.15.7:0/635791888 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0ce800640 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04db980 0x5614b04dd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cf200640 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ed230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cde00640 1 -- 172.21.15.7:0/635791888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04db980 msgr2=0x5614b04dd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cde00640 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04db980 0x5614b04dd730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cde00640 1 -- 172.21.15.7:0/635791888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 msgr2=0x5614b04ed230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cde00640 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ed230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cde00640 1 -- 172.21.15.7:0/635791888 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614b0113170 con 0x5614b04dc370 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.297+0000 7fa0cf200640 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ed230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.298+0000 7fa0cde00640 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04dc370 0x5614b04ddc70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc097d80 tx=0x7fa0bc0054d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6abe2a4a2ad6fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.298+0000 7fa0baa00640 1 -- 172.21.15.7:0/635791888 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc01c070 con 0x5614b04dc370 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.298+0000 7fa0baa00640 1 -- 172.21.15.7:0/635791888 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0bc09f500 con 0x5614b04dc370 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.298+0000 7fa0baa00640 1 -- 172.21.15.7:0/635791888 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc0a36c0 con 0x5614b04dc370 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 -- 172.21.15.7:0/635791888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04dc370 msgr2=0x5614b04ddc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04dc370 0x5614b04ddc70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc097d80 tx=0x7fa0bc0054d0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 -- 172.21.15.7:0/635791888 shutdown_connections 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ed230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04dc370 0x5614b04ddc70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/635791888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04db980 0x5614b04dd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 -- 172.21.15.7:0/635791888 >> 172.21.15.7:0/635791888 conn(0x5614b04a3930 msgr2=0x5614b04e5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 -- 172.21.15.7:0/635791888 shutdown_connections 2024-03-18T01:38:11.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 -- 172.21.15.7:0/635791888 wait complete. 2024-03-18T01:38:11.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.299+0000 7fa0d81c1740 1 -- start start 2024-03-18T01:38:11.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0d81c1740 1 Processor -- start 2024-03-18T01:38:11.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04eb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0d81c1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ec150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d01ad7f0 con 0x5614b04e14e0 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d01ad120 con 0x5614b04dc370 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0d81c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d01aa360 con 0x5614b04db980 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0d81c1740 10 client.0 _ll_register_callbacks cb 0x7fa0c0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0ce800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0cde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0ce800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04eacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44742/0 (socket says 172.21.15.7:44742) 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0ce800640 1 -- 172.21.15.7:0/2589994275 learned_addr learned my addr 172.21.15.7:0/2589994275 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.300+0000 7fa0cf200640 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0ce800640 1 -- 172.21.15.7:0/2589994275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 msgr2=0x5614b04eb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0ce800640 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04eb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0ce800640 1 -- 172.21.15.7:0/2589994275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 msgr2=0x5614b04ec150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0ce800640 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ec150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0ce800640 1 -- 172.21.15.7:0/2589994275 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614b04ec740 con 0x5614b04db980 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0cf200640 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0ce800640 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04eacc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa0c002c180 tx=0x7fa0c002bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b0a24bcc1bc157e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0afe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0afe00640 1 -- 172.21.15.7:0/2589994275 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00543c0 con 0x5614b04db980 2024-03-18T01:38:11.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2589994275 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0bc09f500 con 0x5614b04db980 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.301+0000 7fa0afe00640 1 -- 172.21.15.7:0/2589994275 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa0c0054560 con 0x5614b04db980 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0afe00640 1 -- 172.21.15.7:0/2589994275 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa0c0062050 con 0x5614b04db980 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0d81c1740 10 client.25597 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2589994275 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5614b0557ba0 con 0x5614b04db980 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0afe00640 1 -- 172.21.15.7:0/2589994275 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0bc09f500 con 0x5614b04db980 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0afe00640 1 -- 172.21.15.7:0/2589994275 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa0c0046040 con 0x5614b04db980 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0d81c1740 10 client.25597 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0d81c1740 10 client.25597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0d81c1740 20 client.25597 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5614b05585d0 0x5614b055a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0d81c1740 4 client.25597 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2589994275 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5614b04dd240 con 0x5614b05585d0 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.302+0000 7fa0cde00640 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5614b05585d0 0x5614b055a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.303+0000 7fa0cde00640 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5614b05585d0 0x5614b055a9f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fcbccb81aac0983 server_cookie=e1eb28362390fb7b in_seq=0 out_seq=0 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.303+0000 7fa0afe00640 10 client.25597 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.304+0000 7fa0afe00640 1 -- 172.21.15.7:0/2589994275 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5614b04dd240 con 0x5614b05585d0 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.304+0000 7fa0afe00640 10 client.25597 handle_command_reply: tid=0 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.304+0000 7fa0d81c1740 1 client.25597 shutdown 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.304+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2589994275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 msgr2=0x5614b04eacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.304+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04eacc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa0c002c180 tx=0x7fa0c002bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.304+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2589994275 shutdown_connections 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.304+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5614b05585d0 0x5614b055a9f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.305+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5614b04e14e0 0x5614b04ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.305+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5614b04dc370 0x5614b04eb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.305+0000 7fa0d81c1740 1 --2- 172.21.15.7:0/2589994275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5614b04db980 0x5614b04eacc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.305+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2589994275 >> 172.21.15.7:0/2589994275 conn(0x5614b04a3930 msgr2=0x5614b04dd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.305+0000 7fa0afe00640 0 client.25597 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:11.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.305+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2589994275 shutdown_connections 2024-03-18T01:38:11.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.305+0000 7fa0d81c1740 1 -- 172.21.15.7:0/2589994275 wait complete. 2024-03-18T01:38:11.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.305+0000 7fa0d81c1740 20 client.25597 trim_cache size 0 max 16384 2024-03-18T01:38:11.310 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:38:11.310 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-03-18T01:38:11.310 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "834539fb-ca6e-4e8c-a9c8-baa84d427423", 2024-03-18T01:38:11.310 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-03-18T01:38:11.310 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:38:11.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 -- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0b40a2800 msgr2=0x7fa0b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0b40a2800 0x7fa0b40a4cc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0005ab0 tx=0x7fa0c002cf90 comp rx=0 tx=0).stop 2024-03-18T01:38:11.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 -- 172.21.15.7:0/1895814976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0104a30 msgr2=0x7fa0d019a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0104a30 0x7fa0d019a0e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc00e9f0 tx=0x7fa0bc00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 -- 172.21.15.7:0/1895814976 shutdown_connections 2024-03-18T01:38:11.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0d01053f0 0x7fa0d01a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d0104a30 0x7fa0d019a0e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa0b40a2800 0x7fa0b40a4cc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 --2- 172.21.15.7:0/1895814976 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa0d0103840 0x7fa0d0199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 -- 172.21.15.7:0/1895814976 >> 172.21.15.7:0/1895814976 conn(0x7fa0d00fedb0 msgr2=0x7fa0d0107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 -- 172.21.15.7:0/1895814976 shutdown_connections 2024-03-18T01:38:11.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.306+0000 7fa0d4a00640 1 -- 172.21.15.7:0/1895814976 wait complete. 2024-03-18T01:38:11.331 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:38:11.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:38:11.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 -- 172.21.15.7:0/1998093725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc107320 msgr2=0x7fefdc109770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 --2- 172.21.15.7:0/1998093725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc107320 0x7fefdc109770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fefd4009be0 tx=0x7fefd402bef0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 -- 172.21.15.7:0/1998093725 shutdown_connections 2024-03-18T01:38:11.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 --2- 172.21.15.7:0/1998093725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc107320 0x7fefdc109770 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 --2- 172.21.15.7:0/1998093725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefdc1049f0 0x7fefdc106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 --2- 172.21.15.7:0/1998093725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefdc1020c0 0x7fefdc1044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 -- 172.21.15.7:0/1998093725 >> 172.21.15.7:0/1998093725 conn(0x7fefdc0fbad0 msgr2=0x7fefdc0fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 -- 172.21.15.7:0/1998093725 shutdown_connections 2024-03-18T01:38:11.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.593+0000 7fefe2800640 1 -- 172.21.15.7:0/1998093725 wait complete. 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.594+0000 7fefe2800640 1 Processor -- start 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.594+0000 7fefe2800640 1 -- start start 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.594+0000 7fefe2800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefdc1020c0 0x7fefdc101a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.594+0000 7fefe2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc1049f0 0x7fefdc100070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.594+0000 7fefe2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefdc107320 0x7fefdc1005b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.594+0000 7fefe2800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc10d310 con 0x7fefdc1020c0 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.594+0000 7fefe2800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc10d190 con 0x7fefdc107320 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.594+0000 7fefe2800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc10d490 con 0x7fefdc1049f0 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe1400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefdc1020c0 0x7fefdc101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc1049f0 0x7fefdc100070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe1400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefdc1020c0 0x7fefdc101a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60234/0 (socket says 172.21.15.7:60234) 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe1400640 1 -- 172.21.15.7:0/2358798077 learned_addr learned my addr 172.21.15.7:0/2358798077 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe1e00640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefdc107320 0x7fefdc1005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe0a00640 1 -- 172.21.15.7:0/2358798077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefdc107320 msgr2=0x7fefdc1005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe0a00640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefdc107320 0x7fefdc1005b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe0a00640 1 -- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefdc1020c0 msgr2=0x7fefdc101a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe0a00640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefdc1020c0 0x7fefdc101a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe0a00640 1 -- 172.21.15.7:0/2358798077 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefdc100e40 con 0x7fefdc1049f0 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.595+0000 7fefe0a00640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc1049f0 0x7fefdc100070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fefc8005fd0 tx=0x7fefc800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.596+0000 7fefda000640 1 -- 172.21.15.7:0/2358798077 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc800ccc0 con 0x7fefdc1049f0 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.596+0000 7fefda000640 1 -- 172.21.15.7:0/2358798077 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc800ee70 con 0x7fefdc1049f0 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.596+0000 7fefe2800640 1 -- 172.21.15.7:0/2358798077 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefdc1a1a00 con 0x7fefdc1049f0 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.596+0000 7fefda000640 1 -- 172.21.15.7:0/2358798077 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc80187e0 con 0x7fefdc1049f0 2024-03-18T01:38:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.596+0000 7fefe2800640 1 -- 172.21.15.7:0/2358798077 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefdc1a1fa0 con 0x7fefdc1049f0 2024-03-18T01:38:11.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.599+0000 7fefda000640 1 -- 172.21.15.7:0/2358798077 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fefc8010430 con 0x7fefdc1049f0 2024-03-18T01:38:11.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.599+0000 7fefe6bb5740 1 Processor -- start 2024-03-18T01:38:11.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.599+0000 7fefe6bb5740 1 -- start start 2024-03-18T01:38:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d5a10 0x55f67e1d5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d6400 0x55f67e1daf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1dda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f67e13a1f0 con 0x55f67e1d6400 2024-03-18T01:38:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f67e0e2a80 con 0x55f67e1db570 2024-03-18T01:38:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f67e136290 con 0x55f67e1d5a10 2024-03-18T01:38:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d5a10 0x55f67e1d5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d5a10 0x55f67e1d5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44770/0 (socket says 172.21.15.7:44770) 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe1e00640 1 -- 172.21.15.7:0/2273731125 learned_addr learned my addr 172.21.15.7:0/2273731125 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe0a00640 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1dda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.600+0000 7fefe1400640 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d6400 0x55f67e1daf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefe1e00640 1 -- 172.21.15.7:0/2273731125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 msgr2=0x55f67e1dda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefe1e00640 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1dda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefe1e00640 1 -- 172.21.15.7:0/2273731125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d6400 msgr2=0x55f67e1daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefe1e00640 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d6400 0x55f67e1daf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefe1e00640 1 -- 172.21.15.7:0/2273731125 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f67df187d0 con 0x55f67e1d5a10 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefe1400640 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d6400 0x55f67e1daf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefda000640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefc40a2880 0x7fefc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefe1e00640 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d5a10 0x55f67e1d5e30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fefd402c500 tx=0x7fefd4009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c447b8c9705f0f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.603 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefda000640 1 -- 172.21.15.7:0/2358798077 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fefc80cef90 con 0x7fefdc1049f0 2024-03-18T01:38:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefce600640 1 -- 172.21.15.7:0/2273731125 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4043030 con 0x55f67e1d5a10 2024-03-18T01:38:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefe1400640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefc40a2880 0x7fefc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefce600640 1 -- 172.21.15.7:0/2273731125 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd4033580 con 0x55f67e1d5a10 2024-03-18T01:38:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.601+0000 7fefce600640 1 -- 172.21.15.7:0/2273731125 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd402c890 con 0x55f67e1d5a10 2024-03-18T01:38:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2273731125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d5a10 msgr2=0x55f67e1d5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d5a10 0x55f67e1d5e30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fefd402c500 tx=0x7fefd4009330 comp rx=0 tx=0).stop 2024-03-18T01:38:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe1400640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefc40a2880 0x7fefc40a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55f67e1d6800 tx=0x7fefd0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2273731125 shutdown_connections 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1dda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d6400 0x55f67e1daf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/2273731125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d5a10 0x55f67e1d5e30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2273731125 >> 172.21.15.7:0/2273731125 conn(0x55f67e1d0f80 msgr2=0x55f67e1d1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2273731125 shutdown_connections 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.602+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2273731125 wait complete. 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.603+0000 7fefe6bb5740 1 -- start start 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.603+0000 7fefe6bb5740 1 Processor -- start 2024-03-18T01:38:11.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.603+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1dffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.603+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d6400 0x55f67e1e04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.603+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1db570 0x55f67e1e7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.603+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f67e1de4c0 con 0x55f67e1d5a10 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.603+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f67e0f5ed0 con 0x55f67e1d6400 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.603+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f67e1de640 con 0x55f67e1db570 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe6bb5740 10 client.0 _ll_register_callbacks cb 0x55f67e131220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe1400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1dffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe1e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1db570 0x55f67e1e7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe0a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d6400 0x55f67e1e04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe1400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1dffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:60254/0 (socket says 172.21.15.7:60254) 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe1400640 1 -- 172.21.15.7:0/480765766 learned_addr learned my addr 172.21.15.7:0/480765766 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe1e00640 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1db570 0x55f67e1e7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44780/0 (socket says 172.21.15.7:44780) 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe0a00640 1 -- 172.21.15.7:0/480765766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1db570 msgr2=0x55f67e1e7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe0a00640 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1db570 0x55f67e1e7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe0a00640 1 -- 172.21.15.7:0/480765766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 msgr2=0x55f67e1dffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe0a00640 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1dffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe0a00640 1 -- 172.21.15.7:0/480765766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f67de4e3f0 con 0x55f67e1d6400 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe1e00640 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1db570 0x55f67e1e7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefe0a00640 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d6400 0x55f67e1e04f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fefc809b320 tx=0x7fefc800cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7848a68e723cd451 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.604+0000 7fefc2a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.605+0000 7fefc2a00640 1 -- 172.21.15.7:0/480765766 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc809b160 con 0x55f67e1d6400 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.605+0000 7fefe6bb5740 1 -- 172.21.15.7:0/480765766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefd4033580 con 0x55f67e1d6400 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.605+0000 7fefc2a00640 1 -- 172.21.15.7:0/480765766 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fefc809f400 con 0x55f67e1d6400 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.605+0000 7fefc2a00640 1 -- 172.21.15.7:0/480765766 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fefc809af80 con 0x55f67e1d6400 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.605+0000 7fefe6bb5740 10 client.25616 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.605+0000 7fefe6bb5740 1 -- 172.21.15.7:0/480765766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f67de0e0f0 con 0x55f67e1d6400 2024-03-18T01:38:11.610 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.605+0000 7fefc2a00640 1 -- 172.21.15.7:0/480765766 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd4033580 con 0x55f67e1d6400 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.605+0000 7fefc2a00640 1 -- 172.21.15.7:0/480765766 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fefc8084bb0 con 0x55f67e1d6400 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.606+0000 7fefe6bb5740 10 client.25616 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.606+0000 7fefe6bb5740 10 client.25616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.606+0000 7fefe6bb5740 20 client.25616 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.606+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f67e1ebd00 0x55f67e1ee130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.606+0000 7fefe6bb5740 4 client.25616 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.606+0000 7fefe6bb5740 1 -- 172.21.15.7:0/480765766 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f67e14dbe0 con 0x55f67e1ebd00 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.606+0000 7fefe1400640 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f67e1ebd00 0x55f67e1ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.607+0000 7fefe1400640 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f67e1ebd00 0x55f67e1ee130 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8292e4b0e9b30e5 server_cookie=fc9a21dc63e5747b in_seq=0 out_seq=0 2024-03-18T01:38:11.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.607+0000 7fefc2a00640 10 client.25616 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.607+0000 7fefc2a00640 1 -- 172.21.15.7:0/480765766 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f67e14dbe0 con 0x55f67e1ebd00 2024-03-18T01:38:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefc2a00640 10 client.25616 handle_command_reply: tid=0 2024-03-18T01:38:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 client.25616 shutdown 2024-03-18T01:38:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 -- 172.21.15.7:0/480765766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d6400 msgr2=0x55f67e1e04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d6400 0x55f67e1e04f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fefc809b320 tx=0x7fefc800cf20 comp rx=0 tx=0).stop 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 -- 172.21.15.7:0/480765766 shutdown_connections 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f67e1ebd00 0x55f67e1ee130 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1db570 0x55f67e1e7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d6400 0x55f67e1e04f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefc2a00640 0 client.25616 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/480765766 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1dffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 -- 172.21.15.7:0/480765766 >> 172.21.15.7:0/480765766 conn(0x55f67e1d0f80 msgr2=0x55f67e1d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 -- 172.21.15.7:0/480765766 shutdown_connections 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 1 -- 172.21.15.7:0/480765766 wait complete. 2024-03-18T01:38:11.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.608+0000 7fefe6bb5740 20 client.25616 trim_cache size 0 max 16384 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.627+0000 7fefe6bb5740 1 Processor -- start 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.627+0000 7fefe6bb5740 1 -- start start 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d5a10 0x55f67e1cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1db570 0x55f67e1f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc1ac370 con 0x55f67e1db570 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc1adeb0 con 0x55f67e1d5a10 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc1aca40 con 0x55f67e1d6400 2024-03-18T01:38:11.628 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d5a10 0x55f67e1cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1db570 0x55f67e1f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d5a10 0x55f67e1cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:42530/0 (socket says 172.21.15.7:42530) 2024-03-18T01:38:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1400640 1 -- 172.21.15.7:0/2019273379 learned_addr learned my addr 172.21.15.7:0/2019273379 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.629 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1400640 1 -- 172.21.15.7:0/2019273379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 msgr2=0x55f67e1e74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1400640 1 --2- 172.21.15.7:0/2019273379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1400640 1 -- 172.21.15.7:0/2019273379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1db570 msgr2=0x55f67e1f2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1400640 1 --2- 172.21.15.7:0/2019273379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1db570 0x55f67e1f2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.628+0000 7fefe1400640 1 -- 172.21.15.7:0/2019273379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f67e1d5650 con 0x55f67e1d5a10 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.629+0000 7fefe1400640 1 --2- 172.21.15.7:0/2019273379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d5a10 0x55f67e1cff90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55f67e1ee970 tx=0x7fefd0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d0e4063d11f8e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.629+0000 7fefcdc00640 1 -- 172.21.15.7:0/2019273379 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd0031070 con 0x55f67e1d5a10 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.629+0000 7fefcdc00640 1 -- 172.21.15.7:0/2019273379 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd001dd80 con 0x55f67e1d5a10 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.629+0000 7fefcdc00640 1 -- 172.21.15.7:0/2019273379 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd002aa70 con 0x55f67e1d5a10 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2019273379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d5a10 msgr2=0x55f67e1cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/2019273379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d5a10 0x55f67e1cff90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x55f67e1ee970 tx=0x7fefd0019000 comp rx=0 tx=0).stop 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2019273379 shutdown_connections 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/2019273379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1db570 0x55f67e1f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/2019273379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/2019273379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1d5a10 0x55f67e1cff90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2019273379 >> 172.21.15.7:0/2019273379 conn(0x55f67e1d0f80 msgr2=0x55f67e1cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2019273379 shutdown_connections 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 -- 172.21.15.7:0/2019273379 wait complete. 2024-03-18T01:38:11.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 -- start start 2024-03-18T01:38:11.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.630+0000 7fefe6bb5740 1 Processor -- start 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1e4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe6bb5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1e6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc1ad7e0 con 0x55f67e1d5a10 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc1ad110 con 0x55f67e1db570 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe6bb5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc1aa350 con 0x55f67e1d6400 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe6bb5740 10 client.0 _ll_register_callbacks cb 0x7fefc80d26c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe1400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1e4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1e6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1e6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:42538/0 (socket says 172.21.15.7:42538) 2024-03-18T01:38:11.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe1e00640 1 -- 172.21.15.7:0/1873319732 learned_addr learned my addr 172.21.15.7:0/1873319732 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe0a00640 1 -- 172.21.15.7:0/1873319732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 msgr2=0x55f67e1e6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe0a00640 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1e6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe0a00640 1 -- 172.21.15.7:0/1873319732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 msgr2=0x55f67e1e4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.631+0000 7fefe0a00640 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1e4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefe0a00640 1 -- 172.21.15.7:0/1873319732 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f67e1e69f0 con 0x55f67e1d6400 2024-03-18T01:38:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefe1400640 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1e4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefe1e00640 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1e6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:11.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefe0a00640 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e53f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fefc8097ff0 tx=0x7fefc800c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4f08d2b65c9de75 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefc2a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefc2a00640 1 -- 172.21.15.7:0/1873319732 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc809b160 con 0x55f67e1d6400 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefc2a00640 1 -- 172.21.15.7:0/1873319732 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fefc809f1b0 con 0x55f67e1d6400 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefe6bb5740 1 -- 172.21.15.7:0/1873319732 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefd001dd80 con 0x55f67e1d6400 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefc2a00640 1 -- 172.21.15.7:0/1873319732 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fefc80a3d10 con 0x55f67e1d6400 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefe6bb5740 10 client.25615 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.632+0000 7fefe6bb5740 1 -- 172.21.15.7:0/1873319732 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f67e2523c0 con 0x55f67e1d6400 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefc2a00640 1 -- 172.21.15.7:0/1873319732 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefd001dd80 con 0x55f67e1d6400 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefc2a00640 1 -- 172.21.15.7:0/1873319732 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fefc80d2b40 con 0x55f67e1d6400 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefe6bb5740 10 client.25615 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefe6bb5740 10 client.25615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefe6bb5740 20 client.25615 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:11.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f67e252d60 0x55f67e255180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefe6bb5740 4 client.25615 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefe6bb5740 1 -- 172.21.15.7:0/1873319732 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f67e147730 con 0x55f67e252d60 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.633+0000 7fefe1400640 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f67e252d60 0x55f67e255180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe1400640 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f67e252d60 0x55f67e255180 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb8e2a3256b3cddb server_cookie=64d6a3717bb002fd in_seq=0 out_seq=0 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefc2a00640 10 client.25615 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefc2a00640 1 -- 172.21.15.7:0/1873319732 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f67e147730 con 0x55f67e252d60 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefc2a00640 10 client.25615 handle_command_reply: tid=0 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 client.25615 shutdown 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 -- 172.21.15.7:0/1873319732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 msgr2=0x55f67e1e53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e53f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fefc8097ff0 tx=0x7fefc800c840 comp rx=0 tx=0).stop 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 -- 172.21.15.7:0/1873319732 shutdown_connections 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f67e252d60 0x55f67e255180 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f67e1db570 0x55f67e1e6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f67e1d6400 0x55f67e1e53f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 --2- 172.21.15.7:0/1873319732 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f67e1d5a10 0x55f67e1e4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 -- 172.21.15.7:0/1873319732 >> 172.21.15.7:0/1873319732 conn(0x55f67e1d0f80 msgr2=0x55f67e1d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefc2a00640 0 client.25615 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 -- 172.21.15.7:0/1873319732 shutdown_connections 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 1 -- 172.21.15.7:0/1873319732 wait complete. 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.634+0000 7fefe6bb5740 20 client.25615 trim_cache size 0 max 16384 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:38:11.640 INFO:teuthology.orchestra.run.smithi007.stdout: "834539fb-ca6e-4e8c-a9c8-baa84d427423": { 2024-03-18T01:38:11.641 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:38:11.641 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "834539fb-ca6e-4e8c-a9c8-baa84d427423", 2024-03-18T01:38:11.641 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:38:11.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:38:11.641 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:38:11.641 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:38:11.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 -- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefc40a2880 msgr2=0x7fefc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefc40a2880 0x7fefc40a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55f67e1d6800 tx=0x7fefd0005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:11.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 -- 172.21.15.7:0/2358798077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc1049f0 msgr2=0x7fefdc100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:11.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc1049f0 0x7fefdc100070 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fefc8005fd0 tx=0x7fefc800ec20 comp rx=0 tx=0).stop 2024-03-18T01:38:11.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 -- 172.21.15.7:0/2358798077 shutdown_connections 2024-03-18T01:38:11.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefdc107320 0x7fefdc1005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefdc1049f0 0x7fefdc100070 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefc40a2880 0x7fefc40a4d40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 --2- 172.21.15.7:0/2358798077 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefdc1020c0 0x7fefdc101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 -- 172.21.15.7:0/2358798077 >> 172.21.15.7:0/2358798077 conn(0x7fefdc0fbad0 msgr2=0x7fefdc1029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 -- 172.21.15.7:0/2358798077 shutdown_connections 2024-03-18T01:38:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:11.635+0000 7fefe2800640 1 -- 172.21.15.7:0/2358798077 wait complete. 2024-03-18T01:38:11.654 INFO:tasks.cephfs.filesystem:scrub status for tag:834539fb-ca6e-4e8c-a9c8-baa84d427423 - {'path': '/', 'tag': '834539fb-ca6e-4e8c-a9c8-baa84d427423', 'options': 'recursive,force'} 2024-03-18T01:38:12.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:11 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/688076484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:12.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:11 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/688076484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:12.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:11 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/688076484' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:38:13.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:12 smithi007 ceph-mon[31996]: pgmap v418: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:13.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:12 smithi007 ceph-mon[31996]: scrub queued for path: / 2024-03-18T01:38:13.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:12 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:13.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:12 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:38:13.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:12 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:12 smithi203 ceph-mon[36345]: pgmap v418: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:12 smithi203 ceph-mon[36345]: scrub queued for path: / 2024-03-18T01:38:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:12 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:12 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:38:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:12 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:12 smithi046 ceph-mon[36108]: pgmap v418: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:38:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:12 smithi046 ceph-mon[36108]: scrub queued for path: / 2024-03-18T01:38:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:12 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:12 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:38:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:12 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:15.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:14 smithi007 ceph-mon[31996]: pgmap v419: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T01:38:15.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:14 smithi203 ceph-mon[36345]: pgmap v419: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T01:38:15.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:14 smithi046 ceph-mon[36108]: pgmap v419: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T01:38:17.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:16 smithi007 ceph-mon[31996]: pgmap v420: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:38:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:16 smithi203 ceph-mon[36345]: pgmap v420: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:38:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:16 smithi046 ceph-mon[36108]: pgmap v420: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:38:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:18 smithi203 ceph-mon[36345]: pgmap v421: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 852 B/s wr, 10 op/s 2024-03-18T01:38:19.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:18 smithi046 ceph-mon[36108]: pgmap v421: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 852 B/s wr, 10 op/s 2024-03-18T01:38:19.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:18 smithi007 ceph-mon[31996]: pgmap v421: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 852 B/s wr, 10 op/s 2024-03-18T01:38:21.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:20 smithi203 ceph-mon[36345]: pgmap v422: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T01:38:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:20 smithi046 ceph-mon[36108]: pgmap v422: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T01:38:21.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:20 smithi007 ceph-mon[31996]: pgmap v422: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T01:38:23.071 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:22 smithi046 ceph-mon[36108]: pgmap v423: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-03-18T01:38:23.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:22 smithi203 ceph-mon[36345]: pgmap v423: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-03-18T01:38:23.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:22 smithi007 ceph-mon[31996]: pgmap v423: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 852 B/s wr, 10 op/s 2024-03-18T01:38:25.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:24 smithi203 ceph-mon[36345]: pgmap v424: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T01:38:25.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:24 smithi046 ceph-mon[36108]: pgmap v424: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T01:38:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:24 smithi007 ceph-mon[31996]: pgmap v424: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T01:38:26.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:25 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:38:26.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:25 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:38:26.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:25 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:38:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:26 smithi203 ceph-mon[36345]: pgmap v425: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T01:38:27.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:26 smithi046 ceph-mon[36108]: pgmap v425: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T01:38:27.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:26 smithi007 ceph-mon[31996]: pgmap v425: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T01:38:29.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:28 smithi007 ceph-mon[31996]: pgmap v426: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T01:38:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:28 smithi203 ceph-mon[36345]: pgmap v426: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T01:38:29.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:28 smithi046 ceph-mon[36108]: pgmap v426: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T01:38:31.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:30 smithi203 ceph-mon[36345]: pgmap v427: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:31.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:30 smithi046 ceph-mon[36108]: pgmap v427: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:31.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:30 smithi007 ceph-mon[31996]: pgmap v427: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:32 smithi203 ceph-mon[36345]: pgmap v428: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:38:33.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:32 smithi046 ceph-mon[36108]: pgmap v428: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:38:33.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:32 smithi007 ceph-mon[31996]: pgmap v428: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:38:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:34 smithi203 ceph-mon[36345]: pgmap v429: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:35.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:34 smithi046 ceph-mon[36108]: pgmap v429: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:35.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:34 smithi007 ceph-mon[31996]: pgmap v429: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:36 smithi203 ceph-mon[36345]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:36 smithi046 ceph-mon[36108]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:37.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:36 smithi007 ceph-mon[31996]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:38 smithi203 ceph-mon[36345]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:38 smithi046 ceph-mon[36108]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:38 smithi007 ceph-mon[31996]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:40 smithi203 ceph-mon[36345]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:41.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:40 smithi046 ceph-mon[36108]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:41.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:40 smithi007 ceph-mon[31996]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:41.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:38:41.656 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:38:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.889+0000 7fb1b9800640 1 -- 172.21.15.7:0/732756852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b41049e0 msgr2=0x7fb1b4106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.889+0000 7fb1b9800640 1 --2- 172.21.15.7:0/732756852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b41049e0 0x7fb1b4106dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009900 tx=0x7fb1a002bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.890+0000 7fb1b9800640 1 -- 172.21.15.7:0/732756852 shutdown_connections 2024-03-18T01:38:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.890+0000 7fb1b9800640 1 --2- 172.21.15.7:0/732756852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b4107310 0x7fb1b4109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.890+0000 7fb1b9800640 1 --2- 172.21.15.7:0/732756852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b41049e0 0x7fb1b4106dd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.890+0000 7fb1b9800640 1 --2- 172.21.15.7:0/732756852 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1b41020b0 0x7fb1b41044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.890+0000 7fb1b9800640 1 -- 172.21.15.7:0/732756852 >> 172.21.15.7:0/732756852 conn(0x7fb1b40fbac0 msgr2=0x7fb1b40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.890+0000 7fb1b9800640 1 -- 172.21.15.7:0/732756852 shutdown_connections 2024-03-18T01:38:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.890+0000 7fb1b9800640 1 -- 172.21.15.7:0/732756852 wait complete. 2024-03-18T01:38:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.891+0000 7fb1b9800640 1 Processor -- start 2024-03-18T01:38:41.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.891+0000 7fb1b9800640 1 -- start start 2024-03-18T01:38:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b9800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1b41020b0 0x7fb1b4199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b9800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b41049e0 0x7fb1b419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b9800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b4107310 0x7fb1b41a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b9800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b410d2e0 con 0x7fb1b41020b0 2024-03-18T01:38:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b9800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b410d160 con 0x7fb1b41049e0 2024-03-18T01:38:41.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b9800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b410d460 con 0x7fb1b4107310 2024-03-18T01:38:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1b41020b0 0x7fb1b4199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1b41020b0 0x7fb1b4199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53304/0 (socket says 172.21.15.7:53304) 2024-03-18T01:38:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b3e00640 1 -- 172.21.15.7:0/1053282666 learned_addr learned my addr 172.21.15.7:0/1053282666 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:41.893 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.892+0000 7fb1b8e00640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b4107310 0x7fb1b41a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.893+0000 7fb1b3400640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b41049e0 0x7fb1b419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.893+0000 7fb1b8e00640 1 -- 172.21.15.7:0/1053282666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b41049e0 msgr2=0x7fb1b419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.893+0000 7fb1b8e00640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b41049e0 0x7fb1b419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.893+0000 7fb1b8e00640 1 -- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1b41020b0 msgr2=0x7fb1b4199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.893+0000 7fb1b8e00640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1b41020b0 0x7fb1b4199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.893+0000 7fb1b8e00640 1 -- 172.21.15.7:0/1053282666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b41a17b0 con 0x7fb1b4107310 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.893+0000 7fb1b8e00640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b4107310 0x7fb1b41a1140 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac00ec50 tx=0x7fb1ac00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.893+0000 7fb1b0c00640 1 -- 172.21.15.7:0/1053282666 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac003b70 con 0x7fb1b4107310 2024-03-18T01:38:41.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.894+0000 7fb1b0c00640 1 -- 172.21.15.7:0/1053282666 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1ac003d10 con 0x7fb1b4107310 2024-03-18T01:38:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.894+0000 7fb1b9800640 1 -- 172.21.15.7:0/1053282666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b41a1aa0 con 0x7fb1b4107310 2024-03-18T01:38:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.894+0000 7fb1b0c00640 1 -- 172.21.15.7:0/1053282666 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ac010930 con 0x7fb1b4107310 2024-03-18T01:38:41.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.894+0000 7fb1b9800640 1 -- 172.21.15.7:0/1053282666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b41a1f60 con 0x7fb1b4107310 2024-03-18T01:38:41.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.900+0000 7fb1bdb92740 1 Processor -- start 2024-03-18T01:38:41.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.900+0000 7fb1b0c00640 1 -- 172.21.15.7:0/1053282666 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fb1ac040ba0 con 0x7fb1b4107310 2024-03-18T01:38:41.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.900+0000 7fb1bdb92740 1 -- start start 2024-03-18T01:38:41.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934093a10 0x561934093e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x561934098f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934099570 0x56193409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561933ff81f0 con 0x561934093a10 2024-03-18T01:38:41.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561933fa0a80 con 0x561934094400 2024-03-18T01:38:41.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561933ff4290 con 0x561934099570 2024-03-18T01:38:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1b3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934093a10 0x561934093e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1b3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x561934098f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1b3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934093a10 0x561934093e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53316/0 (socket says 172.21.15.7:53316) 2024-03-18T01:38:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1b3400640 1 -- 172.21.15.7:0/961674590 learned_addr learned my addr 172.21.15.7:0/961674590 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:41.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.901+0000 7fb1b3400640 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934099570 0x56193409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.902+0000 7fb1b0c00640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1980a2900 0x7fb1980a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.902+0000 7fb1b0c00640 1 -- 172.21.15.7:0/1053282666 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb1ac0cead0 con 0x7fb1b4107310 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.902+0000 7fb1b3400640 1 -- 172.21.15.7:0/961674590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934099570 msgr2=0x56193409ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.902+0000 7fb1b3400640 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934099570 0x56193409ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.902+0000 7fb1b3400640 1 -- 172.21.15.7:0/961674590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 msgr2=0x561934098f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.902+0000 7fb1b3400640 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x561934098f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.902+0000 7fb1b3400640 1 -- 172.21.15.7:0/961674590 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561933dd67d0 con 0x561934093a10 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.902+0000 7fb1b3400640 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934093a10 0x561934093e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb1a002c250 tx=0x7fb1a0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91aac56afe6532e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.903+0000 7fb1b8e00640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1980a2900 0x7fb1980a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.903+0000 7fb1a9600640 1 -- 172.21.15.7:0/961674590 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0004020 con 0x561934093a10 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.903+0000 7fb1a9600640 1 -- 172.21.15.7:0/961674590 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a00041c0 con 0x561934093a10 2024-03-18T01:38:41.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.903+0000 7fb1a9600640 1 -- 172.21.15.7:0/961674590 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0034730 con 0x561934093a10 2024-03-18T01:38:41.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1b3e00640 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x561934098f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 -- 172.21.15.7:0/961674590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934093a10 msgr2=0x561934093e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934093a10 0x561934093e30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb1a002c250 tx=0x7fb1a0005fd0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1b8e00640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1980a2900 0x7fb1980a4dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac004af0 tx=0x7fb1ac004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 -- 172.21.15.7:0/961674590 shutdown_connections 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934099570 0x56193409ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x561934098f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/961674590 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934093a10 0x561934093e30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 -- 172.21.15.7:0/961674590 >> 172.21.15.7:0/961674590 conn(0x56193408ef80 msgr2=0x56193408f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 -- 172.21.15.7:0/961674590 shutdown_connections 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.904+0000 7fb1bdb92740 1 -- 172.21.15.7:0/961674590 wait complete. 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.905+0000 7fb1bdb92740 1 -- start start 2024-03-18T01:38:41.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.905+0000 7fb1bdb92740 1 Processor -- start 2024-03-18T01:38:41.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934093a10 0x56193409e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934094400 0x56193409e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x5619340a5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56193409c4c0 con 0x561934099570 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561933fb3ed0 con 0x561934093a10 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56193409c640 con 0x561934094400 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1bdb92740 10 client.0 _ll_register_callbacks cb 0x561933fef220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1b3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934094400 0x56193409e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1b3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934094400 0x56193409e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34442/0 (socket says 172.21.15.7:34442) 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.906+0000 7fb1b3e00640 1 -- 172.21.15.7:0/11765612 learned_addr learned my addr 172.21.15.7:0/11765612 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3400640 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x5619340a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3400640 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934093a10 0x56193409e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3e00640 1 -- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934093a10 msgr2=0x56193409e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3e00640 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934093a10 0x56193409e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3e00640 1 -- 172.21.15.7:0/11765612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 msgr2=0x5619340a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3e00640 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x5619340a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3e00640 1 -- 172.21.15.7:0/11765612 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561933d0c3f0 con 0x561934094400 2024-03-18T01:38:41.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3400640 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934093a10 0x56193409e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb1b3e00640 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934094400 0x56193409e550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4012770 tx=0x7fb1a4012c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44bc5ad540ee8dac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.907+0000 7fb19e000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.908+0000 7fb19e000640 1 -- 172.21.15.7:0/11765612 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a400f940 con 0x561934094400 2024-03-18T01:38:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.908+0000 7fb1bdb92740 1 -- 172.21.15.7:0/11765612 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1a00041c0 con 0x561934094400 2024-03-18T01:38:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.908+0000 7fb19e000640 1 -- 172.21.15.7:0/11765612 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb1a400fae0 con 0x561934094400 2024-03-18T01:38:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.908+0000 7fb19e000640 1 -- 172.21.15.7:0/11765612 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb1a400fce0 con 0x561934094400 2024-03-18T01:38:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.911+0000 7fb1bdb92740 10 client.25627 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:41.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.911+0000 7fb1bdb92740 1 -- 172.21.15.7:0/11765612 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561933ccc0f0 con 0x561934094400 2024-03-18T01:38:41.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.911+0000 7fb19e000640 1 -- 172.21.15.7:0/11765612 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a00041c0 con 0x561934094400 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.911+0000 7fb19e000640 1 -- 172.21.15.7:0/11765612 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb1a401c420 con 0x561934094400 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.911+0000 7fb1bdb92740 10 client.25627 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.911+0000 7fb1bdb92740 10 client.25627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.912+0000 7fb1bdb92740 20 client.25627 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.912+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5619340a9f30 0x5619340ac370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.912+0000 7fb1bdb92740 4 client.25627 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.912+0000 7fb1bdb92740 1 -- 172.21.15.7:0/11765612 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56193400bbe0 con 0x5619340a9f30 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.912+0000 7fb1b3400640 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5619340a9f30 0x5619340ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.912+0000 7fb1b3400640 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5619340a9f30 0x5619340ac370 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a61ca9a92f1d3d6e server_cookie=b2c41e7488492fbc in_seq=0 out_seq=0 2024-03-18T01:38:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.912+0000 7fb19e000640 10 client.25627 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.913+0000 7fb19e000640 1 -- 172.21.15.7:0/11765612 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56193400bbe0 con 0x5619340a9f30 2024-03-18T01:38:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.913+0000 7fb19e000640 10 client.25627 handle_command_reply: tid=0 2024-03-18T01:38:41.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.913+0000 7fb1bdb92740 1 client.25627 shutdown 2024-03-18T01:38:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 -- 172.21.15.7:0/11765612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934094400 msgr2=0x56193409e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934094400 0x56193409e550 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb1a4012770 tx=0x7fb1a4012c30 comp rx=0 tx=0).stop 2024-03-18T01:38:41.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 -- 172.21.15.7:0/11765612 shutdown_connections 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5619340a9f30 0x5619340ac370 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x5619340a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934094400 0x56193409e550 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/11765612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934093a10 0x56193409e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 -- 172.21.15.7:0/11765612 >> 172.21.15.7:0/11765612 conn(0x56193408ef80 msgr2=0x56193408e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb19e000640 0 client.25627 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 -- 172.21.15.7:0/11765612 shutdown_connections 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 1 -- 172.21.15.7:0/11765612 wait complete. 2024-03-18T01:38:41.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.914+0000 7fb1bdb92740 20 client.25627 trim_cache size 0 max 16384 2024-03-18T01:38:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.951+0000 7fb1bdb92740 1 Processor -- start 2024-03-18T01:38:41.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.951+0000 7fb1bdb92740 1 -- start start 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.951+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x56193409d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.951+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x56193409c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41ac3a0 con 0x561934099570 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41adee0 con 0x561934094400 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41aca70 con 0x561934093a10 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55154/0 (socket says 172.21.15.7:55154) 2024-03-18T01:38:41.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3e00640 1 -- 172.21.15.7:0/1451117808 learned_addr learned my addr 172.21.15.7:0/1451117808 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:41.953 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3400640 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x56193409d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3400640 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x56193409c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3400640 1 -- 172.21.15.7:0/1451117808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 msgr2=0x5619340a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3400640 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3400640 1 -- 172.21.15.7:0/1451117808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 msgr2=0x56193409c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3400640 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x56193409c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.952+0000 7fb1b3400640 1 -- 172.21.15.7:0/1451117808 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619340947e0 con 0x561934093a10 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.953+0000 7fb1b3400640 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x56193409d630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0005ec0 tx=0x7fb1a0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b94f0b8503c12eb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.953+0000 7fb1a8c00640 1 -- 172.21.15.7:0/1451117808 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0055070 con 0x561934093a10 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.954+0000 7fb1a8c00640 1 -- 172.21.15.7:0/1451117808 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a0048040 con 0x561934093a10 2024-03-18T01:38:41.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.954+0000 7fb1a8c00640 1 -- 172.21.15.7:0/1451117808 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0005790 con 0x561934093a10 2024-03-18T01:38:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 -- 172.21.15.7:0/1451117808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 msgr2=0x56193409d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x56193409d630 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0005ec0 tx=0x7fb1a0000bc0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 -- 172.21.15.7:0/1451117808 shutdown_connections 2024-03-18T01:38:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x56193409c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/1451117808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x56193409d630 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 -- 172.21.15.7:0/1451117808 >> 172.21.15.7:0/1451117808 conn(0x56193408ef80 msgr2=0x561934095a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:41.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 -- 172.21.15.7:0/1451117808 shutdown_connections 2024-03-18T01:38:41.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 -- 172.21.15.7:0/1451117808 wait complete. 2024-03-18T01:38:41.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.955+0000 7fb1bdb92740 1 -- start start 2024-03-18T01:38:41.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.956+0000 7fb1bdb92740 1 Processor -- start 2024-03-18T01:38:41.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.956+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x5619340a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.956+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.956+0000 7fb1bdb92740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x5619340a4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.956+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41ad810 con 0x561934099570 2024-03-18T01:38:41.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.956+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41ad140 con 0x561934094400 2024-03-18T01:38:41.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.956+0000 7fb1bdb92740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b41aa380 con 0x561934093a10 2024-03-18T01:38:41.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.956+0000 7fb1bdb92740 10 client.0 _ll_register_callbacks cb 0x7fb1a4009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:41.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x5619340a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x5619340a2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34458/0 (socket says 172.21.15.7:34458) 2024-03-18T01:38:41.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 -- 172.21.15.7:0/72845182 learned_addr learned my addr 172.21.15.7:0/72845182 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:41.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x5619340a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3e00640 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 -- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 msgr2=0x5619340a3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 -- 172.21.15.7:0/72845182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 msgr2=0x5619340a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x5619340a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3400640 1 -- 172.21.15.7:0/72845182 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619340a4610 con 0x561934093a10 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.957+0000 7fb1b3e00640 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.958+0000 7fb1b3400640 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x5619340a2ad0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00096c0 tx=0x7fb1a0041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6acb24bfd28853a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.958+0000 7fb19e000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.958+0000 7fb19e000640 1 -- 172.21.15.7:0/72845182 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0055070 con 0x561934093a10 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.958+0000 7fb19e000640 1 -- 172.21.15.7:0/72845182 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb1a00503f0 con 0x561934093a10 2024-03-18T01:38:41.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.958+0000 7fb1bdb92740 1 -- 172.21.15.7:0/72845182 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1a0048040 con 0x561934093a10 2024-03-18T01:38:41.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.959+0000 7fb19e000640 1 -- 172.21.15.7:0/72845182 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb1a0047040 con 0x561934093a10 2024-03-18T01:38:41.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.959+0000 7fb1bdb92740 10 client.25633 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:41.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.959+0000 7fb1bdb92740 1 -- 172.21.15.7:0/72845182 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5619340a5920 con 0x561934093a10 2024-03-18T01:38:41.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.959+0000 7fb19e000640 1 -- 172.21.15.7:0/72845182 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1a0048040 con 0x561934093a10 2024-03-18T01:38:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.959+0000 7fb19e000640 1 -- 172.21.15.7:0/72845182 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb1a004d040 con 0x561934093a10 2024-03-18T01:38:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.959+0000 7fb1bdb92740 10 client.25633 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.959+0000 7fb1bdb92740 10 client.25633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.959+0000 7fb1bdb92740 20 client.25633 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.960+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x561934110470 0x561934112890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.960+0000 7fb1bdb92740 4 client.25633 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:38:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.960+0000 7fb1bdb92740 1 -- 172.21.15.7:0/72845182 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56193409de00 con 0x561934110470 2024-03-18T01:38:41.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.960+0000 7fb1b3e00640 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x561934110470 0x561934112890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.961+0000 7fb1b3e00640 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x561934110470 0x561934112890 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc98f020e719aee5 server_cookie=4bede6c7afde20a6 in_seq=0 out_seq=0 2024-03-18T01:38:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.961+0000 7fb19e000640 10 client.25633 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.961+0000 7fb19e000640 1 -- 172.21.15.7:0/72845182 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56193409de00 con 0x561934110470 2024-03-18T01:38:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.961+0000 7fb19e000640 10 client.25633 handle_command_reply: tid=0 2024-03-18T01:38:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.961+0000 7fb1bdb92740 1 client.25633 shutdown 2024-03-18T01:38:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb1bdb92740 1 -- 172.21.15.7:0/72845182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 msgr2=0x5619340a2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x5619340a2ad0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00096c0 tx=0x7fb1a0041040 comp rx=0 tx=0).stop 2024-03-18T01:38:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb1bdb92740 1 -- 172.21.15.7:0/72845182 shutdown_connections 2024-03-18T01:38:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x561934110470 0x561934112890 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561934099570 0x5619340a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561934094400 0x5619340a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb1bdb92740 1 --2- 172.21.15.7:0/72845182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561934093a10 0x5619340a2ad0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb1bdb92740 1 -- 172.21.15.7:0/72845182 >> 172.21.15.7:0/72845182 conn(0x56193408ef80 msgr2=0x56193408d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:41.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.962+0000 7fb19e000640 0 client.25633 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:41.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.963+0000 7fb1bdb92740 1 -- 172.21.15.7:0/72845182 shutdown_connections 2024-03-18T01:38:41.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.963+0000 7fb1bdb92740 1 -- 172.21.15.7:0/72845182 wait complete. 2024-03-18T01:38:41.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.963+0000 7fb1bdb92740 20 client.25633 trim_cache size 0 max 16384 2024-03-18T01:38:41.964 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:38:41.964 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-03-18T01:38:41.965 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-03-18T01:38:41.965 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:38:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 -- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1980a2900 msgr2=0x7fb1980a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1980a2900 0x7fb1980a4dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac004af0 tx=0x7fb1ac004a40 comp rx=0 tx=0).stop 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 -- 172.21.15.7:0/1053282666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b4107310 msgr2=0x7fb1b41a1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b4107310 0x7fb1b41a1140 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac00ec50 tx=0x7fb1ac00c6a0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 -- 172.21.15.7:0/1053282666 shutdown_connections 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1b4107310 0x7fb1b41a1140 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1b41049e0 0x7fb1b419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb1980a2900 0x7fb1980a4dc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 --2- 172.21.15.7:0/1053282666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb1b41020b0 0x7fb1b4199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 -- 172.21.15.7:0/1053282666 >> 172.21.15.7:0/1053282666 conn(0x7fb1b40fbac0 msgr2=0x7fb1b41017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.965+0000 7fb1b9800640 1 -- 172.21.15.7:0/1053282666 shutdown_connections 2024-03-18T01:38:41.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:41.966+0000 7fb1b9800640 1 -- 172.21.15.7:0/1053282666 wait complete. 2024-03-18T01:38:41.983 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T01:38:41.983 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:38:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.201+0000 7f1fa2600640 1 -- 172.21.15.7:0/1938980883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c104a00 msgr2=0x7f1f9c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.201+0000 7f1fa2600640 1 --2- 172.21.15.7:0/1938980883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c104a00 0x7f1f9c104e80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009900 tx=0x7f1f8802bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.201+0000 7f1fa2600640 1 -- 172.21.15.7:0/1938980883 shutdown_connections 2024-03-18T01:38:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.201+0000 7f1fa2600640 1 --2- 172.21.15.7:0/1938980883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f9c1053c0 0x7f1f9c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.202+0000 7f1fa2600640 1 --2- 172.21.15.7:0/1938980883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c104a00 0x7f1f9c104e80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.202+0000 7f1fa2600640 1 --2- 172.21.15.7:0/1938980883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f9c103810 0x7f1f9c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.202+0000 7f1fa2600640 1 -- 172.21.15.7:0/1938980883 >> 172.21.15.7:0/1938980883 conn(0x7f1f9c0fed80 msgr2=0x7f1f9c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.202+0000 7f1fa2600640 1 -- 172.21.15.7:0/1938980883 shutdown_connections 2024-03-18T01:38:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.202+0000 7f1fa2600640 1 -- 172.21.15.7:0/1938980883 wait complete. 2024-03-18T01:38:42.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.204+0000 7f1fa2600640 1 Processor -- start 2024-03-18T01:38:42.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.204+0000 7f1fa2600640 1 -- start start 2024-03-18T01:38:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.204+0000 7f1fa2600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c103810 0x7f1f9c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.204+0000 7f1fa2600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f9c104a00 0x7f1f9c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.204+0000 7f1fa2600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f9c1053c0 0x7f1f9c1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.204+0000 7f1fa2600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9c10f4c0 con 0x7f1f9c104a00 2024-03-18T01:38:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.204+0000 7f1fa2600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9c10f340 con 0x7f1f9c103810 2024-03-18T01:38:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.204+0000 7f1fa2600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9c10f640 con 0x7f1f9c1053c0 2024-03-18T01:38:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c103810 0x7f1f9c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c103810 0x7f1f9c1999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55190/0 (socket says 172.21.15.7:55190) 2024-03-18T01:38:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 -- 172.21.15.7:0/586631989 learned_addr learned my addr 172.21.15.7:0/586631989 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1f9be00640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f9c104a00 0x7f1f9c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1c00640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f9c1053c0 0x7f1f9c1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 -- 172.21.15.7:0/586631989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f9c1053c0 msgr2=0x7f1f9c1a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f9c1053c0 0x7f1f9c1a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 -- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f9c104a00 msgr2=0x7f1f9c199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f9c104a00 0x7f1f9c199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 -- 172.21.15.7:0/586631989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f9c1a1600 con 0x7f1f9c103810 2024-03-18T01:38:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1f9be00640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f9c104a00 0x7f1f9c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:42.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1c00640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f9c1053c0 0x7f1f9c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:42.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.205+0000 7f1fa1200640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c103810 0x7f1f9c1999e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1f9000d770 tx=0x7f1f9000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.206+0000 7f1f99600640 1 -- 172.21.15.7:0/586631989 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90017070 con 0x7f1f9c103810 2024-03-18T01:38:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.206+0000 7f1f99600640 1 -- 172.21.15.7:0/586631989 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f90009c30 con 0x7f1f9c103810 2024-03-18T01:38:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.206+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f9c1a18f0 con 0x7f1f9c103810 2024-03-18T01:38:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.207+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f9c1a1e40 con 0x7f1f9c103810 2024-03-18T01:38:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.208+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f9c04f540 con 0x7f1f9c103810 2024-03-18T01:38:42.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.208+0000 7f1f99600640 1 -- 172.21.15.7:0/586631989 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90003dd0 con 0x7f1f9c103810 2024-03-18T01:38:42.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.212+0000 7f1f99600640 1 -- 172.21.15.7:0/586631989 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f1f90010460 con 0x7f1f9c103810 2024-03-18T01:38:42.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.214+0000 7f1f99600640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1f840a2a10 0x7f1f840a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.214+0000 7f1f99600640 1 -- 172.21.15.7:0/586631989 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1f900cec90 con 0x7f1f9c103810 2024-03-18T01:38:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.215+0000 7f1f99600640 1 -- 172.21.15.7:0/586631989 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f90009de0 con 0x7f1f9c103810 2024-03-18T01:38:42.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.215+0000 7f1f9be00640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1f840a2a10 0x7f1f840a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.216+0000 7f1f9be00640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1f840a2a10 0x7f1f840a4ed0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1f8802efd0 tx=0x7f1f88005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:42.523 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.523+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f9c10aa80 con 0x7f1f9c103810 2024-03-18T01:38:42.524 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.524+0000 7f1f99600640 1 -- 172.21.15.7:0/586631989 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11918 (secure 0 0 0) 0x7f1f900a3a90 con 0x7f1f9c103810 2024-03-18T01:38:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:38:42.524 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:37:59.950482+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24562,24596,24542,24544,24523]},"id":1}]} 2024-03-18T01:38:42.525 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 28 2024-03-18T01:38:42.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1f840a2a10 msgr2=0x7f1f840a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1f840a2a10 0x7f1f840a4ed0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1f8802efd0 tx=0x7f1f88005fd0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c103810 msgr2=0x7f1f9c1999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c103810 0x7f1f9c1999e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1f9000d770 tx=0x7f1f9000dc30 comp rx=0 tx=0).stop 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 shutdown_connections 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f9c1053c0 0x7f1f9c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1f840a2a10 0x7f1f840a4ed0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1f9c104a00 0x7f1f9c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 --2- 172.21.15.7:0/586631989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f9c103810 0x7f1f9c1999e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.527+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 >> 172.21.15.7:0/586631989 conn(0x7f1f9c0fed80 msgr2=0x7f1f9c107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:42.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.528+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 shutdown_connections 2024-03-18T01:38:42.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.528+0000 7f1fa2600640 1 -- 172.21.15.7:0/586631989 wait complete. 2024-03-18T01:38:42.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T01:38:42.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.766+0000 7f61c5e00640 1 -- 172.21.15.7:0/121450714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01049a0 msgr2=0x7f61c0106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.766+0000 7f61c5e00640 1 --2- 172.21.15.7:0/121450714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01049a0 0x7f61c0106d90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009900 tx=0x7f61ac02bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.766+0000 7f61c5e00640 1 -- 172.21.15.7:0/121450714 shutdown_connections 2024-03-18T01:38:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.766+0000 7f61c5e00640 1 --2- 172.21.15.7:0/121450714 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61c01072d0 0x7f61c0109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.766+0000 7f61c5e00640 1 --2- 172.21.15.7:0/121450714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01049a0 0x7f61c0106d90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.766+0000 7f61c5e00640 1 --2- 172.21.15.7:0/121450714 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61c0102070 0x7f61c0104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.766+0000 7f61c5e00640 1 -- 172.21.15.7:0/121450714 >> 172.21.15.7:0/121450714 conn(0x7f61c00fbaa0 msgr2=0x7f61c00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:42.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.767+0000 7f61c5e00640 1 -- 172.21.15.7:0/121450714 shutdown_connections 2024-03-18T01:38:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.767+0000 7f61c5e00640 1 -- 172.21.15.7:0/121450714 wait complete. 2024-03-18T01:38:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.768+0000 7f61c5e00640 1 Processor -- start 2024-03-18T01:38:42.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.768+0000 7f61c5e00640 1 -- start start 2024-03-18T01:38:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.769+0000 7f61c5e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61c0102070 0x7f61c0070160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.769+0000 7f61c5e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61c01049a0 0x7f61c00706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61c5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01072d0 0x7f61c01a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61c0102070 0x7f61c0070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61c0102070 0x7f61c0070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53386/0 (socket says 172.21.15.7:53386) 2024-03-18T01:38:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61c5e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f61c010d2a0 con 0x7f61c0102070 2024-03-18T01:38:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61c4a00640 1 -- 172.21.15.7:0/2324305161 learned_addr learned my addr 172.21.15.7:0/2324305161 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61c010d120 con 0x7f61c01072d0 2024-03-18T01:38:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61bfe00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61c01049a0 0x7f61c00706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61c5400640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01072d0 0x7f61c01a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.770+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61c010d420 con 0x7f61c01049a0 2024-03-18T01:38:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.771+0000 7f61c5400640 1 -- 172.21.15.7:0/2324305161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61c01049a0 msgr2=0x7f61c00706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.771+0000 7f61c5400640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61c01049a0 0x7f61c00706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.771+0000 7f61c5400640 1 -- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61c0102070 msgr2=0x7f61c0070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.771+0000 7f61c5400640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61c0102070 0x7f61c0070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.771+0000 7f61c5400640 1 -- 172.21.15.7:0/2324305161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c01a7ef0 con 0x7f61c01072d0 2024-03-18T01:38:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.771+0000 7f61bfe00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61c01049a0 0x7f61c00706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.771+0000 7f61c5400640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01072d0 0x7f61c01a79b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f61b800e940 tx=0x7f61b800ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:42.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.772+0000 7f61bd600640 1 -- 172.21.15.7:0/2324305161 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b8004d20 con 0x7f61c01072d0 2024-03-18T01:38:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.772+0000 7f61bd600640 1 -- 172.21.15.7:0/2324305161 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b8005020 con 0x7f61c01072d0 2024-03-18T01:38:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.772+0000 7f61bd600640 1 -- 172.21.15.7:0/2324305161 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b80116b0 con 0x7f61c01072d0 2024-03-18T01:38:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.772+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c01a80c0 con 0x7f61c01072d0 2024-03-18T01:38:42.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.772+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c01a84d0 con 0x7f61c01072d0 2024-03-18T01:38:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.776+0000 7f61ca188740 1 Processor -- start 2024-03-18T01:38:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.777+0000 7f61bd600640 1 -- 172.21.15.7:0/2324305161 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f61b80054d0 con 0x7f61c01072d0 2024-03-18T01:38:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.777+0000 7f61ca188740 1 -- start start 2024-03-18T01:38:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.777+0000 7f61ca188740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f4497eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.777+0000 7f61ca188740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4498480 0x5591f449cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.777+0000 7f61ca188740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 0x5591f449fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.777+0000 7f61ca188740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5591f445f3b0 con 0x5591f4498480 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.777+0000 7f61ca188740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5591f43fc450 con 0x5591f4497a90 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.777+0000 7f61ca188740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5591f43a4cb0 con 0x5591f449d5f0 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.778+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4498480 0x5591f449cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.778+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4498480 0x5591f449cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53398/0 (socket says 172.21.15.7:53398) 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.778+0000 7f61c4a00640 1 -- 172.21.15.7:0/470497519 learned_addr learned my addr 172.21.15.7:0/470497519 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.778+0000 7f61bfe00640 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f4497eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61c4a00640 1 -- 172.21.15.7:0/470497519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 msgr2=0x5591f449fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61bfe00640 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 0x5591f449fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61c4a00640 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 0x5591f449fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61c4a00640 1 -- 172.21.15.7:0/470497519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 msgr2=0x5591f4497eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61c4a00640 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f4497eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61c4a00640 1 -- 172.21.15.7:0/470497519 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591f43c61f0 con 0x5591f4498480 2024-03-18T01:38:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61bfe00640 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f4497eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61c4a00640 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4498480 0x5591f449cf80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f61b400b890 tx=0x7f61b400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1786f5e8deaca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61bd600640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61a80a2870 0x7f61a80a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.779+0000 7f61b1c00640 1 -- 172.21.15.7:0/470497519 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4003dd0 con 0x5591f4498480 2024-03-18T01:38:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.780+0000 7f61bd600640 1 -- 172.21.15.7:0/2324305161 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f61b80cedf0 con 0x7f61c01072d0 2024-03-18T01:38:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.780+0000 7f61b1c00640 1 -- 172.21.15.7:0/470497519 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b4037070 con 0x5591f4498480 2024-03-18T01:38:42.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.780+0000 7f61b1c00640 1 -- 172.21.15.7:0/470497519 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4033890 con 0x5591f4498480 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61ca188740 1 -- 172.21.15.7:0/470497519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4498480 msgr2=0x5591f449cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61ca188740 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4498480 0x5591f449cf80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f61b400b890 tx=0x7f61b400bd50 comp rx=0 tx=0).stop 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61bfe00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61a80a2870 0x7f61a80a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61ca188740 1 -- 172.21.15.7:0/470497519 shutdown_connections 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61ca188740 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 0x5591f449fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61ca188740 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4498480 0x5591f449cf80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61ca188740 1 --2- 172.21.15.7:0/470497519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f4497eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61ca188740 1 -- 172.21.15.7:0/470497519 >> 172.21.15.7:0/470497519 conn(0x5591f4493000 msgr2=0x5591f4493400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.781+0000 7f61ca188740 1 -- 172.21.15.7:0/470497519 shutdown_connections 2024-03-18T01:38:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.782+0000 7f61ca188740 1 -- 172.21.15.7:0/470497519 wait complete. 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.782+0000 7f61bfe00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61a80a2870 0x7f61a80a4d30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f61ac02c180 tx=0x7f61ac02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.782+0000 7f61ca188740 1 -- start start 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.782+0000 7f61ca188740 1 Processor -- start 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61ca188740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61ca188740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61ca188740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61ca188740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5591f43b8110 con 0x5591f449d5f0 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61ca188740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5591f43f84f0 con 0x5591f4497a90 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61ca188740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5591f44a05a0 con 0x5591f4498480 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61ca188740 10 client.0 _ll_register_callbacks cb 0x5591f43a5b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55206/0 (socket says 172.21.15.7:55206) 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61c4a00640 1 -- 172.21.15.7:0/2880581151 learned_addr learned my addr 172.21.15.7:0/2880581151 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:42.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61bfe00640 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61c5400640 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61c4a00640 1 -- 172.21.15.7:0/2880581151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 msgr2=0x5591f44a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.783+0000 7f61c4a00640 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.784+0000 7f61c4a00640 1 -- 172.21.15.7:0/2880581151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 msgr2=0x5591f44a9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.784+0000 7f61c4a00640 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.784+0000 7f61c4a00640 1 -- 172.21.15.7:0/2880581151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591f41103f0 con 0x5591f4497a90 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.784+0000 7f61c5400640 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.784+0000 7f61c4a00640 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a1d20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f61b403b000 tx=0x7f61b4003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee4fd9d7bf920560 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.784+0000 7f61a6000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.784+0000 7f61a6000640 1 -- 172.21.15.7:0/2880581151 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4049020 con 0x5591f4497a90 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.785+0000 7f61a6000640 1 -- 172.21.15.7:0/2880581151 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f61b4005450 con 0x5591f4497a90 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.785+0000 7f61ca188740 1 -- 172.21.15.7:0/2880581151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b4037070 con 0x5591f4497a90 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.785+0000 7f61a6000640 1 -- 172.21.15.7:0/2880581151 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f61b4005650 con 0x5591f4497a90 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.785+0000 7f61ca188740 10 client.25658 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:42.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.785+0000 7f61ca188740 1 -- 172.21.15.7:0/2880581151 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5591f41da7d0 con 0x5591f4497a90 2024-03-18T01:38:42.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.786+0000 7f61a6000640 1 -- 172.21.15.7:0/2880581151 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61b4037070 con 0x5591f4497a90 2024-03-18T01:38:42.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.787+0000 7f61a6000640 1 -- 172.21.15.7:0/2880581151 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f61b4004be0 con 0x5591f4497a90 2024-03-18T01:38:42.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.787+0000 7f61ca188740 10 client.25658 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:42.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.787+0000 7f61ca188740 10 client.25658 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:38:42.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.787+0000 7f61ca188740 20 client.25658 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.787+0000 7f61ca188740 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5591f44ad7d0 0x5591f44afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.787+0000 7f61ca188740 4 client.25658 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.787+0000 7f61ca188740 1 -- 172.21.15.7:0/2880581151 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591f4409990 con 0x5591f44ad7d0 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.787+0000 7f61bfe00640 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5591f44ad7d0 0x5591f44afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.788+0000 7f61bfe00640 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5591f44ad7d0 0x5591f44afbc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=460a0c48e331f4e5 server_cookie=296b8f940390ea91 in_seq=0 out_seq=0 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.788+0000 7f61a6000640 10 client.25658 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.788+0000 7f61a6000640 1 -- 172.21.15.7:0/2880581151 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5591f4409990 con 0x5591f44ad7d0 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.788+0000 7f61a6000640 10 client.25658 handle_command_reply: tid=0 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.788+0000 7f61ca188740 1 client.25658 shutdown 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 -- 172.21.15.7:0/2880581151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 msgr2=0x5591f44a1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a1d20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f61b403b000 tx=0x7f61b4003e00 comp rx=0 tx=0).stop 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 -- 172.21.15.7:0/2880581151 shutdown_connections 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5591f44ad7d0 0x5591f44afbc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 --2- 172.21.15.7:0/2880581151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a1d20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 -- 172.21.15.7:0/2880581151 >> 172.21.15.7:0/2880581151 conn(0x5591f4493000 msgr2=0x5591f4492220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61a6000640 0 client.25658 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:38:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 -- 172.21.15.7:0/2880581151 shutdown_connections 2024-03-18T01:38:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 1 -- 172.21.15.7:0/2880581151 wait complete. 2024-03-18T01:38:42.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.789+0000 7f61ca188740 20 client.25658 trim_cache size 0 max 16384 2024-03-18T01:38:42.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.804+0000 7f61ca188740 1 Processor -- start 2024-03-18T01:38:42.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.804+0000 7f61ca188740 1 -- start start 2024-03-18T01:38:42.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.804+0000 7f61ca188740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4497a90 0x5591f44a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.804+0000 7f61ca188740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4498480 0x5591f44a9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.804+0000 7f61ca188740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 0x5591f44a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.804+0000 7f61ca188740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f61c019ecc0 con 0x5591f4497a90 2024-03-18T01:38:42.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.804+0000 7f61ca188740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61c01a0800 con 0x5591f4498480 2024-03-18T01:38:42.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.804+0000 7f61ca188740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61c019f390 con 0x5591f449d5f0 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4498480 0x5591f44a9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4498480 0x5591f44a9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55214/0 (socket says 172.21.15.7:55214) 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 -- 172.21.15.7:0/1429047479 learned_addr learned my addr 172.21.15.7:0/1429047479 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61c4a00640 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4497a90 0x5591f44a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61c5400640 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 0x5591f44a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 -- 172.21.15.7:0/1429047479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 msgr2=0x5591f44a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 0x5591f44a1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 -- 172.21.15.7:0/1429047479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4497a90 msgr2=0x5591f44a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4497a90 0x5591f44a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 -- 172.21.15.7:0/1429047479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591f449a0d0 con 0x5591f4498480 2024-03-18T01:38:42.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61c4a00640 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4497a90 0x5591f44a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:42.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.805+0000 7f61bfe00640 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4498480 0x5591f44a9220 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61ac02c8f0 tx=0x7f61ac009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565fe13407f71ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:42.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.806+0000 7f61b1200640 1 -- 172.21.15.7:0/1429047479 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac004100 con 0x5591f4498480 2024-03-18T01:38:42.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.806+0000 7f61b1200640 1 -- 172.21.15.7:0/1429047479 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ac056740 con 0x5591f4498480 2024-03-18T01:38:42.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.806+0000 7f61b1200640 1 -- 172.21.15.7:0/1429047479 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac004eb0 con 0x5591f4498480 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.806+0000 7f61ca188740 1 -- 172.21.15.7:0/1429047479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4498480 msgr2=0x5591f44a9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.806+0000 7f61ca188740 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4498480 0x5591f44a9220 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61ac02c8f0 tx=0x7f61ac009260 comp rx=0 tx=0).stop 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.807+0000 7f61ca188740 1 -- 172.21.15.7:0/1429047479 shutdown_connections 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.807+0000 7f61ca188740 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f449d5f0 0x5591f44a1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.807+0000 7f61ca188740 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4498480 0x5591f44a9220 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.807+0000 7f61ca188740 1 --2- 172.21.15.7:0/1429047479 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f4497a90 0x5591f44a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.807+0000 7f61ca188740 1 -- 172.21.15.7:0/1429047479 >> 172.21.15.7:0/1429047479 conn(0x5591f4493000 msgr2=0x5591f44af7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.807+0000 7f61ca188740 1 -- 172.21.15.7:0/1429047479 shutdown_connections 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.807+0000 7f61ca188740 1 -- 172.21.15.7:0/1429047479 wait complete. 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.807+0000 7f61ca188740 1 -- start start 2024-03-18T01:38:42.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61ca188740 1 Processor -- start 2024-03-18T01:38:42.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61ca188740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61ca188740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61ca188740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61ca188740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f61c01a0130 con 0x5591f449d5f0 2024-03-18T01:38:42.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61ca188740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61c019fa60 con 0x5591f4497a90 2024-03-18T01:38:42.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61ca188740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61c019cca0 con 0x5591f4498480 2024-03-18T01:38:42.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61ca188740 10 client.0 _ll_register_callbacks cb 0x7f61b4004530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61c4a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55230/0 (socket says 172.21.15.7:55230) 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.808+0000 7f61c4a00640 1 -- 172.21.15.7:0/479577065 learned_addr learned my addr 172.21.15.7:0/479577065 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61bfe00640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61c5400640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61c4a00640 1 -- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 msgr2=0x5591f44a7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61c4a00640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61c4a00640 1 -- 172.21.15.7:0/479577065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 msgr2=0x5591f44a80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61c4a00640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61c4a00640 1 -- 172.21.15.7:0/479577065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591f44a86a0 con 0x5591f4497a90 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61bfe00640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61c5400640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61c4a00640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a6c20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f61b400bd90 tx=0x7f61b4003f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=445c021bd7adfb2a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:42.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.809+0000 7f61a6000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:38:42.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.810+0000 7f61a6000640 1 -- 172.21.15.7:0/479577065 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4049040 con 0x5591f4497a90 2024-03-18T01:38:42.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.810+0000 7f61ca188740 1 -- 172.21.15.7:0/479577065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61ac056740 con 0x5591f4497a90 2024-03-18T01:38:42.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.810+0000 7f61a6000640 1 -- 172.21.15.7:0/479577065 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f61b403c670 con 0x5591f4497a90 2024-03-18T01:38:42.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.810+0000 7f61a6000640 1 -- 172.21.15.7:0/479577065 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f61b4066050 con 0x5591f4497a90 2024-03-18T01:38:42.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.810+0000 7f61ca188740 10 client.25664 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:42.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.810+0000 7f61ca188740 1 -- 172.21.15.7:0/479577065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5591f451c4f0 con 0x5591f4497a90 2024-03-18T01:38:42.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.810+0000 7f61a6000640 1 -- 172.21.15.7:0/479577065 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ac056740 con 0x5591f4497a90 2024-03-18T01:38:42.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.811+0000 7f61a6000640 1 -- 172.21.15.7:0/479577065 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f61b4061030 con 0x5591f4497a90 2024-03-18T01:38:42.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.811+0000 7f61ca188740 10 client.25664 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:42.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.811+0000 7f61ca188740 10 client.25664 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:38:42.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.811+0000 7f61ca188740 20 client.25664 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.811+0000 7f61ca188740 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5591f451ce30 0x5591f451f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.811+0000 7f61ca188740 4 client.25664 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.811+0000 7f61ca188740 1 -- 172.21.15.7:0/479577065 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5591f4403ce0 con 0x5591f451ce30 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.811+0000 7f61bfe00640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5591f451ce30 0x5591f451f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61bfe00640 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5591f451ce30 0x5591f451f250 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4e2e362f3f170d60 server_cookie=d1d8a7798656fd5c in_seq=0 out_seq=0 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61a6000640 10 client.25664 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61a6000640 1 -- 172.21.15.7:0/479577065 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5591f4403ce0 con 0x5591f451ce30 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61a6000640 10 client.25664 handle_command_reply: tid=0 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61ca188740 1 client.25664 shutdown 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61ca188740 1 -- 172.21.15.7:0/479577065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 msgr2=0x5591f44a6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61ca188740 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a6c20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f61b400bd90 tx=0x7f61b4003f70 comp rx=0 tx=0).stop 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61ca188740 1 -- 172.21.15.7:0/479577065 shutdown_connections 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61ca188740 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5591f451ce30 0x5591f451f250 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.812+0000 7f61ca188740 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5591f449d5f0 0x5591f44a80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.813+0000 7f61ca188740 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5591f4498480 0x5591f44a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.813+0000 7f61ca188740 1 --2- 172.21.15.7:0/479577065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591f4497a90 0x5591f44a6c20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.813+0000 7f61ca188740 1 -- 172.21.15.7:0/479577065 >> 172.21.15.7:0/479577065 conn(0x5591f4493000 msgr2=0x5591f44a1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:42.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.813+0000 7f61a6000640 0 client.25664 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:38:42.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.813+0000 7f61ca188740 1 -- 172.21.15.7:0/479577065 shutdown_connections 2024-03-18T01:38:42.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.813+0000 7f61ca188740 1 -- 172.21.15.7:0/479577065 wait complete. 2024-03-18T01:38:42.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.813+0000 7f61ca188740 20 client.25664 trim_cache size 0 max 16384 2024-03-18T01:38:42.816 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61a80a2870 msgr2=0x7f61a80a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61a80a2870 0x7f61a80a4d30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f61ac02c180 tx=0x7f61ac02e040 comp rx=0 tx=0).stop 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01072d0 msgr2=0x7f61c01a79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01072d0 0x7f61c01a79b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f61b800e940 tx=0x7f61b800ee00 comp rx=0 tx=0).stop 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 shutdown_connections 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61c01072d0 0x7f61c01a79b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61c01049a0 0x7f61c00706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61a80a2870 0x7f61a80a4d30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 --2- 172.21.15.7:0/2324305161 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61c0102070 0x7f61c0070160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:42.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 >> 172.21.15.7:0/2324305161 conn(0x7f61c00fbaa0 msgr2=0x7f61c0102990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:42.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 shutdown_connections 2024-03-18T01:38:42.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:42.816+0000 7f61c5e00640 1 -- 172.21.15.7:0/2324305161 wait complete. 2024-03-18T01:38:42.835 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T01:38:43.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 -- 172.21.15.7:0/2343942218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8101620 msgr2=0x7f72d8101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 --2- 172.21.15.7:0/2343942218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8101620 0x7f72d8101a40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009900 tx=0x7f72cc02bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:43.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 -- 172.21.15.7:0/2343942218 shutdown_connections 2024-03-18T01:38:43.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 --2- 172.21.15.7:0/2343942218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d81031d0 0x7f72d8109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 --2- 172.21.15.7:0/2343942218 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8102810 0x7f72d8102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 --2- 172.21.15.7:0/2343942218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8101620 0x7f72d8101a40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 -- 172.21.15.7:0/2343942218 >> 172.21.15.7:0/2343942218 conn(0x7f72d80fcb90 msgr2=0x7f72d80fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 -- 172.21.15.7:0/2343942218 shutdown_connections 2024-03-18T01:38:43.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.035+0000 7f72de400640 1 -- 172.21.15.7:0/2343942218 wait complete. 2024-03-18T01:38:43.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.036+0000 7f72de400640 1 Processor -- start 2024-03-18T01:38:43.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.037+0000 7f72de400640 1 -- start start 2024-03-18T01:38:43.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.037+0000 7f72de400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8101620 0x7f72d8199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.037+0000 7f72de400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8102810 0x7f72d819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.037+0000 7f72de400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d81031d0 0x7f72d81a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.037+0000 7f72dd000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8101620 0x7f72d8199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72dd000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8101620 0x7f72d8199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53438/0 (socket says 172.21.15.7:53438) 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72dd000640 1 -- 172.21.15.7:0/1179384187 learned_addr learned my addr 172.21.15.7:0/1179384187 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f72d810d2e0 con 0x7f72d8101620 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72d810d160 con 0x7f72d8102810 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72d810d460 con 0x7f72d81031d0 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72d7e00640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8102810 0x7f72d819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72dda00640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d81031d0 0x7f72d81a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72dd000640 1 -- 172.21.15.7:0/1179384187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d81031d0 msgr2=0x7f72d81a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72dd000640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d81031d0 0x7f72d81a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72dd000640 1 -- 172.21.15.7:0/1179384187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8102810 msgr2=0x7f72d819a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72dd000640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8102810 0x7f72d819a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72dd000640 1 -- 172.21.15.7:0/1179384187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d81a1740 con 0x7f72d8101620 2024-03-18T01:38:43.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.038+0000 7f72d7e00640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8102810 0x7f72d819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:43.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.039+0000 7f72dd000640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8101620 0x7f72d8199b20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f72cc02efd0 tx=0x7f72cc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.039+0000 7f72d5600640 1 -- 172.21.15.7:0/1179384187 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc041070 con 0x7f72d8101620 2024-03-18T01:38:43.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.040+0000 7f72d5600640 1 -- 172.21.15.7:0/1179384187 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72cc0044d0 con 0x7f72d8101620 2024-03-18T01:38:43.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.040+0000 7f72d5600640 1 -- 172.21.15.7:0/1179384187 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc034440 con 0x7f72d8101620 2024-03-18T01:38:43.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.040+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d81a19d0 con 0x7f72d8101620 2024-03-18T01:38:43.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.040+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d81a1ee0 con 0x7f72d8101620 2024-03-18T01:38:43.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.044+0000 7f72e279b740 1 Processor -- start 2024-03-18T01:38:43.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.045+0000 7f72d5600640 1 -- 172.21.15.7:0/1179384187 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f72cc044070 con 0x7f72d8101620 2024-03-18T01:38:43.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.046+0000 7f72e279b740 1 -- start start 2024-03-18T01:38:43.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.046+0000 7f72e279b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06da90 0x5596bb06deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.047+0000 7f72e279b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 0x5596bb072f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.047+0000 7f72e279b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb0735f0 0x5596bb075a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.047+0000 7f72e279b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5596bb0353b0 con 0x5596bb06da90 2024-03-18T01:38:43.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.047+0000 7f72e279b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5596bafd2450 con 0x5596bb06e480 2024-03-18T01:38:43.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.047+0000 7f72e279b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5596baf7acb0 con 0x5596bb0735f0 2024-03-18T01:38:43.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.048+0000 7f72dd000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 0x5596bb072f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.048+0000 7f72dda00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06da90 0x5596bb06deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.048+0000 7f72dda00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06da90 0x5596bb06deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53446/0 (socket says 172.21.15.7:53446) 2024-03-18T01:38:43.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.048+0000 7f72dda00640 1 -- 172.21.15.7:0/1544490876 learned_addr learned my addr 172.21.15.7:0/1544490876 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.048+0000 7f72d7e00640 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb0735f0 0x5596bb075a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.048+0000 7f72d5600640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72c00a2870 0x7f72c00a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.049+0000 7f72d7e00640 1 -- 172.21.15.7:0/1544490876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 msgr2=0x5596bb072f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.049+0000 7f72d7e00640 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 0x5596bb072f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.049+0000 7f72d7e00640 1 -- 172.21.15.7:0/1544490876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06da90 msgr2=0x5596bb06deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.049+0000 7f72d7e00640 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06da90 0x5596bb06deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.049+0000 7f72d7e00640 1 -- 172.21.15.7:0/1544490876 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596baf9c1f0 con 0x5596bb0735f0 2024-03-18T01:38:43.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.049+0000 7f72dda00640 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06da90 0x5596bb06deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:43.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.050+0000 7f72d7e00640 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb0735f0 0x5596bb075a80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f72c4005d00 tx=0x7f72c4010930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a950398efebb8bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.050+0000 7f72ca000640 1 -- 172.21.15.7:0/1544490876 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c4011760 con 0x5596bb0735f0 2024-03-18T01:38:43.053 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.050+0000 7f72ca000640 1 -- 172.21.15.7:0/1544490876 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c4011de0 con 0x5596bb0735f0 2024-03-18T01:38:43.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.050+0000 7f72ca000640 1 -- 172.21.15.7:0/1544490876 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c401a380 con 0x5596bb0735f0 2024-03-18T01:38:43.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.050+0000 7f72d5600640 1 -- 172.21.15.7:0/1179384187 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72cc0f08d0 con 0x7f72d8101620 2024-03-18T01:38:43.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.051+0000 7f72e279b740 1 -- 172.21.15.7:0/1544490876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb0735f0 msgr2=0x5596bb075a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.051+0000 7f72e279b740 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb0735f0 0x5596bb075a80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f72c4005d00 tx=0x7f72c4010930 comp rx=0 tx=0).stop 2024-03-18T01:38:43.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.051+0000 7f72dda00640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72c00a2870 0x7f72c00a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.052+0000 7f72dda00640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72c00a2870 0x7f72c00a4d30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x5596bb06ece0 tx=0x7f72d0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.052+0000 7f72e279b740 1 -- 172.21.15.7:0/1544490876 shutdown_connections 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.052+0000 7f72e279b740 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb0735f0 0x5596bb075a80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.052+0000 7f72e279b740 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 0x5596bb072f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.052+0000 7f72e279b740 1 --2- 172.21.15.7:0/1544490876 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06da90 0x5596bb06deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.052+0000 7f72e279b740 1 -- 172.21.15.7:0/1544490876 >> 172.21.15.7:0/1544490876 conn(0x5596bb069000 msgr2=0x5596bb069400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.052+0000 7f72e279b740 1 -- 172.21.15.7:0/1544490876 shutdown_connections 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.052+0000 7f72e279b740 1 -- 172.21.15.7:0/1544490876 wait complete. 2024-03-18T01:38:43.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.053+0000 7f72e279b740 1 -- start start 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.053+0000 7f72e279b740 1 Processor -- start 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72e279b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb077d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72e279b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 0x5596bb078290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72dd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb077d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72dd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb077d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34598/0 (socket says 172.21.15.7:34598) 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72e279b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb0735f0 0x5596bb07f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72e279b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5596baf8e110 con 0x5596bb0735f0 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72e279b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5596bafce4f0 con 0x5596bb06e480 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72e279b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5596bb0765a0 con 0x5596bb06da90 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.054+0000 7f72e279b740 10 client.0 _ll_register_callbacks cb 0x5596baf7bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dd000640 1 -- 172.21.15.7:0/3598346300 learned_addr learned my addr 172.21.15.7:0/3598346300 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72d7e00640 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 0x5596bb078290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dda00640 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb0735f0 0x5596bb07f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dd000640 1 -- 172.21.15.7:0/3598346300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 msgr2=0x5596bb078290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dd000640 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 0x5596bb078290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dd000640 1 -- 172.21.15.7:0/3598346300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb0735f0 msgr2=0x5596bb07f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dd000640 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb0735f0 0x5596bb07f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dd000640 1 -- 172.21.15.7:0/3598346300 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596bace63f0 con 0x5596bb06da90 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dda00640 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb0735f0 0x5596bb07f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72dd000640 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb077d50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f72cc02f020 tx=0x7f72cc0bcd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be0bfdb0924bb84c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.055+0000 7f72bea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.056+0000 7f72bea00640 1 -- 172.21.15.7:0/3598346300 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc04b070 con 0x5596bb06da90 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.056+0000 7f72bea00640 1 -- 172.21.15.7:0/3598346300 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72cc0c5420 con 0x5596bb06da90 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.056+0000 7f72e279b740 1 -- 172.21.15.7:0/3598346300 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72c4011de0 con 0x5596bb06da90 2024-03-18T01:38:43.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.056+0000 7f72bea00640 1 -- 172.21.15.7:0/3598346300 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72cc0bcfc0 con 0x5596bb06da90 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.056+0000 7f72e279b740 10 client.25663 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.056+0000 7f72e279b740 1 -- 172.21.15.7:0/3598346300 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5596badb07d0 con 0x5596bb06da90 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.057+0000 7f72bea00640 1 -- 172.21.15.7:0/3598346300 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72cc0c1060 con 0x5596bb06da90 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.057+0000 7f72bea00640 1 -- 172.21.15.7:0/3598346300 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f72cc0d2020 con 0x5596bb06da90 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.057+0000 7f72e279b740 10 client.25663 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.057+0000 7f72e279b740 10 client.25663 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.057+0000 7f72e279b740 20 client.25663 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.057+0000 7f72e279b740 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5596bb08c390 0x5596bb08e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.057+0000 7f72e279b740 4 client.25663 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.058+0000 7f72e279b740 1 -- 172.21.15.7:0/3598346300 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f72c4011de0 con 0x5596bb08c390 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.058+0000 7f72d7e00640 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5596bb08c390 0x5596bb08e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.059+0000 7f72d7e00640 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5596bb08c390 0x5596bb08e810 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=46f441af0038d8f7 server_cookie=24387e0fe0d7076e in_seq=0 out_seq=0 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.059+0000 7f72bea00640 10 client.25663 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:38:43.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72bea00640 1 -- 172.21.15.7:0/3598346300 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f72c4011de0 con 0x5596bb08c390 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72bea00640 10 client.25663 handle_command_reply: tid=0 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 client.25663 shutdown 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 -- 172.21.15.7:0/3598346300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 msgr2=0x5596bb077d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb077d50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f72cc02f020 tx=0x7f72cc0bcd90 comp rx=0 tx=0).stop 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 -- 172.21.15.7:0/3598346300 shutdown_connections 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5596bb08c390 0x5596bb08e810 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb0735f0 0x5596bb07f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb06e480 0x5596bb078290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 --2- 172.21.15.7:0/3598346300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb077d50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72e279b740 1 -- 172.21.15.7:0/3598346300 >> 172.21.15.7:0/3598346300 conn(0x5596bb069000 msgr2=0x5596bb068220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.060+0000 7f72bea00640 0 client.25663 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.061+0000 7f72e279b740 1 -- 172.21.15.7:0/3598346300 shutdown_connections 2024-03-18T01:38:43.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.061+0000 7f72e279b740 1 -- 172.21.15.7:0/3598346300 wait complete. 2024-03-18T01:38:43.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.061+0000 7f72e279b740 20 client.25663 trim_cache size 0 max 16384 2024-03-18T01:38:43.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.089+0000 7f72e279b740 1 Processor -- start 2024-03-18T01:38:43.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.089+0000 7f72e279b740 1 -- start start 2024-03-18T01:38:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.089+0000 7f72e279b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb06c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.089+0000 7f72e279b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 0x5596bb07f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.089+0000 7f72e279b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb0930d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.089+0000 7f72e279b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f72d81a7ee0 con 0x5596bb06e480 2024-03-18T01:38:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.089+0000 7f72e279b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72d81a9a20 con 0x5596bb0735f0 2024-03-18T01:38:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.089+0000 7f72e279b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72d81a85b0 con 0x5596bb06da90 2024-03-18T01:38:43.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dda00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb0930d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb06c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72d7e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 0x5596bb07f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb06c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34600/0 (socket says 172.21.15.7:34600) 2024-03-18T01:38:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 -- 172.21.15.7:0/1662002081 learned_addr learned my addr 172.21.15.7:0/1662002081 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 -- 172.21.15.7:0/1662002081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 msgr2=0x5596bb0930d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 --2- 172.21.15.7:0/1662002081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb0930d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 -- 172.21.15.7:0/1662002081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 msgr2=0x5596bb07f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 --2- 172.21.15.7:0/1662002081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 0x5596bb07f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 -- 172.21.15.7:0/1662002081 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596bb069d70 con 0x5596bb06da90 2024-03-18T01:38:43.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.090+0000 7f72dd000640 1 --2- 172.21.15.7:0/1662002081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb06c970 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0b9990 tx=0x7f72cc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974524489b8ffd2a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.091+0000 7f72c9600640 1 -- 172.21.15.7:0/1662002081 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc04b070 con 0x5596bb06da90 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.091+0000 7f72c9600640 1 -- 172.21.15.7:0/1662002081 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72cc0bc410 con 0x5596bb06da90 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.091+0000 7f72c9600640 1 -- 172.21.15.7:0/1662002081 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0035f0 con 0x5596bb06da90 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 -- 172.21.15.7:0/1662002081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 msgr2=0x5596bb06c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 --2- 172.21.15.7:0/1662002081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb06c970 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0b9990 tx=0x7f72cc009260 comp rx=0 tx=0).stop 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 -- 172.21.15.7:0/1662002081 shutdown_connections 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 --2- 172.21.15.7:0/1662002081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb0930d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 --2- 172.21.15.7:0/1662002081 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 0x5596bb07f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 --2- 172.21.15.7:0/1662002081 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb06c970 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 -- 172.21.15.7:0/1662002081 >> 172.21.15.7:0/1662002081 conn(0x5596bb069000 msgr2=0x5596bb06c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 -- 172.21.15.7:0/1662002081 shutdown_connections 2024-03-18T01:38:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.092+0000 7f72e279b740 1 -- 172.21.15.7:0/1662002081 wait complete. 2024-03-18T01:38:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.093+0000 7f72e279b740 1 -- start start 2024-03-18T01:38:43.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.093+0000 7f72e279b740 1 Processor -- start 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72e279b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb07cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72e279b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 0x5596bb07d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72e279b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb07e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72e279b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f72d81a9350 con 0x5596bb06e480 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72e279b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72d81a8c80 con 0x5596bb0735f0 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72e279b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72d81a5f00 con 0x5596bb06da90 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72e279b740 10 client.0 _ll_register_callbacks cb 0x7f72cc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72dd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb07cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72dd000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb07cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34610/0 (socket says 172.21.15.7:34610) 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72dd000640 1 -- 172.21.15.7:0/840976217 learned_addr learned my addr 172.21.15.7:0/840976217 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72dda00640 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb07e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72d7e00640 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 0x5596bb07d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.094+0000 7f72dd000640 1 -- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 msgr2=0x5596bb07e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72dd000640 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb07e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72dd000640 1 -- 172.21.15.7:0/840976217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 msgr2=0x5596bb07d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72dd000640 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 0x5596bb07d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72dd000640 1 -- 172.21.15.7:0/840976217 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596bb07eb20 con 0x5596bb06da90 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72dda00640 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb07e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72dd000640 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb07cfe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f72cc02c180 tx=0x7f72cc02bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f671383fac86d16d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72bea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72bea00640 1 -- 172.21.15.7:0/840976217 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0d9060 con 0x5596bb06da90 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72bea00640 1 -- 172.21.15.7:0/840976217 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72cc003800 con 0x5596bb06da90 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.095+0000 7f72e279b740 1 -- 172.21.15.7:0/840976217 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72cc0bc410 con 0x5596bb06da90 2024-03-18T01:38:43.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72bea00640 1 -- 172.21.15.7:0/840976217 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f72cc04b030 con 0x5596bb06da90 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72e279b740 10 client.25669 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72e279b740 1 -- 172.21.15.7:0/840976217 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5596bb0f2ca0 con 0x5596bb06da90 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72bea00640 1 -- 172.21.15.7:0/840976217 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72cc0bc410 con 0x5596bb06da90 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72bea00640 1 -- 172.21.15.7:0/840976217 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f72cc0d2070 con 0x5596bb06da90 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72e279b740 10 client.25669 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72e279b740 10 client.25669 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72e279b740 20 client.25669 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72e279b740 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5596bb0f33a0 0x5596bb0f57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72e279b740 4 client.25669 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:38:43.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.096+0000 7f72e279b740 1 -- 172.21.15.7:0/840976217 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596bafdf990 con 0x5596bb0f33a0 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.097+0000 7f72d7e00640 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5596bb0f33a0 0x5596bb0f57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.097+0000 7f72d7e00640 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5596bb0f33a0 0x5596bb0f57c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a473475b86d928d3 server_cookie=51bfca33c5ea697c in_seq=0 out_seq=0 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.097+0000 7f72bea00640 10 client.25669 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.097+0000 7f72bea00640 1 -- 172.21.15.7:0/840976217 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596bafdf990 con 0x5596bb0f33a0 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.097+0000 7f72bea00640 10 client.25669 handle_command_reply: tid=0 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 client.25669 shutdown 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 -- 172.21.15.7:0/840976217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 msgr2=0x5596bb07cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb07cfe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f72cc02c180 tx=0x7f72cc02bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 -- 172.21.15.7:0/840976217 shutdown_connections 2024-03-18T01:38:43.101 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5596bb0f33a0 0x5596bb0f57c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5596bb0735f0 0x5596bb07e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5596bb06e480 0x5596bb07d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 --2- 172.21.15.7:0/840976217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5596bb06da90 0x5596bb07cfe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 -- 172.21.15.7:0/840976217 >> 172.21.15.7:0/840976217 conn(0x5596bb069000 msgr2=0x5596bb06caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72bea00640 0 client.25669 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 -- 172.21.15.7:0/840976217 shutdown_connections 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 1 -- 172.21.15.7:0/840976217 wait complete. 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.098+0000 7f72e279b740 20 client.25669 trim_cache size 0 max 16384 2024-03-18T01:38:43.102 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:38:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.099+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72c00a2870 msgr2=0x7f72c00a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.099+0000 7f72de400640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72c00a2870 0x7f72c00a4d30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x5596bb06ece0 tx=0x7f72d0004a40 comp rx=0 tx=0).stop 2024-03-18T01:38:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.099+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8101620 msgr2=0x7f72d8199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.099+0000 7f72de400640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8101620 0x7f72d8199b20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f72cc02efd0 tx=0x7f72cc005d50 comp rx=0 tx=0).stop 2024-03-18T01:38:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.100+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 shutdown_connections 2024-03-18T01:38:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.100+0000 7f72de400640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d81031d0 0x7f72d81a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.100+0000 7f72de400640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72d8102810 0x7f72d819a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.100+0000 7f72de400640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72c00a2870 0x7f72c00a4d30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.100+0000 7f72de400640 1 --2- 172.21.15.7:0/1179384187 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72d8101620 0x7f72d8199b20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.100+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 >> 172.21.15.7:0/1179384187 conn(0x7f72d80fcb90 msgr2=0x7f72d8073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.100+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 shutdown_connections 2024-03-18T01:38:43.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.100+0000 7f72de400640 1 -- 172.21.15.7:0/1179384187 wait complete. 2024-03-18T01:38:43.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:42 smithi007 ceph-mon[31996]: pgmap v433: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:38:43.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:42 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/586631989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:43.121 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T01:38:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:42 smithi203 ceph-mon[36345]: pgmap v433: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:38:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:42 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/586631989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:43.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:42 smithi046 ceph-mon[36108]: pgmap v433: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:38:43.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:42 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/586631989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:38:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.359+0000 7f008ba00640 1 -- 172.21.15.7:0/2018736907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0084104c10 msgr2=0x7f0084105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.359+0000 7f008ba00640 1 --2- 172.21.15.7:0/2018736907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0084104c10 0x7f0084105090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0070009900 tx=0x7f007002bc50 comp rx=0 tx=0).stop 2024-03-18T01:38:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.360+0000 7f008ba00640 1 -- 172.21.15.7:0/2018736907 shutdown_connections 2024-03-18T01:38:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.360+0000 7f008ba00640 1 --2- 172.21.15.7:0/2018736907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00841055d0 0x7f008410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.360+0000 7f008ba00640 1 --2- 172.21.15.7:0/2018736907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0084104c10 0x7f0084105090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.360+0000 7f008ba00640 1 --2- 172.21.15.7:0/2018736907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0084103a20 0x7f0084103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.360+0000 7f008ba00640 1 -- 172.21.15.7:0/2018736907 >> 172.21.15.7:0/2018736907 conn(0x7f00840fef90 msgr2=0x7f00841013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.360+0000 7f008ba00640 1 -- 172.21.15.7:0/2018736907 shutdown_connections 2024-03-18T01:38:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.360+0000 7f008ba00640 1 -- 172.21.15.7:0/2018736907 wait complete. 2024-03-18T01:38:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.361+0000 7f008ba00640 1 Processor -- start 2024-03-18T01:38:43.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.361+0000 7f008ba00640 1 -- start start 2024-03-18T01:38:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f008ba00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0084103a20 0x7f0084199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f008ba00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0084104c10 0x7f008419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f008ba00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00841055d0 0x7f00841a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f008ba00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f008410f6a0 con 0x7f0084103a20 2024-03-18T01:38:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f008ba00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f008410f520 con 0x7f0084104c10 2024-03-18T01:38:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f008ba00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f008410f820 con 0x7f00841055d0 2024-03-18T01:38:43.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f0089a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00841055d0 0x7f00841a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f0083e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0084104c10 0x7f008419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f0089a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00841055d0 0x7f00841a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34628/0 (socket says 172.21.15.7:34628) 2024-03-18T01:38:43.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f0089a00640 1 -- 172.21.15.7:0/1565690136 learned_addr learned my addr 172.21.15.7:0/1565690136 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.362+0000 7f0089000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0084103a20 0x7f0084199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0089a00640 1 -- 172.21.15.7:0/1565690136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0084104c10 msgr2=0x7f008419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0089a00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0084104c10 0x7f008419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0089a00640 1 -- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0084103a20 msgr2=0x7f0084199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0089a00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0084103a20 0x7f0084199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0089a00640 1 -- 172.21.15.7:0/1565690136 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00841a19b0 con 0x7f00841055d0 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0089a00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00841055d0 0x7f00841a1310 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f007c00eb10 tx=0x7f007c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0081600640 1 -- 172.21.15.7:0/1565690136 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c004b20 con 0x7f00841055d0 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0081600640 1 -- 172.21.15.7:0/1565690136 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c004cc0 con 0x7f00841055d0 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f0081600640 1 -- 172.21.15.7:0/1565690136 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c010760 con 0x7f00841055d0 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f008ba00640 1 -- 172.21.15.7:0/1565690136 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00841a1ca0 con 0x7f00841055d0 2024-03-18T01:38:43.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.363+0000 7f008ba00640 1 -- 172.21.15.7:0/1565690136 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00841a2190 con 0x7f00841055d0 2024-03-18T01:38:43.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.367+0000 7f008e7f5740 1 Processor -- start 2024-03-18T01:38:43.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.368+0000 7f0081600640 1 -- 172.21.15.7:0/1565690136 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f007c005020 con 0x7f00841055d0 2024-03-18T01:38:43.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.368+0000 7f008e7f5740 1 -- start start 2024-03-18T01:38:43.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540d0d0 0x55e96540d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e9654125c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 0x55e9654150c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f008e7f5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e9653cc3b0 con 0x55e965412c30 2024-03-18T01:38:43.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f008e7f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e965369450 con 0x55e96540dac0 2024-03-18T01:38:43.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f008e7f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e965311cb0 con 0x55e96540d0d0 2024-03-18T01:38:43.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f0083e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540d0d0 0x55e96540d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f0083e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540d0d0 0x55e96540d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34638/0 (socket says 172.21.15.7:34638) 2024-03-18T01:38:43.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f0089000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e9654125c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.369+0000 7f0083e00640 1 -- 172.21.15.7:0/1791230115 learned_addr learned my addr 172.21.15.7:0/1791230115 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0083e00640 1 -- 172.21.15.7:0/1791230115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 msgr2=0x55e9654125c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0083e00640 1 --2- 172.21.15.7:0/1791230115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e9654125c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0083e00640 1 -- 172.21.15.7:0/1791230115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 msgr2=0x55e9654150c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0083e00640 1 --2- 172.21.15.7:0/1791230115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 0x55e9654150c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0083e00640 1 -- 172.21.15.7:0/1791230115 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9653331f0 con 0x55e96540d0d0 2024-03-18T01:38:43.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0081600640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f006c0a2880 0x7f006c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0083e00640 1 --2- 172.21.15.7:0/1791230115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540d0d0 0x55e96540d4f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0070009900 tx=0x7f007002c5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62d4d7effc07e8ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0081600640 1 -- 172.21.15.7:0/1565690136 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f007c0cf4e0 con 0x7f00841055d0 2024-03-18T01:38:43.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0089000640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f006c0a2880 0x7f006c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.370+0000 7f0076000640 1 -- 172.21.15.7:0/1791230115 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007002c9a0 con 0x55e96540d0d0 2024-03-18T01:38:43.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.371+0000 7f0076000640 1 -- 172.21.15.7:0/1791230115 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007002cb40 con 0x55e96540d0d0 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.371+0000 7f008e7f5740 1 -- 172.21.15.7:0/1791230115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540d0d0 msgr2=0x55e96540d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.371+0000 7f0089000640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f006c0a2880 0x7f006c0a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55e96540dec0 tx=0x7f0078005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.371+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1791230115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540d0d0 0x55e96540d4f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0070009900 tx=0x7f007002c5f0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.371+0000 7f0076000640 1 -- 172.21.15.7:0/1791230115 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007002c9a0 con 0x55e96540d0d0 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.372+0000 7f008e7f5740 1 -- 172.21.15.7:0/1791230115 shutdown_connections 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.372+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1791230115 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 0x55e9654150c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.372+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1791230115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e9654125c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.372+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1791230115 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540d0d0 0x55e96540d4f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.372+0000 7f008e7f5740 1 -- 172.21.15.7:0/1791230115 >> 172.21.15.7:0/1791230115 conn(0x55e9653ffe30 msgr2=0x55e965400230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.372+0000 7f008e7f5740 1 -- 172.21.15.7:0/1791230115 shutdown_connections 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.372+0000 7f008e7f5740 1 -- 172.21.15.7:0/1791230115 wait complete. 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.372+0000 7f008e7f5740 1 -- start start 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f008e7f5740 1 Processor -- start 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 0x55e9654064d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540b500 0x55e96540b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f008e7f5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e965325110 con 0x55e965412c30 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f008e7f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e9653654f0 con 0x55e96540dac0 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f008e7f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e965404960 con 0x55e96540b500 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f008e7f5740 10 client.0 _ll_register_callbacks cb 0x55e965312b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f0089000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f0089000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55326/0 (socket says 172.21.15.7:55326) 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.373+0000 7f0089000640 1 -- 172.21.15.7:0/1843320245 learned_addr learned my addr 172.21.15.7:0/1843320245 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.374+0000 7f0083e00640 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 0x55e9654064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.374+0000 7f0089a00640 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540b500 0x55e96540b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.374+0000 7f0083e00640 1 -- 172.21.15.7:0/1843320245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540b500 msgr2=0x55e96540b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.374+0000 7f0083e00640 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540b500 0x55e96540b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.374+0000 7f0083e00640 1 -- 172.21.15.7:0/1843320245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 msgr2=0x55e965405f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.374+0000 7f0083e00640 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.374+0000 7f0083e00640 1 -- 172.21.15.7:0/1843320245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e96507d3f0 con 0x55e965412c30 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.375+0000 7f0083e00640 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 0x55e9654064d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f007002c7d0 tx=0x7f0070004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9f42b3f12f373e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.375+0000 7f006aa00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.375+0000 7f006aa00640 1 -- 172.21.15.7:0/1843320245 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007004e070 con 0x55e965412c30 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.375+0000 7f006aa00640 1 -- 172.21.15.7:0/1843320245 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0070034440 con 0x55e965412c30 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.375+0000 7f008e7f5740 1 -- 172.21.15.7:0/1843320245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007002cb40 con 0x55e965412c30 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.375+0000 7f006aa00640 1 -- 172.21.15.7:0/1843320245 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f007005a050 con 0x55e965412c30 2024-03-18T01:38:43.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f008e7f5740 10 client.15522 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f008e7f5740 1 -- 172.21.15.7:0/1843320245 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e9651477d0 con 0x55e965412c30 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f006aa00640 1 -- 172.21.15.7:0/1843320245 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007002cb40 con 0x55e965412c30 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f006aa00640 1 -- 172.21.15.7:0/1843320245 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f0070034c20 con 0x55e965412c30 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f008e7f5740 10 client.15522 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f008e7f5740 10 client.15522 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f008e7f5740 20 client.15522 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55e96542b770 0x55e96542db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f008e7f5740 4 client.15522 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f008e7f5740 1 -- 172.21.15.7:0/1843320245 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e965376990 con 0x55e96542b770 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.376+0000 7f0089000640 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55e96542b770 0x55e96542db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.377+0000 7f0089000640 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55e96542b770 0x55e96542db60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de2ad3522287c1ba server_cookie=3e09e92c012fada7 in_seq=0 out_seq=0 2024-03-18T01:38:43.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.377+0000 7f006aa00640 10 client.15522 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:38:43.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f006aa00640 1 -- 172.21.15.7:0/1843320245 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e965376990 con 0x55e96542b770 2024-03-18T01:38:43.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f006aa00640 10 client.15522 handle_command_reply: tid=0 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 client.15522 shutdown 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 -- 172.21.15.7:0/1843320245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 msgr2=0x55e9654064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 0x55e9654064d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f007002c7d0 tx=0x7f0070004e40 comp rx=0 tx=0).stop 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 -- 172.21.15.7:0/1843320245 shutdown_connections 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55e96542b770 0x55e96542db60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540b500 0x55e96540b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e965412c30 0x55e9654064d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1843320245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f008e7f5740 1 -- 172.21.15.7:0/1843320245 >> 172.21.15.7:0/1843320245 conn(0x55e9653ffe30 msgr2=0x55e96540d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.378+0000 7f006aa00640 0 client.15522 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.379+0000 7f008e7f5740 1 -- 172.21.15.7:0/1843320245 shutdown_connections 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.379+0000 7f008e7f5740 1 -- 172.21.15.7:0/1843320245 wait complete. 2024-03-18T01:38:43.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.379+0000 7f008e7f5740 20 client.15522 trim_cache size 0 max 16384 2024-03-18T01:38:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.395+0000 7f008e7f5740 1 Processor -- start 2024-03-18T01:38:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.395+0000 7f008e7f5740 1 -- start start 2024-03-18T01:38:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.395+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e965400210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.395+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.395+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e965412c30 0x55e9654064d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.395+0000 7f008e7f5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f00841a82e0 con 0x55e96540b500 2024-03-18T01:38:43.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.395+0000 7f008e7f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f00841a9e20 con 0x55e96540dac0 2024-03-18T01:38:43.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.395+0000 7f008e7f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f00841a89b0 con 0x55e965412c30 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0089000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e965400210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0083e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0083e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55328/0 (socket says 172.21.15.7:55328) 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0083e00640 1 -- 172.21.15.7:0/210927259 learned_addr learned my addr 172.21.15.7:0/210927259 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0089a00640 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e965412c30 0x55e9654064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0089000640 1 -- 172.21.15.7:0/210927259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e965412c30 msgr2=0x55e9654064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0089000640 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e965412c30 0x55e9654064d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0089000640 1 -- 172.21.15.7:0/210927259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 msgr2=0x55e965405f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0089000640 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0089000640 1 -- 172.21.15.7:0/210927259 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9653fbc10 con 0x55e96540b500 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0089a00640 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e965412c30 0x55e9654064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:43.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.396+0000 7f0083e00640 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.397+0000 7f0089000640 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e965400210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55e96542e1f0 tx=0x7f0078017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401fe3535a8c775d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.397+0000 7f0075600640 1 -- 172.21.15.7:0/210927259 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0078004400 con 0x55e96540b500 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.397+0000 7f0075600640 1 -- 172.21.15.7:0/210927259 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007802b510 con 0x55e96540b500 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.397+0000 7f0075600640 1 -- 172.21.15.7:0/210927259 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007801e6b0 con 0x55e96540b500 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 -- 172.21.15.7:0/210927259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 msgr2=0x55e965400210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e965400210 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x55e96542e1f0 tx=0x7f0078017000 comp rx=0 tx=0).stop 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 -- 172.21.15.7:0/210927259 shutdown_connections 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e965412c30 0x55e9654064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e96540dac0 0x55e965405f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 --2- 172.21.15.7:0/210927259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e965400210 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 -- 172.21.15.7:0/210927259 >> 172.21.15.7:0/210927259 conn(0x55e9653ffe30 msgr2=0x55e965412340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 -- 172.21.15.7:0/210927259 shutdown_connections 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.398+0000 7f008e7f5740 1 -- 172.21.15.7:0/210927259 wait complete. 2024-03-18T01:38:43.400 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.399+0000 7f008e7f5740 1 -- start start 2024-03-18T01:38:43.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.399+0000 7f008e7f5740 1 Processor -- start 2024-03-18T01:38:43.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.399+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e96543c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540dac0 0x55e96543c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f008e7f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e965412c30 0x55e96543d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f008e7f5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f00841a9750 con 0x55e96540b500 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f008e7f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f00841a9080 con 0x55e965412c30 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f008e7f5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f00841a62c0 con 0x55e96540dac0 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f008e7f5740 10 client.0 _ll_register_callbacks cb 0x7f00780097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e96543c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e96543c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53494/0 (socket says 172.21.15.7:53494) 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089000640 1 -- 172.21.15.7:0/1446418365 learned_addr learned my addr 172.21.15.7:0/1446418365 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089a00640 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e965412c30 0x55e96543d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089a00640 1 -- 172.21.15.7:0/1446418365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540dac0 msgr2=0x55e96543c5a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089a00640 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540dac0 0x55e96543c5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089a00640 1 -- 172.21.15.7:0/1446418365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 msgr2=0x55e96543c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.403 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089a00640 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e96543c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.400+0000 7f0089a00640 1 -- 172.21.15.7:0/1446418365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e96543dba0 con 0x55e965412c30 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.401+0000 7f0089a00640 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e965412c30 0x55e96543d5b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f007c0049d0 tx=0x7f007c09b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3fe5ea931fbe5e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.401+0000 7f006aa00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.401+0000 7f006aa00640 1 -- 172.21.15.7:0/1446418365 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c09c1c0 con 0x55e965412c30 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.401+0000 7f006aa00640 1 -- 172.21.15.7:0/1446418365 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f007c0b4b80 con 0x55e965412c30 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.402+0000 7f008e7f5740 1 -- 172.21.15.7:0/1446418365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007802b510 con 0x55e965412c30 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.402+0000 7f006aa00640 1 -- 172.21.15.7:0/1446418365 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f007c09bac0 con 0x55e965412c30 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.402+0000 7f008e7f5740 10 client.25700 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.402+0000 7f008e7f5740 1 -- 172.21.15.7:0/1446418365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e965491270 con 0x55e965412c30 2024-03-18T01:38:43.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.402+0000 7f006aa00640 1 -- 172.21.15.7:0/1446418365 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007802b510 con 0x55e965412c30 2024-03-18T01:38:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.403+0000 7f006aa00640 1 -- 172.21.15.7:0/1446418365 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f007c005310 con 0x55e965412c30 2024-03-18T01:38:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.403+0000 7f008e7f5740 10 client.25700 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.403+0000 7f008e7f5740 10 client.25700 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:38:43.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.403+0000 7f008e7f5740 20 client.25700 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.403+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55e965491ae0 0x55e965493f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.403+0000 7f008e7f5740 4 client.25700 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.403+0000 7f008e7f5740 1 -- 172.21.15.7:0/1446418365 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e965370ce0 con 0x55e965491ae0 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.404+0000 7f0089000640 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55e965491ae0 0x55e965493f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.404+0000 7f0089000640 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55e965491ae0 0x55e965493f00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95220d0e88889635 server_cookie=fd1c57f12b35d613 in_seq=0 out_seq=0 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.404+0000 7f006aa00640 10 client.25700 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.404+0000 7f006aa00640 1 -- 172.21.15.7:0/1446418365 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e965370ce0 con 0x55e965491ae0 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.404+0000 7f006aa00640 10 client.25700 handle_command_reply: tid=0 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.404+0000 7f008e7f5740 1 client.25700 shutdown 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.404+0000 7f008e7f5740 1 -- 172.21.15.7:0/1446418365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e965412c30 msgr2=0x55e96543d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.404+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e965412c30 0x55e96543d5b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f007c0049d0 tx=0x7f007c09b9c0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 1 -- 172.21.15.7:0/1446418365 shutdown_connections 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e965412c30 0x55e96543d5b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55e965491ae0 0x55e965493f00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e96540dac0 0x55e96543c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 1 --2- 172.21.15.7:0/1446418365 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e96540b500 0x55e96543c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 1 -- 172.21.15.7:0/1446418365 >> 172.21.15.7:0/1446418365 conn(0x55e9653ffe30 msgr2=0x55e96540ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f006aa00640 0 client.25700 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 1 -- 172.21.15.7:0/1446418365 shutdown_connections 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 1 -- 172.21.15.7:0/1446418365 wait complete. 2024-03-18T01:38:43.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.405+0000 7f008e7f5740 20 client.25700 trim_cache size 0 max 16384 2024-03-18T01:38:43.410 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:38:43.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.408+0000 7f008ba00640 1 -- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f006c0a2880 msgr2=0x7f006c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.408+0000 7f008ba00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f006c0a2880 0x7f006c0a4d40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55e96540dec0 tx=0x7f0078005f90 comp rx=0 tx=0).stop 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.408+0000 7f008ba00640 1 -- 172.21.15.7:0/1565690136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00841055d0 msgr2=0x7f00841a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.408+0000 7f008ba00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00841055d0 0x7f00841a1310 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f007c00eb10 tx=0x7f007c00c2d0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.408+0000 7f008ba00640 1 -- 172.21.15.7:0/1565690136 shutdown_connections 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.408+0000 7f008ba00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f00841055d0 0x7f00841a1310 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.408+0000 7f008ba00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0084104c10 0x7f008419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.409+0000 7f008ba00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f006c0a2880 0x7f006c0a4d40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.409+0000 7f008ba00640 1 --2- 172.21.15.7:0/1565690136 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0084103a20 0x7f0084199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.409+0000 7f008ba00640 1 -- 172.21.15.7:0/1565690136 >> 172.21.15.7:0/1565690136 conn(0x7f00840fef90 msgr2=0x7f0084107f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.409+0000 7f008ba00640 1 -- 172.21.15.7:0/1565690136 shutdown_connections 2024-03-18T01:38:43.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.409+0000 7f008ba00640 1 -- 172.21.15.7:0/1565690136 wait complete. 2024-03-18T01:38:43.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.630+0000 7f9b94a00640 1 -- 172.21.15.7:0/3666993239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 msgr2=0x7f9b90102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.630+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3666993239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 0x7f9b90102c90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009900 tx=0x7f9b7c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.630+0000 7f9b94a00640 1 -- 172.21.15.7:0/3666993239 shutdown_connections 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.630+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3666993239 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b901031d0 0x7f9b90109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.630+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3666993239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 0x7f9b90102c90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.630+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3666993239 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b90101620 0x7f9b90101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.630+0000 7f9b94a00640 1 -- 172.21.15.7:0/3666993239 >> 172.21.15.7:0/3666993239 conn(0x7f9b900fcb90 msgr2=0x7f9b900fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.630+0000 7f9b94a00640 1 -- 172.21.15.7:0/3666993239 shutdown_connections 2024-03-18T01:38:43.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.631+0000 7f9b94a00640 1 -- 172.21.15.7:0/3666993239 wait complete. 2024-03-18T01:38:43.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.632+0000 7f9b94a00640 1 Processor -- start 2024-03-18T01:38:43.632 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.632+0000 7f9b94a00640 1 -- start start 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90101620 0x7f9b90199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 0x7f9b9019a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b94a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b901031d0 0x7f9b901a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b94a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9010d2e0 con 0x7f9b90101620 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b94a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9010d160 con 0x7f9b90102810 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b94a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9010d460 con 0x7f9b901031d0 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b8fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b901031d0 0x7f9b901a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b8fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b901031d0 0x7f9b901a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34670/0 (socket says 172.21.15.7:34670) 2024-03-18T01:38:43.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b8fe00640 1 -- 172.21.15.7:0/3906664025 learned_addr learned my addr 172.21.15.7:0/3906664025 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.633+0000 7f9b8f400640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90101620 0x7f9b90199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 0x7f9b9019a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8fe00640 1 -- 172.21.15.7:0/3906664025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 msgr2=0x7f9b9019a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 0x7f9b9019a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8fe00640 1 -- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90101620 msgr2=0x7f9b90199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90101620 0x7f9b90199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8fe00640 1 -- 172.21.15.7:0/3906664025 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b901a16e0 con 0x7f9b901031d0 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 0x7f9b9019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8f400640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90101620 0x7f9b90199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:43.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b901031d0 0x7f9b901a1070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800eaa0 tx=0x7f9b8800ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b87e00640 1 -- 172.21.15.7:0/3906664025 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8800cdb0 con 0x7f9b901031d0 2024-03-18T01:38:43.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b87e00640 1 -- 172.21.15.7:0/3906664025 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b88005020 con 0x7f9b901031d0 2024-03-18T01:38:43.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b87e00640 1 -- 172.21.15.7:0/3906664025 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b880116d0 con 0x7f9b901031d0 2024-03-18T01:38:43.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.634+0000 7f9b94a00640 1 -- 172.21.15.7:0/3906664025 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b901a19d0 con 0x7f9b901031d0 2024-03-18T01:38:43.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.635+0000 7f9b94a00640 1 -- 172.21.15.7:0/3906664025 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b901a1f20 con 0x7f9b901031d0 2024-03-18T01:38:43.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.639+0000 7f9b98da9740 1 Processor -- start 2024-03-18T01:38:43.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.640+0000 7f9b87e00640 1 -- 172.21.15.7:0/3906664025 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9b88005540 con 0x7f9b901031d0 2024-03-18T01:38:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.640+0000 7f9b98da9740 1 -- start start 2024-03-18T01:38:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.641+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.641+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.641+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.641+0000 7f9b98da9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556b0a7883b0 con 0x556b0a7c1480 2024-03-18T01:38:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.641+0000 7f9b98da9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b0a725450 con 0x556b0a7c0a90 2024-03-18T01:38:43.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.641+0000 7f9b98da9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b0a6cdcb0 con 0x556b0a7c65f0 2024-03-18T01:38:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.641+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55364/0 (socket says 172.21.15.7:55364) 2024-03-18T01:38:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b8ea00640 1 -- 172.21.15.7:0/1087191362 learned_addr learned my addr 172.21.15.7:0/1087191362 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b8f400640 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b8ea00640 1 -- 172.21.15.7:0/1087191362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 msgr2=0x556b0a7c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b8ea00640 1 -- 172.21.15.7:0/1087191362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 msgr2=0x556b0a7c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b87e00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9b740a2870 0x7f9b740a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b87e00640 1 -- 172.21.15.7:0/3906664025 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9b880cf370 con 0x7f9b901031d0 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.642+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.643+0000 7f9b8ea00640 1 -- 172.21.15.7:0/1087191362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b0a6ef1f0 con 0x556b0a7c0a90 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.643+0000 7f9b8f400640 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.643+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7c0eb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c02cc00 tx=0x7f9b7c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b326ecbfb4697475 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.643+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9b740a2870 0x7f9b740a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.643+0000 7f9b7be00640 1 -- 172.21.15.7:0/1087191362 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c048070 con 0x556b0a7c0a90 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.643+0000 7f9b7be00640 1 -- 172.21.15.7:0/1087191362 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b7c02cd20 con 0x556b0a7c0a90 2024-03-18T01:38:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.643+0000 7f9b7be00640 1 -- 172.21.15.7:0/1087191362 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c0436b0 con 0x556b0a7c0a90 2024-03-18T01:38:43.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.644+0000 7f9b98da9740 1 -- 172.21.15.7:0/1087191362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 msgr2=0x556b0a7c0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.644+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9b740a2870 0x7f9b740a4d30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0096c0 tx=0x7f9b7c003e10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.644+0000 7f9b98da9740 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7c0eb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c02cc00 tx=0x7f9b7c000bc0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.645+0000 7f9b98da9740 1 -- 172.21.15.7:0/1087191362 shutdown_connections 2024-03-18T01:38:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.645+0000 7f9b98da9740 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.645+0000 7f9b98da9740 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.645+0000 7f9b98da9740 1 --2- 172.21.15.7:0/1087191362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7c0eb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.645+0000 7f9b98da9740 1 -- 172.21.15.7:0/1087191362 >> 172.21.15.7:0/1087191362 conn(0x556b0a7bc000 msgr2=0x556b0a7bc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.645+0000 7f9b98da9740 1 -- 172.21.15.7:0/1087191362 shutdown_connections 2024-03-18T01:38:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.645+0000 7f9b98da9740 1 -- 172.21.15.7:0/1087191362 wait complete. 2024-03-18T01:38:43.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.645+0000 7f9b98da9740 1 -- start start 2024-03-18T01:38:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.646+0000 7f9b98da9740 1 Processor -- start 2024-03-18T01:38:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.646+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.646+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.646+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7d2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.646+0000 7f9b98da9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556b0a6e1110 con 0x556b0a7c1480 2024-03-18T01:38:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.646+0000 7f9b98da9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556b0a7214f0 con 0x556b0a7c0a90 2024-03-18T01:38:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.646+0000 7f9b98da9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556b0a7c9540 con 0x556b0a7c65f0 2024-03-18T01:38:43.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b98da9740 10 client.0 _ll_register_callbacks cb 0x556b0a6ceb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55368/0 (socket says 172.21.15.7:55368) 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7d2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8f400640 1 -- 172.21.15.7:0/308037710 learned_addr learned my addr 172.21.15.7:0/308037710 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8ea00640 1 -- 172.21.15.7:0/308037710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 msgr2=0x556b0a7d2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7d2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8ea00640 1 -- 172.21.15.7:0/308037710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 msgr2=0x556b0a7cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8ea00640 1 -- 172.21.15.7:0/308037710 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b0a4393f0 con 0x556b0a7c1480 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.647+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7d2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.648+0000 7f9b8ea00640 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7cb260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009900 tx=0x7f9b7c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be4872ac13ad42a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.648+0000 7f9b78c00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.648+0000 7f9b78c00640 1 -- 172.21.15.7:0/308037710 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c05d060 con 0x556b0a7c1480 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.648+0000 7f9b78c00640 1 -- 172.21.15.7:0/308037710 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9b7c003fc0 con 0x556b0a7c1480 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.648+0000 7f9b98da9740 1 -- 172.21.15.7:0/308037710 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556b0a732990 con 0x556b0a7c1480 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.649+0000 7f9b78c00640 1 -- 172.21.15.7:0/308037710 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b7c048040 con 0x556b0a7c1480 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.649+0000 7f9b98da9740 10 client.15534 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.649+0000 7f9b98da9740 1 -- 172.21.15.7:0/308037710 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b0a5037d0 con 0x556b0a7c1480 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.649+0000 7f9b78c00640 1 -- 172.21.15.7:0/308037710 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556b0a732990 con 0x556b0a7c1480 2024-03-18T01:38:43.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.650+0000 7f9b78c00640 1 -- 172.21.15.7:0/308037710 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9b7c0041c0 con 0x556b0a7c1480 2024-03-18T01:38:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.650+0000 7f9b98da9740 10 client.15534 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.650+0000 7f9b98da9740 10 client.15534 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:38:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.650+0000 7f9b98da9740 20 client.15534 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.650+0000 7f9b98da9740 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x556b0a7deef0 0x556b0a7e12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.650+0000 7f9b98da9740 4 client.15534 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:43.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.650+0000 7f9b98da9740 1 -- 172.21.15.7:0/308037710 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b0a72cce0 con 0x556b0a7deef0 2024-03-18T01:38:43.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.651+0000 7f9b8f400640 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x556b0a7deef0 0x556b0a7e12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.652+0000 7f9b8f400640 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x556b0a7deef0 0x556b0a7e12e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c972d315fd52cda server_cookie=54934a4c5199fcf7 in_seq=0 out_seq=0 2024-03-18T01:38:43.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.652+0000 7f9b78c00640 10 client.15534 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:38:43.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.653+0000 7f9b78c00640 1 -- 172.21.15.7:0/308037710 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556b0a72cce0 con 0x556b0a7deef0 2024-03-18T01:38:43.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.653+0000 7f9b78c00640 10 client.15534 handle_command_reply: tid=0 2024-03-18T01:38:43.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.653+0000 7f9b98da9740 1 client.15534 shutdown 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.653+0000 7f9b98da9740 1 -- 172.21.15.7:0/308037710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 msgr2=0x556b0a7cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.653+0000 7f9b98da9740 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7cb260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009900 tx=0x7f9b7c03e000 comp rx=0 tx=0).stop 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 1 -- 172.21.15.7:0/308037710 shutdown_connections 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x556b0a7deef0 0x556b0a7e12e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7d2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c1480 0x556b0a7cb260 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009900 tx=0x7f9b7c03e000 comp rx=0 tx=0).stop 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 1 --2- 172.21.15.7:0/308037710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7c0a90 0x556b0a7cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b78c00640 0 client.15534 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 1 -- 172.21.15.7:0/308037710 >> 172.21.15.7:0/308037710 conn(0x556b0a7bc000 msgr2=0x556b0a7bb220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 1 -- 172.21.15.7:0/308037710 shutdown_connections 2024-03-18T01:38:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 1 -- 172.21.15.7:0/308037710 wait complete. 2024-03-18T01:38:43.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.654+0000 7f9b98da9740 20 client.15534 trim_cache size 0 max 16384 2024-03-18T01:38:43.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.683+0000 7f9b98da9740 1 Processor -- start 2024-03-18T01:38:43.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.683+0000 7f9b98da9740 1 -- start start 2024-03-18T01:38:43.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.683+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c0a90 0x556b0a7c2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.683+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7c2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.683+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7ba990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.683+0000 7f9b98da9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901a7f80 con 0x556b0a7c0a90 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.683+0000 7f9b98da9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901a9ac0 con 0x556b0a7d2220 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.683+0000 7f9b98da9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901a8650 con 0x556b0a7c65f0 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c0a90 0x556b0a7c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7c2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c0a90 0x556b0a7c2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53530/0 (socket says 172.21.15.7:53530) 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8f400640 1 -- 172.21.15.7:0/2914489406 learned_addr learned my addr 172.21.15.7:0/2914489406 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7ba990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8f400640 1 -- 172.21.15.7:0/2914489406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 msgr2=0x556b0a7c2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8f400640 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7c2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8f400640 1 -- 172.21.15.7:0/2914489406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 msgr2=0x556b0a7ba990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8f400640 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7ba990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8f400640 1 -- 172.21.15.7:0/2914489406 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b0a7bb950 con 0x556b0a7c0a90 2024-03-18T01:38:43.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.684+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:43.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.685+0000 7f9b8f400640 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c0a90 0x556b0a7c2840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x556b0a7e1a00 tx=0x7f9b80014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d19568a809d9bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.685+0000 7f9b7b400640 1 -- 172.21.15.7:0/2914489406 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b800140d0 con 0x556b0a7c0a90 2024-03-18T01:38:43.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.685+0000 7f9b7b400640 1 -- 172.21.15.7:0/2914489406 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b80014270 con 0x556b0a7c0a90 2024-03-18T01:38:43.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.685+0000 7f9b7b400640 1 -- 172.21.15.7:0/2914489406 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80012b20 con 0x556b0a7c0a90 2024-03-18T01:38:43.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.686+0000 7f9b98da9740 1 -- 172.21.15.7:0/2914489406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c0a90 msgr2=0x556b0a7c2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.686+0000 7f9b98da9740 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c0a90 0x556b0a7c2840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x556b0a7e1a00 tx=0x7f9b80014840 comp rx=0 tx=0).stop 2024-03-18T01:38:43.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.686+0000 7f9b98da9740 1 -- 172.21.15.7:0/2914489406 shutdown_connections 2024-03-18T01:38:43.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.686+0000 7f9b98da9740 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7ba990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.686+0000 7f9b98da9740 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c65f0 0x556b0a7c2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.686+0000 7f9b98da9740 1 --2- 172.21.15.7:0/2914489406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c0a90 0x556b0a7c2840 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.686+0000 7f9b98da9740 1 -- 172.21.15.7:0/2914489406 >> 172.21.15.7:0/2914489406 conn(0x556b0a7bc000 msgr2=0x556b0a7bc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.687+0000 7f9b98da9740 1 -- 172.21.15.7:0/2914489406 shutdown_connections 2024-03-18T01:38:43.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.687+0000 7f9b98da9740 1 -- 172.21.15.7:0/2914489406 wait complete. 2024-03-18T01:38:43.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.687+0000 7f9b98da9740 1 -- start start 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.687+0000 7f9b98da9740 1 Processor -- start 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.688+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c0a90 0x556b0a7cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.688+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c65f0 0x556b0a7cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.688+0000 7f9b98da9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.688+0000 7f9b98da9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901a93f0 con 0x556b0a7c65f0 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.688+0000 7f9b98da9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901a8d20 con 0x556b0a7d2220 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.688+0000 7f9b98da9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b901a5f60 con 0x556b0a7c0a90 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.688+0000 7f9b98da9740 10 client.0 _ll_register_callbacks cb 0x7f9b7c046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8ea00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c65f0 0x556b0a7cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c0a90 0x556b0a7cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c0a90 0x556b0a7cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34698/0 (socket says 172.21.15.7:34698) 2024-03-18T01:38:43.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8f400640 1 -- 172.21.15.7:0/610288965 learned_addr learned my addr 172.21.15.7:0/610288965 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8fe00640 1 -- 172.21.15.7:0/610288965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c0a90 msgr2=0x556b0a7cfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c0a90 0x556b0a7cfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8fe00640 1 -- 172.21.15.7:0/610288965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c65f0 msgr2=0x556b0a7cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c65f0 0x556b0a7cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.689+0000 7f9b8fe00640 1 -- 172.21.15.7:0/610288965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b0a7cb5d0 con 0x556b0a7d2220 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.690+0000 7f9b8fe00640 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7d0f60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800efd0 tx=0x7f9b88021440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=339f4fc071dd4765 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.690+0000 7f9b78c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.690+0000 7f9b78c00640 1 -- 172.21.15.7:0/610288965 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b880a40f0 con 0x556b0a7d2220 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.690+0000 7f9b78c00640 1 -- 172.21.15.7:0/610288965 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9b880051d0 con 0x556b0a7d2220 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.690+0000 7f9b98da9740 1 -- 172.21.15.7:0/610288965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b80014270 con 0x556b0a7d2220 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.690+0000 7f9b78c00640 1 -- 172.21.15.7:0/610288965 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9b880976a0 con 0x556b0a7d2220 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.690+0000 7f9b98da9740 10 client.25715 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.690+0000 7f9b98da9740 1 -- 172.21.15.7:0/610288965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556b0a7cb950 con 0x556b0a7d2220 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b78c00640 1 -- 172.21.15.7:0/610288965 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b80014270 con 0x556b0a7d2220 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b78c00640 1 -- 172.21.15.7:0/610288965 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9b8800f040 con 0x556b0a7d2220 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b98da9740 10 client.25715 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b98da9740 10 client.25715 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b98da9740 20 client.25715 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:43.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b98da9740 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x556b0a7d9560 0x556b0a7d9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b98da9740 4 client.25715 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:38:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b98da9740 1 -- 172.21.15.7:0/610288965 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556b0a6aa560 con 0x556b0a7d9560 2024-03-18T01:38:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.691+0000 7f9b8f400640 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x556b0a7d9560 0x556b0a7d9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.692+0000 7f9b8f400640 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x556b0a7d9560 0x556b0a7d9970 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ab112bfd3c80888 server_cookie=74cc0f0baeb13d85 in_seq=0 out_seq=0 2024-03-18T01:38:43.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.692+0000 7f9b78c00640 10 client.25715 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:38:43.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.693+0000 7f9b78c00640 1 -- 172.21.15.7:0/610288965 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556b0a6aa560 con 0x556b0a7d9560 2024-03-18T01:38:43.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.693+0000 7f9b78c00640 10 client.25715 handle_command_reply: tid=0 2024-03-18T01:38:43.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.693+0000 7f9b98da9740 1 client.25715 shutdown 2024-03-18T01:38:43.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.693+0000 7f9b98da9740 1 -- 172.21.15.7:0/610288965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 msgr2=0x556b0a7d0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.693+0000 7f9b98da9740 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7d0f60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800efd0 tx=0x7f9b88021440 comp rx=0 tx=0).stop 2024-03-18T01:38:43.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 1 -- 172.21.15.7:0/610288965 shutdown_connections 2024-03-18T01:38:43.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x556b0a7d9560 0x556b0a7d9970 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b0a7d2220 0x556b0a7d0f60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556b0a7c65f0 0x556b0a7cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 1 --2- 172.21.15.7:0/610288965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556b0a7c0a90 0x556b0a7cfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 1 -- 172.21.15.7:0/610288965 >> 172.21.15.7:0/610288965 conn(0x556b0a7bc000 msgr2=0x556b0a7c2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b78c00640 0 client.25715 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 1 -- 172.21.15.7:0/610288965 shutdown_connections 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 1 -- 172.21.15.7:0/610288965 wait complete. 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.694+0000 7f9b98da9740 20 client.25715 trim_cache size 0 max 16384 2024-03-18T01:38:43.695 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:38:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 -- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9b740a2870 msgr2=0x7f9b740a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9b740a2870 0x7f9b740a4d30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0096c0 tx=0x7f9b7c003e10 comp rx=0 tx=0).stop 2024-03-18T01:38:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 -- 172.21.15.7:0/3906664025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b901031d0 msgr2=0x7f9b901a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b901031d0 0x7f9b901a1070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800eaa0 tx=0x7f9b8800ef60 comp rx=0 tx=0).stop 2024-03-18T01:38:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 -- 172.21.15.7:0/3906664025 shutdown_connections 2024-03-18T01:38:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b901031d0 0x7f9b901a1070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b90102810 0x7f9b9019a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9b740a2870 0x7f9b740a4d30 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 --2- 172.21.15.7:0/3906664025 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9b90101620 0x7f9b90199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 -- 172.21.15.7:0/3906664025 >> 172.21.15.7:0/3906664025 conn(0x7f9b900fcb90 msgr2=0x7f9b90073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 -- 172.21.15.7:0/3906664025 shutdown_connections 2024-03-18T01:38:43.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.699+0000 7f9b94a00640 1 -- 172.21.15.7:0/3906664025 wait complete. 2024-03-18T01:38:43.719 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:38:43.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.904+0000 7f2576400640 1 -- 172.21.15.7:0/1418864957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 msgr2=0x7f2570102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.904+0000 7f2576400640 1 --2- 172.21.15.7:0/1418864957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 0x7f2570102c90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f255c009900 tx=0x7f255c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:43.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.905+0000 7f2576400640 1 -- 172.21.15.7:0/1418864957 shutdown_connections 2024-03-18T01:38:43.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.905+0000 7f2576400640 1 --2- 172.21.15.7:0/1418864957 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f25701031d0 0x7f2570109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.905+0000 7f2576400640 1 --2- 172.21.15.7:0/1418864957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 0x7f2570102c90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.905+0000 7f2576400640 1 --2- 172.21.15.7:0/1418864957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2570101620 0x7f2570101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.905+0000 7f2576400640 1 -- 172.21.15.7:0/1418864957 >> 172.21.15.7:0/1418864957 conn(0x7f25700fcb90 msgr2=0x7f25700fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.905+0000 7f2576400640 1 -- 172.21.15.7:0/1418864957 shutdown_connections 2024-03-18T01:38:43.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.905+0000 7f2576400640 1 -- 172.21.15.7:0/1418864957 wait complete. 2024-03-18T01:38:43.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.906+0000 7f2576400640 1 Processor -- start 2024-03-18T01:38:43.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.906+0000 7f2576400640 1 -- start start 2024-03-18T01:38:43.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2576400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2570101620 0x7f2570112130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2576400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 0x7f2570112670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2576400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25701031d0 0x7f2570110770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2576400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f257010b0c0 con 0x7f2570101620 2024-03-18T01:38:43.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2576400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f257010af40 con 0x7f25701031d0 2024-03-18T01:38:43.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2576400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f257010b240 con 0x7f2570102810 2024-03-18T01:38:43.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2575000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2570101620 0x7f2570112130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2575000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2570101620 0x7f2570112130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53548/0 (socket says 172.21.15.7:53548) 2024-03-18T01:38:43.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2575000640 1 -- 172.21.15.7:0/873494259 learned_addr learned my addr 172.21.15.7:0/873494259 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2575a00640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25701031d0 0x7f2570110770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f256fe00640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 0x7f2570112670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.907+0000 7f2575a00640 1 -- 172.21.15.7:0/873494259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 msgr2=0x7f2570112670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f2575a00640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 0x7f2570112670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f2575a00640 1 -- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2570101620 msgr2=0x7f2570112130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f2575a00640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2570101620 0x7f2570112130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f2575a00640 1 -- 172.21.15.7:0/873494259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2570110ea0 con 0x7f25701031d0 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f256fe00640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 0x7f2570112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f2575000640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2570101620 0x7f2570112130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f2575a00640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25701031d0 0x7f2570110770 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f256800da10 tx=0x7f256800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f256d600640 1 -- 172.21.15.7:0/873494259 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256800bd70 con 0x7f25701031d0 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f256d600640 1 -- 172.21.15.7:0/873494259 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2568003e50 con 0x7f25701031d0 2024-03-18T01:38:43.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.908+0000 7f256d600640 1 -- 172.21.15.7:0/873494259 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568010660 con 0x7f25701031d0 2024-03-18T01:38:43.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.909+0000 7f2576400640 1 -- 172.21.15.7:0/873494259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2570111130 con 0x7f25701031d0 2024-03-18T01:38:43.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.909+0000 7f2576400640 1 -- 172.21.15.7:0/873494259 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25701aec60 con 0x7f25701031d0 2024-03-18T01:38:43.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.913+0000 7f256d600640 1 -- 172.21.15.7:0/873494259 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f2568003970 con 0x7f25701031d0 2024-03-18T01:38:43.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.914+0000 7f257a7bb740 1 Processor -- start 2024-03-18T01:38:43.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.914+0000 7f257a7bb740 1 -- start start 2024-03-18T01:38:43.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.914+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f467eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.914+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f46cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.914+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f46d5f0 0x557e8f46fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.914+0000 7f257a7bb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557e8f42f3b0 con 0x557e8f467a90 2024-03-18T01:38:43.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.914+0000 7f257a7bb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557e8f3cc450 con 0x557e8f468480 2024-03-18T01:38:43.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.914+0000 7f257a7bb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557e8f374cb0 con 0x557e8f46d5f0 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f2575000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f46cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f2575000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f46cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55384/0 (socket says 172.21.15.7:55384) 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f2575000640 1 -- 172.21.15.7:0/4235391038 learned_addr learned my addr 172.21.15.7:0/4235391038 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f256fe00640 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f467eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f2575000640 1 -- 172.21.15.7:0/4235391038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f46d5f0 msgr2=0x557e8f46fa80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f2575000640 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f46d5f0 0x557e8f46fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f2575000640 1 -- 172.21.15.7:0/4235391038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 msgr2=0x557e8f467eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f2575000640 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f467eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.915+0000 7f2575000640 1 -- 172.21.15.7:0/4235391038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e8f3961f0 con 0x557e8f468480 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.916+0000 7f256d600640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f25580a28f0 0x7f25580a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.916+0000 7f256d600640 1 -- 172.21.15.7:0/873494259 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f25680ce740 con 0x7f25701031d0 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.916+0000 7f2575000640 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f46cf80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f25640115f0 tx=0x7f2564011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13c15189a29b4429 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.916+0000 7f256fe00640 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f467eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.916+0000 7f2562000640 1 -- 172.21.15.7:0/4235391038 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256401e070 con 0x557e8f468480 2024-03-18T01:38:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.916+0000 7f2562000640 1 -- 172.21.15.7:0/4235391038 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f256400f660 con 0x557e8f468480 2024-03-18T01:38:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.916+0000 7f2562000640 1 -- 172.21.15.7:0/4235391038 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25640038c0 con 0x557e8f468480 2024-03-18T01:38:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.917+0000 7f256fe00640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f25580a28f0 0x7f25580a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.917+0000 7f257a7bb740 1 -- 172.21.15.7:0/4235391038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 msgr2=0x557e8f46cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.917+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f46cf80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f25640115f0 tx=0x7f2564011ab0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.918+0000 7f257a7bb740 1 -- 172.21.15.7:0/4235391038 shutdown_connections 2024-03-18T01:38:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.918+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f46d5f0 0x557e8f46fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.918+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f46cf80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.918+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4235391038 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f467eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.918+0000 7f257a7bb740 1 -- 172.21.15.7:0/4235391038 >> 172.21.15.7:0/4235391038 conn(0x557e8f463000 msgr2=0x557e8f463400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.918+0000 7f257a7bb740 1 -- 172.21.15.7:0/4235391038 shutdown_connections 2024-03-18T01:38:43.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.918+0000 7f256fe00640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f25580a28f0 0x7f25580a4db0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f255c000c00 tx=0x7f255c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.918+0000 7f257a7bb740 1 -- 172.21.15.7:0/4235391038 wait complete. 2024-03-18T01:38:43.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.919+0000 7f257a7bb740 1 -- start start 2024-03-18T01:38:43.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.919+0000 7f257a7bb740 1 Processor -- start 2024-03-18T01:38:43.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f468480 0x557e8f4722d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f479310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f257a7bb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557e8f388110 con 0x557e8f467a90 2024-03-18T01:38:43.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f257a7bb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557e8f3c84f0 con 0x557e8f46d5f0 2024-03-18T01:38:43.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f257a7bb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557e8f4705a0 con 0x557e8f468480 2024-03-18T01:38:43.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f257a7bb740 10 client.0 _ll_register_callbacks cb 0x557e8f375b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f256fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f468480 0x557e8f4722d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f256fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f468480 0x557e8f4722d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34722/0 (socket says 172.21.15.7:34722) 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f256fe00640 1 -- 172.21.15.7:0/4010479258 learned_addr learned my addr 172.21.15.7:0/4010479258 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f2575a00640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f479310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f2575000640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f2575a00640 1 -- 172.21.15.7:0/4010479258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f468480 msgr2=0x557e8f4722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f2575a00640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f468480 0x557e8f4722d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.920+0000 7f2575a00640 1 -- 172.21.15.7:0/4010479258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 msgr2=0x557e8f471d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.921+0000 7f2575a00640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.921+0000 7f2575a00640 1 -- 172.21.15.7:0/4010479258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e8f0e03f0 con 0x557e8f46d5f0 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.921+0000 7f256fe00640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f468480 0x557e8f4722d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.921+0000 7f2575000640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.921+0000 7f2575a00640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f479310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f25680cbe80 tx=0x7f256809b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a294c3671f1b0ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.921+0000 7f2556a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:38:43.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.921+0000 7f2556a00640 1 -- 172.21.15.7:0/4010479258 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25680a34c0 con 0x557e8f46d5f0 2024-03-18T01:38:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.921+0000 7f2556a00640 1 -- 172.21.15.7:0/4010479258 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f25680b9550 con 0x557e8f46d5f0 2024-03-18T01:38:43.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.922+0000 7f257a7bb740 1 -- 172.21.15.7:0/4010479258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f256400f660 con 0x557e8f46d5f0 2024-03-18T01:38:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.922+0000 7f2556a00640 1 -- 172.21.15.7:0/4010479258 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2568003c60 con 0x557e8f46d5f0 2024-03-18T01:38:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.922+0000 7f257a7bb740 10 client.25727 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.922+0000 7f257a7bb740 1 -- 172.21.15.7:0/4010479258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557e8f1aa7d0 con 0x557e8f46d5f0 2024-03-18T01:38:43.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.922+0000 7f2556a00640 1 -- 172.21.15.7:0/4010479258 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f256400f660 con 0x557e8f46d5f0 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.923+0000 7f2556a00640 1 -- 172.21.15.7:0/4010479258 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2568097390 con 0x557e8f46d5f0 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.923+0000 7f257a7bb740 10 client.25727 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.923+0000 7f257a7bb740 10 client.25727 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.923+0000 7f257a7bb740 20 client.25727 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.923+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557e8f47db80 0x557e8f480040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.923+0000 7f257a7bb740 4 client.25727 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.923+0000 7f257a7bb740 1 -- 172.21.15.7:0/4010479258 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e8f3d9990 con 0x557e8f47db80 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.923+0000 7f2575000640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557e8f47db80 0x557e8f480040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.924+0000 7f2575000640 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557e8f47db80 0x557e8f480040 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa3a2c29996e14f3 server_cookie=f721e485c7c0705e in_seq=0 out_seq=0 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.924+0000 7f2556a00640 10 client.25727 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.924+0000 7f2556a00640 1 -- 172.21.15.7:0/4010479258 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557e8f3d9990 con 0x557e8f47db80 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.924+0000 7f2556a00640 10 client.25727 handle_command_reply: tid=0 2024-03-18T01:38:43.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.924+0000 7f257a7bb740 1 client.25727 shutdown 2024-03-18T01:38:43.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 -- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 msgr2=0x557e8f479310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f479310 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f25680cbe80 tx=0x7f256809b420 comp rx=0 tx=0).stop 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 -- 172.21.15.7:0/4010479258 shutdown_connections 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557e8f47db80 0x557e8f480040 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f479310 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f468480 0x557e8f4722d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 --2- 172.21.15.7:0/4010479258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 -- 172.21.15.7:0/4010479258 >> 172.21.15.7:0/4010479258 conn(0x557e8f463000 msgr2=0x557e8f462220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f2556a00640 0 client.25727 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 -- 172.21.15.7:0/4010479258 shutdown_connections 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.925+0000 7f257a7bb740 1 -- 172.21.15.7:0/4010479258 wait complete. 2024-03-18T01:38:43.927 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.926+0000 7f257a7bb740 20 client.25727 trim_cache size 0 max 16384 2024-03-18T01:38:43.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.954+0000 7f257a7bb740 1 Processor -- start 2024-03-18T01:38:43.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.954+0000 7f257a7bb740 1 -- start start 2024-03-18T01:38:43.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.954+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.954+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f4792d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.954+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f46d5f0 0x557e8f4722d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f257a7bb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f25701b8eb0 con 0x557e8f46d5f0 2024-03-18T01:38:43.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f257a7bb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ba9f0 con 0x557e8f468480 2024-03-18T01:38:43.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f257a7bb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25701b9580 con 0x557e8f467a90 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34726/0 (socket says 172.21.15.7:34726) 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575000640 1 -- 172.21.15.7:0/3405757124 learned_addr learned my addr 172.21.15.7:0/3405757124 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f256fe00640 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f4792d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575a00640 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f46d5f0 0x557e8f4722d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575000640 1 -- 172.21.15.7:0/3405757124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 msgr2=0x557e8f4792d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575000640 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f4792d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575000640 1 -- 172.21.15.7:0/3405757124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f46d5f0 msgr2=0x557e8f4722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575000640 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f46d5f0 0x557e8f4722d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575000640 1 -- 172.21.15.7:0/3405757124 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e8f463a50 con 0x557e8f467a90 2024-03-18T01:38:43.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.955+0000 7f2575a00640 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f46d5f0 0x557e8f4722d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:43.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.956+0000 7f2575000640 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f471d90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f256401c040 tx=0x7f256400ff60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6a1d7e69c472b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.956+0000 7f2561600640 1 -- 172.21.15.7:0/3405757124 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256402b070 con 0x557e8f467a90 2024-03-18T01:38:43.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.956+0000 7f2561600640 1 -- 172.21.15.7:0/3405757124 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25640050b0 con 0x557e8f467a90 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.956+0000 7f2561600640 1 -- 172.21.15.7:0/3405757124 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256401e040 con 0x557e8f467a90 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 -- 172.21.15.7:0/3405757124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 msgr2=0x557e8f471d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f471d90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f256401c040 tx=0x7f256400ff60 comp rx=0 tx=0).stop 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 -- 172.21.15.7:0/3405757124 shutdown_connections 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f46d5f0 0x557e8f4722d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f468480 0x557e8f4792d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3405757124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f471d90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 -- 172.21.15.7:0/3405757124 >> 172.21.15.7:0/3405757124 conn(0x557e8f463000 msgr2=0x557e8f469980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 -- 172.21.15.7:0/3405757124 shutdown_connections 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.957+0000 7f257a7bb740 1 -- 172.21.15.7:0/3405757124 wait complete. 2024-03-18T01:38:43.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.958+0000 7f257a7bb740 1 -- start start 2024-03-18T01:38:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.958+0000 7f257a7bb740 1 Processor -- start 2024-03-18T01:38:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f4769a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f468480 0x557e8f476ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f257a7bb740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f477e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f257a7bb740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f25701ba320 con 0x557e8f468480 2024-03-18T01:38:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f257a7bb740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f25701b9c50 con 0x557e8f46d5f0 2024-03-18T01:38:43.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f257a7bb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25701b6e90 con 0x557e8f467a90 2024-03-18T01:38:43.960 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f257a7bb740 10 client.0 _ll_register_callbacks cb 0x7f2568004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f2575a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f477e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f2575000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f4769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f2575a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f477e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55418/0 (socket says 172.21.15.7:55418) 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f2575a00640 1 -- 172.21.15.7:0/3669646850 learned_addr learned my addr 172.21.15.7:0/3669646850 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.959+0000 7f256fe00640 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f468480 0x557e8f476ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.960+0000 7f2575a00640 1 -- 172.21.15.7:0/3669646850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 msgr2=0x557e8f4769a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.960+0000 7f2575a00640 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f4769a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.960+0000 7f2575a00640 1 -- 172.21.15.7:0/3669646850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f468480 msgr2=0x557e8f476ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:43.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.960+0000 7f2575a00640 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f468480 0x557e8f476ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.960+0000 7f2575a00640 1 -- 172.21.15.7:0/3669646850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e8f478470 con 0x557e8f46d5f0 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.960+0000 7f256fe00640 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f468480 0x557e8f476ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.960+0000 7f2575a00640 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f477e80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f25680a2f90 tx=0x7f256809efc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5092f82dff7ca08b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.960+0000 7f2556a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.961+0000 7f2556a00640 1 -- 172.21.15.7:0/3669646850 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568014070 con 0x557e8f46d5f0 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.961+0000 7f2556a00640 1 -- 172.21.15.7:0/3669646850 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f25680042c0 con 0x557e8f46d5f0 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.961+0000 7f257a7bb740 1 -- 172.21.15.7:0/3669646850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25640050b0 con 0x557e8f46d5f0 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.961+0000 7f2556a00640 1 -- 172.21.15.7:0/3669646850 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f256807e050 con 0x557e8f46d5f0 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.961+0000 7f257a7bb740 10 client.25733 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.961+0000 7f257a7bb740 1 -- 172.21.15.7:0/3669646850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557e8f4e3f40 con 0x557e8f46d5f0 2024-03-18T01:38:43.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.961+0000 7f2556a00640 1 -- 172.21.15.7:0/3669646850 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25640050b0 con 0x557e8f46d5f0 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.961+0000 7f2556a00640 1 -- 172.21.15.7:0/3669646850 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f25680b93f0 con 0x557e8f46d5f0 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.962+0000 7f257a7bb740 10 client.25733 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.962+0000 7f257a7bb740 10 client.25733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.962+0000 7f257a7bb740 20 client.25733 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.962+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557e8f4e4550 0x557e8f4e6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.962+0000 7f257a7bb740 4 client.25733 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.962+0000 7f257a7bb740 1 -- 172.21.15.7:0/3669646850 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557e8f3d3ce0 con 0x557e8f4e4550 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.962+0000 7f2575000640 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557e8f4e4550 0x557e8f4e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f2575000640 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557e8f4e4550 0x557e8f4e6970 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fab596016443f43c server_cookie=9f346085d28e3d1 in_seq=0 out_seq=0 2024-03-18T01:38:43.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f2556a00640 10 client.25733 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f2556a00640 1 -- 172.21.15.7:0/3669646850 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557e8f3d3ce0 con 0x557e8f4e4550 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f2556a00640 10 client.25733 handle_command_reply: tid=0 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 client.25733 shutdown 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 -- 172.21.15.7:0/3669646850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 msgr2=0x557e8f477e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f477e80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f25680a2f90 tx=0x7f256809efc0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 -- 172.21.15.7:0/3669646850 shutdown_connections 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557e8f4e4550 0x557e8f4e6970 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557e8f46d5f0 0x557e8f477e80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557e8f468480 0x557e8f476ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 --2- 172.21.15.7:0/3669646850 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557e8f467a90 0x557e8f4769a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f257a7bb740 1 -- 172.21.15.7:0/3669646850 >> 172.21.15.7:0/3669646850 conn(0x557e8f463000 msgr2=0x557e8f471cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.963+0000 7f2556a00640 0 client.25733 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.964+0000 7f257a7bb740 1 -- 172.21.15.7:0/3669646850 shutdown_connections 2024-03-18T01:38:43.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.964+0000 7f257a7bb740 1 -- 172.21.15.7:0/3669646850 wait complete. 2024-03-18T01:38:43.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.964+0000 7f257a7bb740 20 client.25733 trim_cache size 0 max 16384 2024-03-18T01:38:43.966 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:38:43.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.964+0000 7f2576400640 1 -- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f25580a28f0 msgr2=0x7f25580a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.964+0000 7f2576400640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f25580a28f0 0x7f25580a4db0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f255c000c00 tx=0x7f255c02cf90 comp rx=0 tx=0).stop 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.964+0000 7f2576400640 1 -- 172.21.15.7:0/873494259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25701031d0 msgr2=0x7f2570110770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25701031d0 0x7f2570110770 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f256800da10 tx=0x7f256800ded0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 -- 172.21.15.7:0/873494259 shutdown_connections 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25701031d0 0x7f2570110770 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2570102810 0x7f2570112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f25580a28f0 0x7f25580a4db0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 --2- 172.21.15.7:0/873494259 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f2570101620 0x7f2570112130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 -- 172.21.15.7:0/873494259 >> 172.21.15.7:0/873494259 conn(0x7f25700fcb90 msgr2=0x7f2570105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 -- 172.21.15.7:0/873494259 shutdown_connections 2024-03-18T01:38:43.967 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:43.965+0000 7f2576400640 1 -- 172.21.15.7:0/873494259 wait complete. 2024-03-18T01:38:44.984 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T01:38:44.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:38:44.985 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T01:38:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:44 smithi203 ceph-mon[36345]: pgmap v434: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:45.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 -- 172.21.15.7:0/3386583574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24104a30 msgr2=0x7fdd24104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3386583574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24104a30 0x7fdd24104eb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009900 tx=0x7fdd1002bc40 comp rx=0 tx=0).stop 2024-03-18T01:38:45.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 -- 172.21.15.7:0/3386583574 shutdown_connections 2024-03-18T01:38:45.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3386583574 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd241053f0 0x7fdd2410bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3386583574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24104a30 0x7fdd24104eb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3386583574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd24103840 0x7fdd24103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.171 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 -- 172.21.15.7:0/3386583574 >> 172.21.15.7:0/3386583574 conn(0x7fdd240fedb0 msgr2=0x7fdd241011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 -- 172.21.15.7:0/3386583574 shutdown_connections 2024-03-18T01:38:45.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.170+0000 7fdd29e00640 1 -- 172.21.15.7:0/3386583574 wait complete. 2024-03-18T01:38:45.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.171+0000 7fdd29e00640 1 Processor -- start 2024-03-18T01:38:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd29e00640 1 -- start start 2024-03-18T01:38:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd29e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24103840 0x7fdd24199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd29e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd24104a30 0x7fdd2419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd29e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd241053f0 0x7fdd241a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd29e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410f4f0 con 0x7fdd24104a30 2024-03-18T01:38:45.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd29e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410f370 con 0x7fdd241053f0 2024-03-18T01:38:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd29e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410f670 con 0x7fdd24103840 2024-03-18T01:38:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd24104a30 0x7fdd2419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd28a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24103840 0x7fdd24199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd28a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24103840 0x7fdd24199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34754/0 (socket says 172.21.15.7:34754) 2024-03-18T01:38:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd28a00640 1 -- 172.21.15.7:0/3789017826 learned_addr learned my addr 172.21.15.7:0/3789017826 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.172+0000 7fdd29400640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd241053f0 0x7fdd241a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd28a00640 1 -- 172.21.15.7:0/3789017826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd241053f0 msgr2=0x7fdd241a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd28a00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd241053f0 0x7fdd241a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd28a00640 1 -- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd24104a30 msgr2=0x7fdd2419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd28a00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd24104a30 0x7fdd2419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd28a00640 1 -- 172.21.15.7:0/3789017826 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd241a1720 con 0x7fdd24103840 2024-03-18T01:38:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd29400640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd241053f0 0x7fdd241a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:45.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd23e00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd24104a30 0x7fdd2419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd28a00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24103840 0x7fdd24199b00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e9c0 tx=0x7fdd1800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.173+0000 7fdd21600640 1 -- 172.21.15.7:0/3789017826 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1800cd80 con 0x7fdd24103840 2024-03-18T01:38:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.174+0000 7fdd21600640 1 -- 172.21.15.7:0/3789017826 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd180035f0 con 0x7fdd24103840 2024-03-18T01:38:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.174+0000 7fdd21600640 1 -- 172.21.15.7:0/3789017826 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18010690 con 0x7fdd24103840 2024-03-18T01:38:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.174+0000 7fdd29e00640 1 -- 172.21.15.7:0/3789017826 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd241a1a10 con 0x7fdd24103840 2024-03-18T01:38:45.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.174+0000 7fdd29e00640 1 -- 172.21.15.7:0/3789017826 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd241a1ee0 con 0x7fdd24103840 2024-03-18T01:38:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.178+0000 7fdd21600640 1 -- 172.21.15.7:0/3789017826 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fdd180037a0 con 0x7fdd24103840 2024-03-18T01:38:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.179+0000 7fdd2e1e1740 1 Processor -- start 2024-03-18T01:38:45.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.179+0000 7fdd21600640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd0c0a2950 0x7fdd0c0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.179+0000 7fdd21600640 1 -- 172.21.15.7:0/3789017826 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdd180cf240 con 0x7fdd24103840 2024-03-18T01:38:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.180+0000 7fdd29400640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd0c0a2950 0x7fdd0c0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.180+0000 7fdd2e1e1740 1 -- start start 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.180+0000 7fdd29400640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd0c0a2950 0x7fdd0c0a4e10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c005fd0 tx=0x7fdd1c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87190 0x55e836b875b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b8c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b8f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e836ae1f80 con 0x55e836b87b80 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e836a8aad0 con 0x55e836b87190 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e836ade020 con 0x55e836b8ccf0 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b8c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd28a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87190 0x55e836b875b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd28a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87190 0x55e836b875b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55444/0 (socket says 172.21.15.7:55444) 2024-03-18T01:38:45.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd28a00640 1 -- 172.21.15.7:0/2039920855 learned_addr learned my addr 172.21.15.7:0/2039920855 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.181+0000 7fdd29400640 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b8f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd28a00640 1 -- 172.21.15.7:0/2039920855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 msgr2=0x55e836b8f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd28a00640 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b8f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd28a00640 1 -- 172.21.15.7:0/2039920855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 msgr2=0x55e836b8c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd28a00640 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b8c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd28a00640 1 -- 172.21.15.7:0/2039920855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e836aa26f0 con 0x55e836b87190 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd29400640 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b8f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd23e00640 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b8c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:45.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd28a00640 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87190 0x55e836b875b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd180cc8f0 tx=0x7fdd1809b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12f0b1b524835266 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd15c00640 1 -- 172.21.15.7:0/2039920855 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd180b3070 con 0x55e836b87190 2024-03-18T01:38:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd15c00640 1 -- 172.21.15.7:0/2039920855 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1809f970 con 0x55e836b87190 2024-03-18T01:38:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.182+0000 7fdd15c00640 1 -- 172.21.15.7:0/2039920855 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1809cb50 con 0x55e836b87190 2024-03-18T01:38:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.183+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/2039920855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87190 msgr2=0x55e836b875b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.184+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87190 0x55e836b875b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd180cc8f0 tx=0x7fdd1809b410 comp rx=0 tx=0).stop 2024-03-18T01:38:45.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.184+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/2039920855 shutdown_connections 2024-03-18T01:38:45.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.184+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b8f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.184+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b8c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.184+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/2039920855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87190 0x55e836b875b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.184+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/2039920855 >> 172.21.15.7:0/2039920855 conn(0x55e836b46930 msgr2=0x55e836b82b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.185+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/2039920855 shutdown_connections 2024-03-18T01:38:45.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.185+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/2039920855 wait complete. 2024-03-18T01:38:45.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.185+0000 7fdd2e1e1740 1 -- start start 2024-03-18T01:38:45.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.186+0000 7fdd2e1e1740 1 Processor -- start 2024-03-18T01:38:45.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.187+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87190 0x55e836b91430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.187+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b91970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.187+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b989b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.187+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e836b8fc40 con 0x55e836b87b80 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.187+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e836a9df30 con 0x55e836b8ccf0 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.187+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e836b8fdc0 con 0x55e836b87190 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.187+0000 7fdd2e1e1740 10 client.0 _ll_register_callbacks cb 0x55e836a8bbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.187+0000 7fdd28a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87190 0x55e836b91430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd28a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87190 0x55e836b91430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34766/0 (socket says 172.21.15.7:34766) 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd28a00640 1 -- 172.21.15.7:0/3351283162 learned_addr learned my addr 172.21.15.7:0/3351283162 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd23e00640 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b91970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd29400640 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd23e00640 1 -- 172.21.15.7:0/3351283162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87190 msgr2=0x55e836b91430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd23e00640 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87190 0x55e836b91430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd23e00640 1 -- 172.21.15.7:0/3351283162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 msgr2=0x55e836b989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd23e00640 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.188+0000 7fdd23e00640 1 -- 172.21.15.7:0/3351283162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8368be7e0 con 0x55e836b87b80 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.189+0000 7fdd23e00640 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b91970 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdd10036040 tx=0x7fdd10004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63058d25f6d6eccd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.189+0000 7fdd0a000640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.189+0000 7fdd0a000640 1 -- 172.21.15.7:0/3351283162 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10005d50 con 0x55e836b87b80 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.189+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3351283162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd1809f970 con 0x55e836b87b80 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.189+0000 7fdd0a000640 1 -- 172.21.15.7:0/3351283162 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdd10005220 con 0x55e836b87b80 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.189+0000 7fdd0a000640 1 -- 172.21.15.7:0/3351283162 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdd10005420 con 0x55e836b87b80 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.189+0000 7fdd2e1e1740 10 client.15561 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:45.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.189+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3351283162 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e83691ef70 con 0x55e836b87b80 2024-03-18T01:38:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.190+0000 7fdd0a000640 1 -- 172.21.15.7:0/3351283162 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1809f970 con 0x55e836b87b80 2024-03-18T01:38:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.190+0000 7fdd0a000640 1 -- 172.21.15.7:0/3351283162 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdd10003a40 con 0x55e836b87b80 2024-03-18T01:38:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.190+0000 7fdd2e1e1740 10 client.15561 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.190+0000 7fdd2e1e1740 10 client.15561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.190+0000 7fdd2e1e1740 20 client.15561 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:45.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.190+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55e836b9d280 0x55e836b9f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.190+0000 7fdd2e1e1740 4 client.15561 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.190+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3351283162 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e836aef4c0 con 0x55e836b9d280 2024-03-18T01:38:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.191+0000 7fdd28a00640 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55e836b9d280 0x55e836b9f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.191+0000 7fdd28a00640 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55e836b9d280 0x55e836b9f690 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbc60f4d411834c4 server_cookie=5a382562cf5ddac2 in_seq=0 out_seq=0 2024-03-18T01:38:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.191+0000 7fdd0a000640 10 client.15561 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:45.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.192+0000 7fdd0a000640 1 -- 172.21.15.7:0/3351283162 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e836aef4c0 con 0x55e836b9d280 2024-03-18T01:38:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.192+0000 7fdd0a000640 10 client.15561 handle_command_reply: tid=0 2024-03-18T01:38:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.192+0000 7fdd2e1e1740 1 client.15561 shutdown 2024-03-18T01:38:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3351283162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 msgr2=0x55e836b91970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b91970 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdd10036040 tx=0x7fdd10004270 comp rx=0 tx=0).stop 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3351283162 shutdown_connections 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55e836b9d280 0x55e836b9f690 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd0a000640 0 client.15561 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87b80 0x55e836b91970 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3351283162 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87190 0x55e836b91430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3351283162 >> 172.21.15.7:0/3351283162 conn(0x55e836b46930 msgr2=0x55e836b79570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.193+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3351283162 shutdown_connections 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.194+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3351283162 wait complete. 2024-03-18T01:38:45.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.194+0000 7fdd2e1e1740 20 client.15561 trim_cache size 0 max 16384 2024-03-18T01:38:45.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.233+0000 7fdd2e1e1740 1 Processor -- start 2024-03-18T01:38:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.233+0000 7fdd2e1e1740 1 -- start start 2024-03-18T01:38:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.233+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b88e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87b80 0x55e836b98970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b89380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241a7f90 con 0x55e836b87190 2024-03-18T01:38:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241a9ad0 con 0x55e836b8ccf0 2024-03-18T01:38:45.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241a8660 con 0x55e836b87b80 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd29400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b89380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd23e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87b80 0x55e836b98970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd29400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b89380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55462/0 (socket says 172.21.15.7:55462) 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd29400640 1 -- 172.21.15.7:0/644491156 learned_addr learned my addr 172.21.15.7:0/644491156 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd28a00640 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b88e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd29400640 1 -- 172.21.15.7:0/644491156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87b80 msgr2=0x55e836b98970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd29400640 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87b80 0x55e836b98970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.234+0000 7fdd29400640 1 -- 172.21.15.7:0/644491156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 msgr2=0x55e836b88e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.235+0000 7fdd29400640 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b88e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.235+0000 7fdd29400640 1 -- 172.21.15.7:0/644491156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8367b6170 con 0x55e836b8ccf0 2024-03-18T01:38:45.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.235+0000 7fdd28a00640 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b88e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:45.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.235+0000 7fdd29400640 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b89380 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0039f0 tx=0x7fdd1c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4b0179836fd56a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.235+0000 7fdd15200640 1 -- 172.21.15.7:0/644491156 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c004530 con 0x55e836b8ccf0 2024-03-18T01:38:45.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.235+0000 7fdd15200640 1 -- 172.21.15.7:0/644491156 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1c0046d0 con 0x55e836b8ccf0 2024-03-18T01:38:45.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.235+0000 7fdd15200640 1 -- 172.21.15.7:0/644491156 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c028620 con 0x55e836b8ccf0 2024-03-18T01:38:45.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.236+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/644491156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 msgr2=0x55e836b89380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.236+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b89380 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0039f0 tx=0x7fdd1c014000 comp rx=0 tx=0).stop 2024-03-18T01:38:45.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.237+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/644491156 shutdown_connections 2024-03-18T01:38:45.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.237+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b8ccf0 0x55e836b89380 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.237+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b87b80 0x55e836b98970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.237+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/644491156 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b88e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.237+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/644491156 >> 172.21.15.7:0/644491156 conn(0x55e836b46930 msgr2=0x55e836b91430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.237+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/644491156 shutdown_connections 2024-03-18T01:38:45.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.237+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/644491156 wait complete. 2024-03-18T01:38:45.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.237+0000 7fdd2e1e1740 1 -- start start 2024-03-18T01:38:45.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.238+0000 7fdd2e1e1740 1 Processor -- start 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.238+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b96640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.238+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87b80 0x55e836b96b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.238+0000 7fdd2e1e1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b97b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.238+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241a9400 con 0x55e836b87190 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.238+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241a8d30 con 0x55e836b87b80 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.238+0000 7fdd2e1e1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd241a5f70 con 0x55e836b8ccf0 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.238+0000 7fdd2e1e1740 10 client.0 _ll_register_callbacks cb 0x7fdd18005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd28a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b96640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd28a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b96640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:53662/0 (socket says 172.21.15.7:53662) 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd28a00640 1 -- 172.21.15.7:0/3872348565 learned_addr learned my addr 172.21.15.7:0/3872348565 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd29400640 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b97b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd23e00640 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87b80 0x55e836b96b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd29400640 1 -- 172.21.15.7:0/3872348565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87b80 msgr2=0x55e836b96b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd29400640 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87b80 0x55e836b96b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd29400640 1 -- 172.21.15.7:0/3872348565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 msgr2=0x55e836b96640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd29400640 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b96640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd29400640 1 -- 172.21.15.7:0/3872348565 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e836b98180 con 0x55e836b8ccf0 2024-03-18T01:38:45.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.239+0000 7fdd28a00640 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b96640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:38:45.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.240+0000 7fdd29400640 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b97b90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c005c10 tx=0x7fdd1c01e620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57725656493c98cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.240+0000 7fdd0a000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:45.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.240+0000 7fdd0a000640 1 -- 172.21.15.7:0/3872348565 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c004530 con 0x55e836b8ccf0 2024-03-18T01:38:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.241+0000 7fdd0a000640 1 -- 172.21.15.7:0/3872348565 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdd1c025040 con 0x55e836b8ccf0 2024-03-18T01:38:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.241+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3872348565 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd1c0046d0 con 0x55e836b8ccf0 2024-03-18T01:38:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.241+0000 7fdd0a000640 1 -- 172.21.15.7:0/3872348565 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdd1c01ea20 con 0x55e836b8ccf0 2024-03-18T01:38:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.241+0000 7fdd2e1e1740 10 client.25732 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.241+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3872348565 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e836c03b10 con 0x55e836b8ccf0 2024-03-18T01:38:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.241+0000 7fdd0a000640 1 -- 172.21.15.7:0/3872348565 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1c0046d0 con 0x55e836b8ccf0 2024-03-18T01:38:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.242+0000 7fdd0a000640 1 -- 172.21.15.7:0/3872348565 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdd1c029420 con 0x55e836b8ccf0 2024-03-18T01:38:45.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.242+0000 7fdd2e1e1740 10 client.25732 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.242+0000 7fdd2e1e1740 10 client.25732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.242+0000 7fdd2e1e1740 20 client.25732 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.242+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55e836b9d280 0x55e836c06520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.242+0000 7fdd2e1e1740 4 client.25732 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T01:38:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.242+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3872348565 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e836ae9810 con 0x55e836b9d280 2024-03-18T01:38:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.242+0000 7fdd28a00640 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55e836b9d280 0x55e836c06520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.243+0000 7fdd28a00640 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55e836b9d280 0x55e836c06520 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fcf2283908a9cf9 server_cookie=121620c28d2953ed in_seq=0 out_seq=0 2024-03-18T01:38:45.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.243+0000 7fdd0a000640 10 client.25732 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:45.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.244+0000 7fdd0a000640 1 -- 172.21.15.7:0/3872348565 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e836ae9810 con 0x55e836b9d280 2024-03-18T01:38:45.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.244+0000 7fdd0a000640 10 client.25732 handle_command_reply: tid=0 2024-03-18T01:38:45.245 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.244+0000 7fdd2e1e1740 1 client.25732 shutdown 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.244+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3872348565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 msgr2=0x55e836b97b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.244+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b97b90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c005c10 tx=0x7fdd1c01e620 comp rx=0 tx=0).stop 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3872348565 shutdown_connections 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55e836b9d280 0x55e836c06520 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e836b8ccf0 0x55e836b97b90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e836b87b80 0x55e836b96b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 1 --2- 172.21.15.7:0/3872348565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e836b87190 0x55e836b96640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3872348565 >> 172.21.15.7:0/3872348565 conn(0x55e836b46930 msgr2=0x55e836b79570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd0a000640 0 client.25732 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:45.246 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3872348565 shutdown_connections 2024-03-18T01:38:45.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 1 -- 172.21.15.7:0/3872348565 wait complete. 2024-03-18T01:38:45.247 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.245+0000 7fdd2e1e1740 20 client.25732 trim_cache size 0 max 16384 2024-03-18T01:38:45.247 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:38:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-03-18T01:38:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "12c5e747-be97-4932-9dcf-a9996e9c5049", 2024-03-18T01:38:45.247 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-03-18T01:38:45.247 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:38:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 -- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd0c0a2950 msgr2=0x7fdd0c0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd0c0a2950 0x7fdd0c0a4e10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c005fd0 tx=0x7fdd1c009500 comp rx=0 tx=0).stop 2024-03-18T01:38:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 -- 172.21.15.7:0/3789017826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24103840 msgr2=0x7fdd24199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24103840 0x7fdd24199b00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdd1800e9c0 tx=0x7fdd1800ee80 comp rx=0 tx=0).stop 2024-03-18T01:38:45.248 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 -- 172.21.15.7:0/3789017826 shutdown_connections 2024-03-18T01:38:45.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd241053f0 0x7fdd241a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdd0c0a2950 0x7fdd0c0a4e10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdd24104a30 0x7fdd2419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 --2- 172.21.15.7:0/3789017826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd24103840 0x7fdd24199b00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 -- 172.21.15.7:0/3789017826 >> 172.21.15.7:0/3789017826 conn(0x7fdd240fedb0 msgr2=0x7fdd24107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.247+0000 7fdd29e00640 1 -- 172.21.15.7:0/3789017826 shutdown_connections 2024-03-18T01:38:45.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.248+0000 7fdd29e00640 1 -- 172.21.15.7:0/3789017826 wait complete. 2024-03-18T01:38:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:44 smithi046 ceph-mon[36108]: pgmap v434: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:45.269 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:44 smithi007 ceph-mon[31996]: pgmap v434: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:38:45.273 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:38:45.273 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:38:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.467+0000 7f5ada600640 1 -- 172.21.15.7:0/1483361674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4077d80 msgr2=0x7f5ad4078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.467+0000 7f5ada600640 1 --2- 172.21.15.7:0/1483361674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4077d80 0x7f5ad4078620 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009be0 tx=0x7f5acc02bef0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.467+0000 7f5ada600640 1 -- 172.21.15.7:0/1483361674 shutdown_connections 2024-03-18T01:38:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.467+0000 7f5ada600640 1 --2- 172.21.15.7:0/1483361674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4077d80 0x7f5ad4078620 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.467+0000 7f5ada600640 1 --2- 172.21.15.7:0/1483361674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ad40773e0 0x7f5ad4077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.467+0000 7f5ada600640 1 --2- 172.21.15.7:0/1483361674 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ad40761a0 0x7f5ad40765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.467+0000 7f5ada600640 1 -- 172.21.15.7:0/1483361674 >> 172.21.15.7:0/1483361674 conn(0x7f5ad40fefa0 msgr2=0x7f5ad41013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.468+0000 7f5ada600640 1 -- 172.21.15.7:0/1483361674 shutdown_connections 2024-03-18T01:38:45.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.468+0000 7f5ada600640 1 -- 172.21.15.7:0/1483361674 wait complete. 2024-03-18T01:38:45.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ada600640 1 Processor -- start 2024-03-18T01:38:45.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ada600640 1 -- start start 2024-03-18T01:38:45.473 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ada600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ad40761a0 0x7f5ad419e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ada600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ad40773e0 0x7f5ad419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ada600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4077d80 0x7f5ad41a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ada600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4113ae0 con 0x7f5ad40773e0 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ada600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4113960 con 0x7f5ad40761a0 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ada600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad4113c60 con 0x7f5ad4077d80 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.469+0000 7f5ad3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ad40761a0 0x7f5ad419e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.470+0000 7f5ad3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ad40761a0 0x7f5ad419e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55480/0 (socket says 172.21.15.7:55480) 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.470+0000 7f5ad3400640 1 -- 172.21.15.7:0/77436838 learned_addr learned my addr 172.21.15.7:0/77436838 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.474 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.470+0000 7f5ad3e00640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4077d80 0x7f5ad41a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.470+0000 7f5ad3400640 1 -- 172.21.15.7:0/77436838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4077d80 msgr2=0x7f5ad41a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.470+0000 7f5ad3400640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4077d80 0x7f5ad41a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.470+0000 7f5ad3400640 1 -- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ad40773e0 msgr2=0x7f5ad419e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.470+0000 7f5ad3400640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ad40773e0 0x7f5ad419e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.470+0000 7f5ad3400640 1 -- 172.21.15.7:0/77436838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad41a5c80 con 0x7f5ad40761a0 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.471+0000 7f5ad3400640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ad40761a0 0x7f5ad419e060 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400d950 tx=0x7f5ac400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.471+0000 7f5acbe00640 1 -- 172.21.15.7:0/77436838 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4017070 con 0x7f5ad40761a0 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.471+0000 7f5acbe00640 1 -- 172.21.15.7:0/77436838 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac4009c30 con 0x7f5ad40761a0 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.471+0000 7f5ada600640 1 -- 172.21.15.7:0/77436838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ad41a5f70 con 0x7f5ad40761a0 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.471+0000 7f5acbe00640 1 -- 172.21.15.7:0/77436838 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400f6b0 con 0x7f5ad40761a0 2024-03-18T01:38:45.475 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.471+0000 7f5ada600640 1 -- 172.21.15.7:0/77436838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ad41a6430 con 0x7f5ad40761a0 2024-03-18T01:38:45.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.474+0000 7f5add398740 1 Processor -- start 2024-03-18T01:38:45.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.475+0000 7f5add398740 1 -- start start 2024-03-18T01:38:45.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.475+0000 7f5add398740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83dec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.476 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.475+0000 7f5add398740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83df200 0x55cbe83e3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.475+0000 7f5add398740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83e4370 0x55cbe83e6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.475+0000 7f5add398740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55cbe83431f0 con 0x55cbe83df200 2024-03-18T01:38:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.475+0000 7f5add398740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55cbe82eba80 con 0x55cbe83de810 2024-03-18T01:38:45.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.475+0000 7f5add398740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cbe833f290 con 0x55cbe83e4370 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.475+0000 7f5ad2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83e4370 0x55cbe83e6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83e4370 0x55cbe83e6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34816/0 (socket says 172.21.15.7:34816) 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad2a00640 1 -- 172.21.15.7:0/3222904723 learned_addr learned my addr 172.21.15.7:0/3222904723 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad3e00640 1 --2- 172.21.15.7:0/3222904723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83dec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad2a00640 1 -- 172.21.15.7:0/3222904723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 msgr2=0x55cbe83dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/3222904723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83dec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad2a00640 1 -- 172.21.15.7:0/3222904723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83df200 msgr2=0x55cbe83e3d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/3222904723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83df200 0x55cbe83e3d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad2a00640 1 -- 172.21.15.7:0/3222904723 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe81217d0 con 0x55cbe83e4370 2024-03-18T01:38:45.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/3222904723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83e4370 0x55cbe83e6800 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009b30 tx=0x7f5ac000e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34454e0692f69a4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.476+0000 7f5acbe00640 1 -- 172.21.15.7:0/77436838 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f5ac40109d0 con 0x7f5ad40761a0 2024-03-18T01:38:45.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.477+0000 7f5ac8c00640 1 -- 172.21.15.7:0/3222904723 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac000ed80 con 0x55cbe83e4370 2024-03-18T01:38:45.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.477+0000 7f5ac8c00640 1 -- 172.21.15.7:0/3222904723 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac0010910 con 0x55cbe83e4370 2024-03-18T01:38:45.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.477+0000 7f5ac8c00640 1 -- 172.21.15.7:0/3222904723 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0018a80 con 0x55cbe83e4370 2024-03-18T01:38:45.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.477+0000 7f5add398740 1 -- 172.21.15.7:0/3222904723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83e4370 msgr2=0x55cbe83e6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.479 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.477+0000 7f5add398740 1 --2- 172.21.15.7:0/3222904723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83e4370 0x55cbe83e6800 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009b30 tx=0x7f5ac000e350 comp rx=0 tx=0).stop 2024-03-18T01:38:45.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.479+0000 7f5acbe00640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5ab80a2800 0x7f5ab80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.479+0000 7f5add398740 1 -- 172.21.15.7:0/3222904723 shutdown_connections 2024-03-18T01:38:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.479+0000 7f5acbe00640 1 -- 172.21.15.7:0/77436838 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5ac40ce9f0 con 0x7f5ad40761a0 2024-03-18T01:38:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.479+0000 7f5add398740 1 --2- 172.21.15.7:0/3222904723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83e4370 0x55cbe83e6800 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.479+0000 7f5add398740 1 --2- 172.21.15.7:0/3222904723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83df200 0x55cbe83e3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.479+0000 7f5add398740 1 --2- 172.21.15.7:0/3222904723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83dec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.479+0000 7f5add398740 1 -- 172.21.15.7:0/3222904723 >> 172.21.15.7:0/3222904723 conn(0x55cbe83d9d80 msgr2=0x55cbe83da180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.480+0000 7f5add398740 1 -- 172.21.15.7:0/3222904723 shutdown_connections 2024-03-18T01:38:45.481 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.480+0000 7f5ad3e00640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5ab80a2800 0x7f5ab80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.480+0000 7f5add398740 1 -- 172.21.15.7:0/3222904723 wait complete. 2024-03-18T01:38:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.480+0000 7f5add398740 1 -- start start 2024-03-18T01:38:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5add398740 1 Processor -- start 2024-03-18T01:38:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5ad3e00640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5ab80a2800 0x7f5ab80a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5acc006010 tx=0x7f5acc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.482 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5add398740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83e8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5add398740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83e9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5add398740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 0x55cbe83f0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5add398740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55cbe83e7320 con 0x55cbe83e4370 2024-03-18T01:38:45.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5add398740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55cbe82feed0 con 0x55cbe83de810 2024-03-18T01:38:45.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5add398740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55cbe83e74a0 con 0x55cbe83df200 2024-03-18T01:38:45.483 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.481+0000 7f5add398740 10 client.0 _ll_register_callbacks cb 0x55cbe833a220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:45.484 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83e8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83e8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55512/0 (socket says 172.21.15.7:55512) 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 -- 172.21.15.7:0/723856749 learned_addr learned my addr 172.21.15.7:0/723856749 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 -- 172.21.15.7:0/723856749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 msgr2=0x55cbe83e9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83e9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 -- 172.21.15.7:0/723856749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 msgr2=0x55cbe83f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 0x55cbe83f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 -- 172.21.15.7:0/723856749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe80573f0 con 0x55cbe83de810 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5ad3400640 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83e8df0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5ac409afc0 tx=0x7f5ac4009ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7fbf4afbd0fb27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.482+0000 7f5abd600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:38:45.486 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.483+0000 7f5abd600640 1 -- 172.21.15.7:0/723856749 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac4020070 con 0x55cbe83de810 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.483+0000 7f5abd600640 1 -- 172.21.15.7:0/723856749 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5ac4095ba0 con 0x55cbe83de810 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.483+0000 7f5add398740 1 -- 172.21.15.7:0/723856749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ac0010910 con 0x55cbe83de810 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.483+0000 7f5abd600640 1 -- 172.21.15.7:0/723856749 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5ac409f180 con 0x55cbe83de810 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.483+0000 7f5add398740 10 client.25763 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.483+0000 7f5add398740 1 -- 172.21.15.7:0/723856749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cbe80170f0 con 0x55cbe83de810 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.483+0000 7f5abd600640 1 -- 172.21.15.7:0/723856749 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac0010910 con 0x55cbe83de810 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.483+0000 7f5abd600640 1 -- 172.21.15.7:0/723856749 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5ac409a480 con 0x55cbe83de810 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.484+0000 7f5add398740 10 client.25763 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.484+0000 7f5add398740 10 client.25763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.484+0000 7f5add398740 20 client.25763 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.484+0000 7f5add398740 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55cbe83f4c70 0x55cbe83f7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.484+0000 7f5add398740 4 client.25763 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.484+0000 7f5add398740 1 -- 172.21.15.7:0/723856749 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbe8356be0 con 0x55cbe83f4c70 2024-03-18T01:38:45.487 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.484+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55cbe83f4c70 0x55cbe83f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.485+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55cbe83f4c70 0x55cbe83f7110 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e07aa879a7d50f5 server_cookie=8bd455e81358e809 in_seq=0 out_seq=0 2024-03-18T01:38:45.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.485+0000 7f5abd600640 10 client.25763 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.485+0000 7f5abd600640 1 -- 172.21.15.7:0/723856749 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cbe8356be0 con 0x55cbe83f4c70 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.485+0000 7f5abd600640 10 client.25763 handle_command_reply: tid=0 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.485+0000 7f5add398740 1 client.25763 shutdown 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 -- 172.21.15.7:0/723856749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 msgr2=0x55cbe83e8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83e8df0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5ac409afc0 tx=0x7f5ac4009ea0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 -- 172.21.15.7:0/723856749 shutdown_connections 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55cbe83f4c70 0x55cbe83f7110 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 0x55cbe83f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 --2- 172.21.15.7:0/723856749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83e8df0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 -- 172.21.15.7:0/723856749 >> 172.21.15.7:0/723856749 conn(0x55cbe83d9d80 msgr2=0x55cbe83d8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5abd600640 0 client.25763 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:45.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 -- 172.21.15.7:0/723856749 shutdown_connections 2024-03-18T01:38:45.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 1 -- 172.21.15.7:0/723856749 wait complete. 2024-03-18T01:38:45.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.486+0000 7f5add398740 20 client.25763 trim_cache size 0 max 16384 2024-03-18T01:38:45.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5add398740 1 Processor -- start 2024-03-18T01:38:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5add398740 1 -- start start 2024-03-18T01:38:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5add398740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83de810 0x55cbe83e8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5add398740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83f0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5add398740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83e4370 0x55cbe83e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5add398740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b0880 con 0x55cbe83de810 2024-03-18T01:38:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5add398740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b23c0 con 0x55cbe83e4370 2024-03-18T01:38:45.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5add398740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b0f50 con 0x55cbe83df200 2024-03-18T01:38:45.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5ad2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.505+0000 7f5ad3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83de810 0x55cbe83e8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83f0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34826/0 (socket says 172.21.15.7:34826) 2024-03-18T01:38:45.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad2a00640 1 -- 172.21.15.7:0/490324447 learned_addr learned my addr 172.21.15.7:0/490324447 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad3e00640 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83e4370 0x55cbe83e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad2a00640 1 -- 172.21.15.7:0/490324447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83e4370 msgr2=0x55cbe83e8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83e4370 0x55cbe83e8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad2a00640 1 -- 172.21.15.7:0/490324447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83de810 msgr2=0x55cbe83e8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83de810 0x55cbe83e8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad2a00640 1 -- 172.21.15.7:0/490324447 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbe83d4370 con 0x55cbe83df200 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad3e00640 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83e4370 0x55cbe83e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.506+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83f0330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000e850 tx=0x7f5ac0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21172bc1ec174fac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.507+0000 7f5abfe00640 1 -- 172.21.15.7:0/490324447 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac002f070 con 0x55cbe83df200 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.507+0000 7f5abfe00640 1 -- 172.21.15.7:0/490324447 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac0004520 con 0x55cbe83df200 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.507+0000 7f5abfe00640 1 -- 172.21.15.7:0/490324447 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0005460 con 0x55cbe83df200 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 -- 172.21.15.7:0/490324447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 msgr2=0x55cbe83f0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83f0330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000e850 tx=0x7f5ac0009260 comp rx=0 tx=0).stop 2024-03-18T01:38:45.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 -- 172.21.15.7:0/490324447 shutdown_connections 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83e4370 0x55cbe83e8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83f0330 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 --2- 172.21.15.7:0/490324447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83de810 0x55cbe83e8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 -- 172.21.15.7:0/490324447 >> 172.21.15.7:0/490324447 conn(0x55cbe83d9d80 msgr2=0x55cbe83e07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 -- 172.21.15.7:0/490324447 shutdown_connections 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 -- 172.21.15.7:0/490324447 wait complete. 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.508+0000 7f5add398740 1 -- start start 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.509+0000 7f5add398740 1 Processor -- start 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.509+0000 7f5add398740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83edc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.509+0000 7f5add398740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.509+0000 7f5add398740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 0x55cbe83ef160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.509+0000 7f5add398740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b1cf0 con 0x55cbe83e4370 2024-03-18T01:38:45.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.509+0000 7f5add398740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41b1620 con 0x55cbe83de810 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.509+0000 7f5add398740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad41ae860 con 0x55cbe83df200 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.509+0000 7f5add398740 10 client.0 _ll_register_callbacks cb 0x7f5ac400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83edc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83edc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55516/0 (socket says 172.21.15.7:55516) 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad3400640 1 -- 172.21.15.7:0/4150371447 learned_addr learned my addr 172.21.15.7:0/4150371447 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad3e00640 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 0x55cbe83ef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad2a00640 1 -- 172.21.15.7:0/4150371447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 msgr2=0x55cbe83edc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83edc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad2a00640 1 -- 172.21.15.7:0/4150371447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 msgr2=0x55cbe83ef160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 0x55cbe83ef160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.510+0000 7f5ad2a00640 1 -- 172.21.15.7:0/4150371447 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbe83e7ed0 con 0x55cbe83df200 2024-03-18T01:38:45.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5ad2a00640 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83ee1c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f010 tx=0x7f5ac0005bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7fa88e3e27c0e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:38:45.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5abd600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:38:45.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5abd600640 1 -- 172.21.15.7:0/4150371447 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac002f070 con 0x55cbe83df200 2024-03-18T01:38:45.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5abd600640 1 -- 172.21.15.7:0/4150371447 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5ac00053e0 con 0x55cbe83df200 2024-03-18T01:38:45.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5add398740 1 -- 172.21.15.7:0/4150371447 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ac0004520 con 0x55cbe83df200 2024-03-18T01:38:45.517 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5abd600640 1 -- 172.21.15.7:0/4150371447 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5ac00055e0 con 0x55cbe83df200 2024-03-18T01:38:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5add398740 10 client.25747 fetch_fsmap learned FSMap version 28 2024-03-18T01:38:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5add398740 1 -- 172.21.15.7:0/4150371447 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cbe845b070 con 0x55cbe83df200 2024-03-18T01:38:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.511+0000 7f5abd600640 1 -- 172.21.15.7:0/4150371447 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ac0004520 con 0x55cbe83df200 2024-03-18T01:38:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.512+0000 7f5abd600640 1 -- 172.21.15.7:0/4150371447 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f5ac002a040 con 0x55cbe83df200 2024-03-18T01:38:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.512+0000 7f5add398740 10 client.25747 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:38:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.512+0000 7f5add398740 10 client.25747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:38:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.512+0000 7f5add398740 20 client.25747 populate_metadata read hostname 'smithi007' 2024-03-18T01:38:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.512+0000 7f5add398740 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55cbe845b770 0x55cbe845db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:38:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.512+0000 7f5add398740 4 client.25747 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:38:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.512+0000 7f5add398740 1 -- 172.21.15.7:0/4150371447 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cbe8350730 con 0x55cbe845b770 2024-03-18T01:38:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.512+0000 7f5ad3400640 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55cbe845b770 0x55cbe845db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:38:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5ad3400640 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55cbe845b770 0x55cbe845db90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3b600e49514fbac server_cookie=95b77571a8236c6c in_seq=0 out_seq=0 2024-03-18T01:38:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5abd600640 10 client.25747 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5abd600640 1 -- 172.21.15.7:0/4150371447 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cbe8350730 con 0x55cbe845b770 2024-03-18T01:38:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5abd600640 10 client.25747 handle_command_reply: tid=0 2024-03-18T01:38:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 client.25747 shutdown 2024-03-18T01:38:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 -- 172.21.15.7:0/4150371447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 msgr2=0x55cbe83ee1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83ee1c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000f010 tx=0x7f5ac0005bb0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 -- 172.21.15.7:0/4150371447 shutdown_connections 2024-03-18T01:38:45.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55cbe845b770 0x55cbe845db90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55cbe83e4370 0x55cbe83ef160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5abd600640 0 client.25747 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:38:45.521 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55cbe83df200 0x55cbe83ee1c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 --2- 172.21.15.7:0/4150371447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cbe83de810 0x55cbe83edc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 -- 172.21.15.7:0/4150371447 >> 172.21.15.7:0/4150371447 conn(0x55cbe83d9d80 msgr2=0x55cbe83e8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.513+0000 7f5add398740 1 -- 172.21.15.7:0/4150371447 shutdown_connections 2024-03-18T01:38:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.514+0000 7f5add398740 1 -- 172.21.15.7:0/4150371447 wait complete. 2024-03-18T01:38:45.522 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.514+0000 7f5add398740 20 client.25747 trim_cache size 0 max 16384 2024-03-18T01:38:45.523 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:38:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T01:38:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:38:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "12c5e747-be97-4932-9dcf-a9996e9c5049": { 2024-03-18T01:38:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:38:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "12c5e747-be97-4932-9dcf-a9996e9c5049", 2024-03-18T01:38:45.523 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:38:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:38:45.524 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:38:45.524 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:38:45.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.514+0000 7f5ada600640 1 -- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5ab80a2800 msgr2=0x7f5ab80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.514+0000 7f5ada600640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5ab80a2800 0x7f5ab80a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5acc006010 tx=0x7f5acc004a40 comp rx=0 tx=0).stop 2024-03-18T01:38:45.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.514+0000 7f5ada600640 1 -- 172.21.15.7:0/77436838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ad40761a0 msgr2=0x7f5ad419e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:38:45.525 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.514+0000 7f5ada600640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ad40761a0 0x7f5ad419e060 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400d950 tx=0x7f5ac400de10 comp rx=0 tx=0).stop 2024-03-18T01:38:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.515+0000 7f5ada600640 1 -- 172.21.15.7:0/77436838 shutdown_connections 2024-03-18T01:38:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.515+0000 7f5ada600640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4077d80 0x7f5ad41a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.515+0000 7f5ada600640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5ab80a2800 0x7f5ab80a4cc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.515+0000 7f5ada600640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5ad40773e0 0x7f5ad419e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.515+0000 7f5ada600640 1 --2- 172.21.15.7:0/77436838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ad40761a0 0x7f5ad419e060 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:38:45.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.515+0000 7f5ada600640 1 -- 172.21.15.7:0/77436838 >> 172.21.15.7:0/77436838 conn(0x7f5ad40fefa0 msgr2=0x7f5ad41005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:38:45.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.515+0000 7f5ada600640 1 -- 172.21.15.7:0/77436838 shutdown_connections 2024-03-18T01:38:45.527 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:38:45.515+0000 7f5ada600640 1 -- 172.21.15.7:0/77436838 wait complete. 2024-03-18T01:38:45.535 INFO:tasks.cephfs.filesystem:scrub status for tag:12c5e747-be97-4932-9dcf-a9996e9c5049 - {'path': '/', 'tag': '12c5e747-be97-4932-9dcf-a9996e9c5049', 'options': 'recursive,force'} 2024-03-18T01:38:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:45 smithi203 ceph-mon[36345]: scrub queued for path: / 2024-03-18T01:38:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:45 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:45 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:38:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:45 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:45 smithi046 ceph-mon[36108]: scrub queued for path: / 2024-03-18T01:38:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:45 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:45 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:38:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:45 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:46.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:45 smithi007 ceph-mon[31996]: scrub queued for path: / 2024-03-18T01:38:46.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:45 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:46.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:45 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:38:46.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:45 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:38:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:46 smithi203 ceph-mon[36345]: pgmap v435: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:47.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:46 smithi046 ceph-mon[36108]: pgmap v435: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:47.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:46 smithi007 ceph-mon[31996]: pgmap v435: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:38:49.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:48 smithi203 ceph-mon[36345]: pgmap v436: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T01:38:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:48 smithi046 ceph-mon[36108]: pgmap v436: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T01:38:49.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:48 smithi007 ceph-mon[31996]: pgmap v436: 129 pgs: 129 active+clean; 656 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T01:38:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:50 smithi203 ceph-mon[36345]: pgmap v437: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:50 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:38:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:50 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:38:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:50 smithi046 ceph-mon[36108]: pgmap v437: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:50 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:38:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:50 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:38:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:50 smithi007 ceph-mon[31996]: pgmap v437: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:50 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:38:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:50 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:38:52.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:51 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:38:52.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:51 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:38:52.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:51 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:38:52.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:51 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:38:52.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:51 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:38:52.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:51 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:38:52.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:51 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:38:52.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:51 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:38:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:51 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:38:53.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:52 smithi203 ceph-mon[36345]: pgmap v438: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:53.170 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:52 smithi046 ceph-mon[36108]: pgmap v438: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:53.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:52 smithi007 ceph-mon[31996]: pgmap v438: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:55.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:54 smithi203 ceph-mon[36345]: pgmap v439: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:54 smithi046 ceph-mon[36108]: pgmap v439: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:54 smithi007 ceph-mon[31996]: pgmap v439: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:56 smithi046 ceph-mon[36108]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:57.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:56 smithi007 ceph-mon[31996]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:56 smithi203 ceph-mon[36345]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:38:58 smithi046 ceph-mon[36108]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:59.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:38:58 smithi007 ceph-mon[31996]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:38:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:38:58 smithi203 ceph-mon[36345]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:39:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:00 smithi046 ceph-mon[36108]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T01:39:01.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:00 smithi007 ceph-mon[31996]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T01:39:01.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:00 smithi203 ceph-mon[36345]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T01:39:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:02 smithi046 ceph-mon[36108]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:39:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:02 smithi007 ceph-mon[31996]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:39:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:02 smithi203 ceph-mon[36345]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:39:03.911 INFO:tasks.workunit.client.0.smithi007.stderr:Note: switching to 'c382027d5c66ef4e0aa1131ea3e8080c658d3184'. 2024-03-18T01:39:03.911 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-18T01:39:03.911 INFO:tasks.workunit.client.0.smithi007.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr: git switch -c 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr:Or undo this operation with: 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr: git switch - 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-18T01:39:03.912 INFO:tasks.workunit.client.0.smithi007.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T01:39:03.913 INFO:tasks.workunit.client.0.smithi007.stderr: 2024-03-18T01:39:03.913 INFO:tasks.workunit.client.0.smithi007.stderr:HEAD is now at c382027d5c6 ignorelist mon down 2024-03-18T01:39:03.924 DEBUG:teuthology.orchestra.run.smithi007:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T01:39:03.989 INFO:tasks.workunit.client.0.smithi007.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T01:39:03.998 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T01:39:03.998 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T01:39:04.965 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T01:39:05.126 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T01:39:05.214 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T01:39:05.220 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T01:39:05.220 INFO:tasks.workunit.client.0.smithi007.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T01:39:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:04 smithi046 ceph-mon[36108]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:39:05.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:04 smithi007 ceph-mon[31996]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:39:05.320 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T01:39:05.331 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:39:05.331 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T01:39:05.382 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-03-18T01:39:05.383 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-03-18T01:39:05.383 DEBUG:teuthology.orchestra.run.smithi007:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c382027d5c66ef4e0aa1131ea3e8080c658d3184 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-18T01:39:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:04 smithi203 ceph-mon[36345]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:39:05.460 INFO:tasks.workunit.client.0.smithi007.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-18T01:39:05.462 INFO:tasks.workunit.client.0.smithi007.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-18T01:39:05.463 INFO:tasks.workunit.client.0.smithi007.stderr:+ trap -- retry EXIT 2024-03-18T01:39:05.463 INFO:tasks.workunit.client.0.smithi007.stderr:+ rm -rf ffsb 2024-03-18T01:39:05.466 INFO:tasks.workunit.client.0.smithi007.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-03-18T01:39:05.473 INFO:tasks.workunit.client.0.smithi007.stderr:Cloning into 'ffsb'... 2024-03-18T01:39:07.022 INFO:tasks.workunit.client.0.smithi007.stderr:+ trap - EXIT 2024-03-18T01:39:07.022 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd ffsb 2024-03-18T01:39:07.022 INFO:tasks.workunit.client.0.smithi007.stderr:+ ./configure 2024-03-18T01:39:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:06 smithi046 ceph-mon[36108]: pgmap v445: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:39:07.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:06 smithi007 ceph-mon[31996]: pgmap v445: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:39:07.359 INFO:tasks.workunit.client.0.smithi007.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-18T01:39:07.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:06 smithi203 ceph-mon[36345]: pgmap v445: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:39:08.379 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether build environment is sane... yes 2024-03-18T01:39:08.403 INFO:tasks.workunit.client.0.smithi007.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-18T01:39:08.403 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gawk... gawk 2024-03-18T01:39:08.439 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether make sets $(MAKE)... yes 2024-03-18T01:39:08.541 INFO:tasks.workunit.client.0.smithi007.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-18T01:39:08.542 INFO:tasks.workunit.client.0.smithi007.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-18T01:39:08.563 INFO:tasks.workunit.client.0.smithi007.stdout:checking for style of include used by make... GNU 2024-03-18T01:39:08.569 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc... gcc 2024-03-18T01:39:08.709 INFO:tasks.workunit.client.0.smithi007.stdout:checking for C compiler default output file name... a.out 2024-03-18T01:39:08.717 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether the C compiler works... yes 2024-03-18T01:39:08.722 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are cross compiling... no 2024-03-18T01:39:08.816 INFO:tasks.workunit.client.0.smithi007.stdout:checking for suffix of executables... 2024-03-18T01:39:08.887 INFO:tasks.workunit.client.0.smithi007.stdout:checking for suffix of object files... o 2024-03-18T01:39:08.960 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are using the GNU C compiler... yes 2024-03-18T01:39:09.039 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether gcc accepts -g... yes 2024-03-18T01:39:09.205 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-18T01:39:09.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:08 smithi046 ceph-mon[36108]: pgmap v446: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:39:09.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:08 smithi007 ceph-mon[31996]: pgmap v446: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:39:09.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:08 smithi203 ceph-mon[36345]: pgmap v446: 129 pgs: 129 active+clean; 661 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T01:39:09.401 INFO:tasks.workunit.client.0.smithi007.stdout:checking dependency style of gcc... gcc3 2024-03-18T01:39:09.553 INFO:tasks.workunit.client.0.smithi007.stdout:checking how to run the C preprocessor... gcc -E 2024-03-18T01:39:09.687 INFO:tasks.workunit.client.0.smithi007.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-18T01:39:09.692 INFO:tasks.workunit.client.0.smithi007.stdout:checking for egrep... /usr/bin/grep -E 2024-03-18T01:39:09.724 INFO:tasks.workunit.client.0.smithi007.stdout:checking for AIX... no 2024-03-18T01:39:09.748 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc... (cached) gcc 2024-03-18T01:39:09.784 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-18T01:39:09.786 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-18T01:39:09.786 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-18T01:39:09.786 INFO:tasks.workunit.client.0.smithi007.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-18T01:39:09.903 INFO:tasks.workunit.client.0.smithi007.stdout:checking for main in -lm... yes 2024-03-18T01:39:10.023 INFO:tasks.workunit.client.0.smithi007.stdout:checking for main in -lpthread... yes 2024-03-18T01:39:10.414 INFO:tasks.workunit.client.0.smithi007.stdout:checking for ANSI C header files... yes 2024-03-18T01:39:10.555 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-18T01:39:10.663 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/types.h... yes 2024-03-18T01:39:10.836 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/stat.h... yes 2024-03-18T01:39:10.962 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdlib.h... yes 2024-03-18T01:39:11.072 INFO:tasks.workunit.client.0.smithi007.stdout:checking for string.h... yes 2024-03-18T01:39:11.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:10 smithi203 ceph-mon[36345]: pgmap v447: 129 pgs: 129 active+clean; 2.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 121 KiB/s wr, 16 op/s 2024-03-18T01:39:11.202 INFO:tasks.workunit.client.0.smithi007.stdout:checking for memory.h... yes 2024-03-18T01:39:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:10 smithi046 ceph-mon[36108]: pgmap v447: 129 pgs: 129 active+clean; 2.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 121 KiB/s wr, 16 op/s 2024-03-18T01:39:11.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:10 smithi007 ceph-mon[31996]: pgmap v447: 129 pgs: 129 active+clean; 2.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 121 KiB/s wr, 16 op/s 2024-03-18T01:39:11.306 INFO:tasks.workunit.client.0.smithi007.stdout:checking for strings.h... yes 2024-03-18T01:39:11.407 INFO:tasks.workunit.client.0.smithi007.stdout:checking for inttypes.h... yes 2024-03-18T01:39:11.517 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdint.h... yes 2024-03-18T01:39:11.632 INFO:tasks.workunit.client.0.smithi007.stdout:checking for unistd.h... yes 2024-03-18T01:39:11.774 INFO:tasks.workunit.client.0.smithi007.stdout:checking pthread.h usability... yes 2024-03-18T01:39:11.831 INFO:tasks.workunit.client.0.smithi007.stdout:checking pthread.h presence... yes 2024-03-18T01:39:11.832 INFO:tasks.workunit.client.0.smithi007.stdout:checking for pthread.h... yes 2024-03-18T01:39:11.967 INFO:tasks.workunit.client.0.smithi007.stdout:checking fcntl.h usability... yes 2024-03-18T01:39:12.025 INFO:tasks.workunit.client.0.smithi007.stdout:checking fcntl.h presence... yes 2024-03-18T01:39:12.026 INFO:tasks.workunit.client.0.smithi007.stdout:checking for fcntl.h... yes 2024-03-18T01:39:12.146 INFO:tasks.workunit.client.0.smithi007.stdout:checking limits.h usability... yes 2024-03-18T01:39:12.199 INFO:tasks.workunit.client.0.smithi007.stdout:checking limits.h presence... yes 2024-03-18T01:39:12.201 INFO:tasks.workunit.client.0.smithi007.stdout:checking for limits.h... yes 2024-03-18T01:39:12.236 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdint.h... (cached) yes 2024-03-18T01:39:12.347 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/time.h usability... yes 2024-03-18T01:39:12.409 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/time.h presence... yes 2024-03-18T01:39:12.411 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/time.h... yes 2024-03-18T01:39:12.445 INFO:tasks.workunit.client.0.smithi007.stdout:checking for unistd.h... (cached) yes 2024-03-18T01:39:12.555 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/vfs.h usability... yes 2024-03-18T01:39:12.608 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/vfs.h presence... yes 2024-03-18T01:39:12.610 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/vfs.h... yes 2024-03-18T01:39:12.715 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/limits.h usability... no 2024-03-18T01:39:12.766 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/limits.h presence... no 2024-03-18T01:39:12.767 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/limits.h... no 2024-03-18T01:39:12.847 INFO:tasks.workunit.client.0.smithi007.stdout:checking for an ANSI C-conforming const... yes 2024-03-18T01:39:12.936 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-18T01:39:12.938 INFO:tasks.workunit.client.0.smithi007.stdout:checking for function prototypes... yes 2024-03-18T01:39:13.010 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether setvbuf arguments are reversed... no 2024-03-18T01:39:13.164 INFO:tasks.workunit.client.0.smithi007.stdout:checking for system... yes 2024-03-18T01:39:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:12 smithi046 ceph-mon[36108]: pgmap v448: 129 pgs: 129 active+clean; 2.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 141 KiB/s wr, 20 op/s 2024-03-18T01:39:13.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:12 smithi007 ceph-mon[31996]: pgmap v448: 129 pgs: 129 active+clean; 2.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 141 KiB/s wr, 20 op/s 2024-03-18T01:39:13.311 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gettimeofday... yes 2024-03-18T01:39:13.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:12 smithi203 ceph-mon[36345]: pgmap v448: 129 pgs: 129 active+clean; 2.2 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 141 KiB/s wr, 20 op/s 2024-03-18T01:39:13.458 INFO:tasks.workunit.client.0.smithi007.stdout:checking for mkdir... yes 2024-03-18T01:39:13.615 INFO:tasks.workunit.client.0.smithi007.stdout:checking for strerror... yes 2024-03-18T01:39:13.765 INFO:tasks.workunit.client.0.smithi007.stdout:checking for open64... yes 2024-03-18T01:39:13.912 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stat64... yes 2024-03-18T01:39:14.063 INFO:tasks.workunit.client.0.smithi007.stdout:checking for fseeko64... yes 2024-03-18T01:39:14.214 INFO:tasks.workunit.client.0.smithi007.stdout:checking for lrand48_r... yes 2024-03-18T01:39:14.359 INFO:tasks.workunit.client.0.smithi007.stdout:checking for srand48_r... yes 2024-03-18T01:39:14.413 INFO:tasks.workunit.client.0.smithi007.stdout:configure: creating ./config.status 2024-03-18T01:39:14.691 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating Makefile 2024-03-18T01:39:14.731 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating config.h 2024-03-18T01:39:14.782 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: executing depfiles commands 2024-03-18T01:39:14.921 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: executing default commands 2024-03-18T01:39:15.001 INFO:tasks.workunit.client.0.smithi007.stderr:+ make 2024-03-18T01:39:15.010 INFO:tasks.workunit.client.0.smithi007.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-03-18T01:39:15.016 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-03-18T01:39:15.018 INFO:tasks.workunit.client.0.smithi007.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-03-18T01:39:15.018 INFO:tasks.workunit.client.0.smithi007.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-03-18T01:39:15.018 INFO:tasks.workunit.client.0.smithi007.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-03-18T01:39:15.018 INFO:tasks.workunit.client.0.smithi007.stderr: any GNU archive site. 2024-03-18T01:39:15.024 INFO:tasks.workunit.client.0.smithi007.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-03-18T01:39:15.029 INFO:tasks.workunit.client.0.smithi007.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-03-18T01:39:15.031 INFO:tasks.workunit.client.0.smithi007.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-03-18T01:39:15.032 INFO:tasks.workunit.client.0.smithi007.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-03-18T01:39:15.032 INFO:tasks.workunit.client.0.smithi007.stderr: You might want to install the `Automake' and `Perl' packages. 2024-03-18T01:39:15.032 INFO:tasks.workunit.client.0.smithi007.stderr: Grab them from any GNU archive site. 2024-03-18T01:39:15.118 INFO:tasks.workunit.client.0.smithi007.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-03-18T01:39:15.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:14 smithi046 ceph-mon[36108]: pgmap v449: 129 pgs: 129 active+clean; 3.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 272 KiB/s wr, 29 op/s 2024-03-18T01:39:15.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:14 smithi007 ceph-mon[31996]: pgmap v449: 129 pgs: 129 active+clean; 3.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 272 KiB/s wr, 29 op/s 2024-03-18T01:39:15.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:14 smithi203 ceph-mon[36345]: pgmap v449: 129 pgs: 129 active+clean; 3.7 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 272 KiB/s wr, 29 op/s 2024-03-18T01:39:15.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:39:15.536 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:39:15.669 INFO:tasks.workunit.client.0.smithi007.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-03-18T01:39:15.669 INFO:tasks.workunit.client.0.smithi007.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-03-18T01:39:15.669 INFO:tasks.workunit.client.0.smithi007.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-03-18T01:39:15.669 INFO:tasks.workunit.client.0.smithi007.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-03-18T01:39:15.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 -- 172.21.15.7:0/195419711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 msgr2=0x7f34c0100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 --2- 172.21.15.7:0/195419711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 0x7f34c0100e90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009900 tx=0x7f34ac02bc30 comp rx=0 tx=0).stop 2024-03-18T01:39:15.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 -- 172.21.15.7:0/195419711 shutdown_connections 2024-03-18T01:39:15.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 --2- 172.21.15.7:0/195419711 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34c01013d0 0x7f34c0101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 --2- 172.21.15.7:0/195419711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 0x7f34c0100e90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 --2- 172.21.15.7:0/195419711 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34c01000d0 0x7f34c01004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 -- 172.21.15.7:0/195419711 >> 172.21.15.7:0/195419711 conn(0x7f34c00fbc70 msgr2=0x7f34c00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:15.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 -- 172.21.15.7:0/195419711 shutdown_connections 2024-03-18T01:39:15.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.762+0000 7f34c7000640 1 -- 172.21.15.7:0/195419711 wait complete. 2024-03-18T01:39:15.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.763+0000 7f34c7000640 1 Processor -- start 2024-03-18T01:39:15.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c7000640 1 -- start start 2024-03-18T01:39:15.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c7000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34c01000d0 0x7f34c0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c7000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 0x7f34c019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c7000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34c01013d0 0x7f34c01a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c7000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f34c010f640 con 0x7f34c01000d0 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c7000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34c010f4c0 con 0x7f34c0100a30 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c7000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34c010f7c0 con 0x7f34c01013d0 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34bf400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 0x7f34c019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c5000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34c01013d0 0x7f34c01a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34bf400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 0x7f34c019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34380/0 (socket says 172.21.15.7:34380) 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34bf400640 1 -- 172.21.15.7:0/294982777 learned_addr learned my addr 172.21.15.7:0/294982777 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34c5000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34c01013d0 0x7f34c01a12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:60944/0 (socket says 172.21.15.7:60944) 2024-03-18T01:39:15.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.764+0000 7f34bfe00640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34c01000d0 0x7f34c0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34c5000640 1 -- 172.21.15.7:0/294982777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 msgr2=0x7f34c019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34c5000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 0x7f34c019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34c5000640 1 -- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34c01000d0 msgr2=0x7f34c0199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34c5000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34c01000d0 0x7f34c0199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34c5000640 1 -- 172.21.15.7:0/294982777 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c01a1940 con 0x7f34c01013d0 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34c5000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34c01013d0 0x7f34c01a12a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009fd0 tx=0x7f34b800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34bcc00640 1 -- 172.21.15.7:0/294982777 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b800ce20 con 0x7f34c01013d0 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34bcc00640 1 -- 172.21.15.7:0/294982777 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b8005020 con 0x7f34c01013d0 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34c7000640 1 -- 172.21.15.7:0/294982777 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c01a1c30 con 0x7f34c01013d0 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.765+0000 7f34bcc00640 1 -- 172.21.15.7:0/294982777 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b8010720 con 0x7f34c01013d0 2024-03-18T01:39:15.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.766+0000 7f34c7000640 1 -- 172.21.15.7:0/294982777 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c01a2180 con 0x7f34c01013d0 2024-03-18T01:39:15.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.769+0000 7f34c9d75740 1 Processor -- start 2024-03-18T01:39:15.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.769+0000 7f34bcc00640 1 -- 172.21.15.7:0/294982777 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f34b8005540 con 0x7f34c01013d0 2024-03-18T01:39:15.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.769+0000 7f34c9d75740 1 -- start start 2024-03-18T01:39:15.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861b810 0x55ba0861bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba08620d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba08623800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34c9d75740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ba085801f0 con 0x55ba08621370 2024-03-18T01:39:15.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34c9d75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ba08528a80 con 0x55ba0861c200 2024-03-18T01:39:15.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34c9d75740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ba0857c290 con 0x55ba0861b810 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861b810 0x55ba0861bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba08620d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba08620d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34392/0 (socket says 172.21.15.7:34392) 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34bfe00640 1 -- 172.21.15.7:0/559988047 learned_addr learned my addr 172.21.15.7:0/559988047 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.770+0000 7f34bf400640 1 --2- 172.21.15.7:0/559988047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba08623800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34bf400640 1 -- 172.21.15.7:0/559988047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 msgr2=0x55ba08620d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34bf400640 1 --2- 172.21.15.7:0/559988047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba08620d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34bf400640 1 -- 172.21.15.7:0/559988047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 msgr2=0x55ba08623800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34bf400640 1 --2- 172.21.15.7:0/559988047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba08623800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34bf400640 1 -- 172.21.15.7:0/559988047 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba0835e7d0 con 0x55ba0861b810 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34bf400640 1 --2- 172.21.15.7:0/559988047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861b810 0x55ba0861bc30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f34ac02c7b0 tx=0x7f34ac005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92c225e6c5868bdb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:15.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34bcc00640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f34a40a2880 0x7f34a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34bcc00640 1 -- 172.21.15.7:0/294982777 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f34b80cf4f0 con 0x7f34c01013d0 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34b5600640 1 -- 172.21.15.7:0/559988047 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac004220 con 0x55ba0861b810 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.771+0000 7f34b5600640 1 -- 172.21.15.7:0/559988047 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac0043c0 con 0x55ba0861b810 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.772+0000 7f34b5600640 1 -- 172.21.15.7:0/559988047 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0052f0 con 0x55ba0861b810 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.772+0000 7f34bfe00640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f34a40a2880 0x7f34a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.772+0000 7f34c9d75740 1 -- 172.21.15.7:0/559988047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861b810 msgr2=0x55ba0861bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.772+0000 7f34c9d75740 1 --2- 172.21.15.7:0/559988047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861b810 0x55ba0861bc30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f34ac02c7b0 tx=0x7f34ac005b70 comp rx=0 tx=0).stop 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.772+0000 7f34bfe00640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f34a40a2880 0x7f34a40a4d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x55ba0861c600 tx=0x7f34b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.773+0000 7f34c9d75740 1 -- 172.21.15.7:0/559988047 shutdown_connections 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.773+0000 7f34c9d75740 1 --2- 172.21.15.7:0/559988047 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba08623800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.773+0000 7f34c9d75740 1 --2- 172.21.15.7:0/559988047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba08620d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.773+0000 7f34c9d75740 1 --2- 172.21.15.7:0/559988047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861b810 0x55ba0861bc30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.773+0000 7f34c9d75740 1 -- 172.21.15.7:0/559988047 >> 172.21.15.7:0/559988047 conn(0x55ba08616d80 msgr2=0x55ba08617180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.773+0000 7f34c9d75740 1 -- 172.21.15.7:0/559988047 shutdown_connections 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.773+0000 7f34c9d75740 1 -- 172.21.15.7:0/559988047 wait complete. 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.773+0000 7f34c9d75740 1 -- start start 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.774+0000 7f34c9d75740 1 Processor -- start 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.774+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861b810 0x55ba08625de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.774+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba08626320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.774+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba0862d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.774+0000 7f34c9d75740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55ba08624260 con 0x55ba08621370 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.774+0000 7f34c9d75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55ba0853bed0 con 0x55ba0861b810 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.774+0000 7f34c9d75740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ba086243e0 con 0x55ba0861c200 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.774+0000 7f34c9d75740 10 client.0 _ll_register_callbacks cb 0x55ba08577220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:15.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861b810 0x55ba08625de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34c5000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba0862d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861b810 0x55ba08625de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:34400/0 (socket says 172.21.15.7:34400) 2024-03-18T01:39:15.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34bfe00640 1 -- 172.21.15.7:0/4081884956 learned_addr learned my addr 172.21.15.7:0/4081884956 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34bfe00640 1 -- 172.21.15.7:0/4081884956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 msgr2=0x55ba08626320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34bfe00640 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba08626320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34bfe00640 1 -- 172.21.15.7:0/4081884956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 msgr2=0x55ba0862d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34bfe00640 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba0862d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34bfe00640 1 -- 172.21.15.7:0/4081884956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba082943f0 con 0x55ba0861b810 2024-03-18T01:39:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.775+0000 7f34c5000640 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba0862d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.776+0000 7f34bfe00640 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861b810 0x55ba08625de0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f34b001e9b0 tx=0x7f34b0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdd82f9880db9b81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:15.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.776+0000 7f34aa000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.776+0000 7f34aa000640 1 -- 172.21.15.7:0/4081884956 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0004020 con 0x55ba0861b810 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.776+0000 7f34aa000640 1 -- 172.21.15.7:0/4081884956 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f34b0018030 con 0x55ba0861b810 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.776+0000 7f34c9d75740 1 -- 172.21.15.7:0/4081884956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34ac0043c0 con 0x55ba0861b810 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.776+0000 7f34aa000640 1 -- 172.21.15.7:0/4081884956 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f34b0033050 con 0x55ba0861b810 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.776+0000 7f34c9d75740 10 client.25781 fetch_fsmap learned FSMap version 28 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.776+0000 7f34c9d75740 1 -- 172.21.15.7:0/4081884956 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ba082540f0 con 0x55ba0861b810 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34aa000640 1 -- 172.21.15.7:0/4081884956 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac0043c0 con 0x55ba0861b810 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34aa000640 1 -- 172.21.15.7:0/4081884956 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f34b00048c0 con 0x55ba0861b810 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34c9d75740 10 client.25781 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34c9d75740 10 client.25781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34c9d75740 20 client.25781 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34c9d75740 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55ba08631d20 0x55ba086341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34c9d75740 4 client.25781 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34c9d75740 1 -- 172.21.15.7:0/4081884956 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba08593be0 con 0x55ba08631d20 2024-03-18T01:39:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.777+0000 7f34bf400640 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55ba08631d20 0x55ba086341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.778+0000 7f34bf400640 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55ba08631d20 0x55ba086341b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=519165f47b34f071 server_cookie=73309ab8ab3ae13b in_seq=0 out_seq=0 2024-03-18T01:39:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.778+0000 7f34aa000640 10 client.25781 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.779+0000 7f34aa000640 1 -- 172.21.15.7:0/4081884956 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ba08593be0 con 0x55ba08631d20 2024-03-18T01:39:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.779+0000 7f34aa000640 10 client.25781 handle_command_reply: tid=0 2024-03-18T01:39:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 client.25781 shutdown 2024-03-18T01:39:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 -- 172.21.15.7:0/4081884956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861b810 msgr2=0x55ba08625de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861b810 0x55ba08625de0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f34b001e9b0 tx=0x7f34b0004730 comp rx=0 tx=0).stop 2024-03-18T01:39:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 -- 172.21.15.7:0/4081884956 shutdown_connections 2024-03-18T01:39:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55ba08631d20 0x55ba086341b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba08621370 0x55ba0862d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba08626320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 --2- 172.21.15.7:0/4081884956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861b810 0x55ba08625de0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34aa000640 0 client.25781 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:15.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 -- 172.21.15.7:0/4081884956 >> 172.21.15.7:0/4081884956 conn(0x55ba08616d80 msgr2=0x55ba08615ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:15.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 -- 172.21.15.7:0/4081884956 shutdown_connections 2024-03-18T01:39:15.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 1 -- 172.21.15.7:0/4081884956 wait complete. 2024-03-18T01:39:15.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.780+0000 7f34c9d75740 20 client.25781 trim_cache size 0 max 16384 2024-03-18T01:39:15.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.799+0000 7f34c9d75740 1 Processor -- start 2024-03-18T01:39:15.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.799+0000 7f34c9d75740 1 -- start start 2024-03-18T01:39:15.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba08617160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba0862d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba08621370 0x55ba08615630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34c9d75740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01ac460 con 0x55ba0861b810 2024-03-18T01:39:15.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34c9d75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01adfa0 con 0x55ba0861c200 2024-03-18T01:39:15.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34c9d75740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01acb30 con 0x55ba08621370 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba08617160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bf400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba0862d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba08617160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:35966/0 (socket says 172.21.15.7:35966) 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bfe00640 1 -- 172.21.15.7:0/1249780310 learned_addr learned my addr 172.21.15.7:0/1249780310 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34c5000640 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba08621370 0x55ba08615630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bf400640 1 -- 172.21.15.7:0/1249780310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba08621370 msgr2=0x55ba08615630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bf400640 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba08621370 0x55ba08615630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bf400640 1 -- 172.21.15.7:0/1249780310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 msgr2=0x55ba08617160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:15.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bf400640 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba08617160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bf400640 1 -- 172.21.15.7:0/1249780310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba08625980 con 0x55ba0861c200 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.800+0000 7f34bfe00640 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba08617160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.801+0000 7f34bf400640 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba0862d2e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009430 tx=0x7f34ac005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=360ebe1c2b8c447d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.801+0000 7f34b4c00640 1 -- 172.21.15.7:0/1249780310 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac004400 con 0x55ba0861c200 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.801+0000 7f34b4c00640 1 -- 172.21.15.7:0/1249780310 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac053030 con 0x55ba0861c200 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.801+0000 7f34b4c00640 1 -- 172.21.15.7:0/1249780310 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac04f600 con 0x55ba0861c200 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 -- 172.21.15.7:0/1249780310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 msgr2=0x55ba0862d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba0862d2e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009430 tx=0x7f34ac005fd0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 -- 172.21.15.7:0/1249780310 shutdown_connections 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba08621370 0x55ba08615630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba0861c200 0x55ba0862d2e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 --2- 172.21.15.7:0/1249780310 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba08617160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 -- 172.21.15.7:0/1249780310 >> 172.21.15.7:0/1249780310 conn(0x55ba08616d80 msgr2=0x55ba08625d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:15.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 -- 172.21.15.7:0/1249780310 shutdown_connections 2024-03-18T01:39:15.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 -- 172.21.15.7:0/1249780310 wait complete. 2024-03-18T01:39:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.802+0000 7f34c9d75740 1 -- start start 2024-03-18T01:39:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c9d75740 1 Processor -- start 2024-03-18T01:39:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba0862a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba0862ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c9d75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba08621370 0x55ba0862be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c9d75740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01ad8d0 con 0x55ba0861b810 2024-03-18T01:39:15.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c9d75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01ad200 con 0x55ba08621370 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c9d75740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34c01aa440 con 0x55ba0861c200 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c9d75740 10 client.0 _ll_register_callbacks cb 0x7f34b0009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba0862ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34bf400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba0862ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:60998/0 (socket says 172.21.15.7:60998) 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34bf400640 1 -- 172.21.15.7:0/3170713812 learned_addr learned my addr 172.21.15.7:0/3170713812 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34c5000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba08621370 0x55ba0862be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.803+0000 7f34bfe00640 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba0862a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34bf400640 1 -- 172.21.15.7:0/3170713812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba08621370 msgr2=0x55ba0862be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34bf400640 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba08621370 0x55ba0862be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34bf400640 1 -- 172.21.15.7:0/3170713812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 msgr2=0x55ba0862a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34bf400640 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba0862a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34bf400640 1 -- 172.21.15.7:0/3170713812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba0862c3f0 con 0x55ba0861c200 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34c5000640 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba08621370 0x55ba0862be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:15.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34bf400640 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba0862ae40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f34ac02ca10 tx=0x7f34ac02c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43f0c4b6d8438c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34aa000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34aa000640 1 -- 172.21.15.7:0/3170713812 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac05b070 con 0x55ba0861c200 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34aa000640 1 -- 172.21.15.7:0/3170713812 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f34ac004400 con 0x55ba0861c200 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.804+0000 7f34c9d75740 1 -- 172.21.15.7:0/3170713812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34ac053030 con 0x55ba0861c200 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34aa000640 1 -- 172.21.15.7:0/3170713812 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f34ac064050 con 0x55ba0861c200 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34c9d75740 10 client.25762 fetch_fsmap learned FSMap version 28 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34c9d75740 1 -- 172.21.15.7:0/3170713812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ba0862d700 con 0x55ba0861c200 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34aa000640 1 -- 172.21.15.7:0/3170713812 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34ac053030 con 0x55ba0861c200 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34aa000640 1 -- 172.21.15.7:0/3170713812 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f34ac05e020 con 0x55ba0861c200 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34c9d75740 10 client.25762 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34c9d75740 10 client.25762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34c9d75740 20 client.25762 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34c9d75740 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55ba086983f0 0x55ba0869a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:15.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34c9d75740 4 client.25762 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34c9d75740 1 -- 172.21.15.7:0/3170713812 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ba0858d730 con 0x55ba086983f0 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.805+0000 7f34bfe00640 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55ba086983f0 0x55ba0869a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.806+0000 7f34bfe00640 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55ba086983f0 0x55ba0869a7e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5d59ff1c09b879a server_cookie=1edf8b98d10e068c in_seq=0 out_seq=0 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.806+0000 7f34aa000640 10 client.25762 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.806+0000 7f34aa000640 1 -- 172.21.15.7:0/3170713812 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ba0858d730 con 0x55ba086983f0 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.806+0000 7f34aa000640 10 client.25762 handle_command_reply: tid=0 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.806+0000 7f34c9d75740 1 client.25762 shutdown 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 -- 172.21.15.7:0/3170713812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 msgr2=0x55ba0862ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba0862ae40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f34ac02ca10 tx=0x7f34ac02c330 comp rx=0 tx=0).stop 2024-03-18T01:39:15.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 -- 172.21.15.7:0/3170713812 shutdown_connections 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55ba086983f0 0x55ba0869a7e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ba08621370 0x55ba0862be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ba0861c200 0x55ba0862ae40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34aa000640 0 client.25762 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 --2- 172.21.15.7:0/3170713812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55ba0861b810 0x55ba0862a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 -- 172.21.15.7:0/3170713812 >> 172.21.15.7:0/3170713812 conn(0x55ba08616d80 msgr2=0x55ba08619fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 -- 172.21.15.7:0/3170713812 shutdown_connections 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 1 -- 172.21.15.7:0/3170713812 wait complete. 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.807+0000 7f34c9d75740 20 client.25762 trim_cache size 0 max 16384 2024-03-18T01:39:15.811 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:39:15.812 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-03-18T01:39:15.812 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-03-18T01:39:15.812 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 -- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f34a40a2880 msgr2=0x7f34a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f34a40a2880 0x7f34a40a4d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x55ba0861c600 tx=0x7f34b0005f90 comp rx=0 tx=0).stop 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 -- 172.21.15.7:0/294982777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34c01013d0 msgr2=0x7f34c01a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34c01013d0 0x7f34c01a12a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009fd0 tx=0x7f34b800ef10 comp rx=0 tx=0).stop 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 -- 172.21.15.7:0/294982777 shutdown_connections 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34c01013d0 0x7f34c01a12a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34c0100a30 0x7f34c019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f34a40a2880 0x7f34a40a4d40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 --2- 172.21.15.7:0/294982777 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f34c01000d0 0x7f34c0199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 -- 172.21.15.7:0/294982777 >> 172.21.15.7:0/294982777 conn(0x7f34c00fbc70 msgr2=0x7f34c00fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 -- 172.21.15.7:0/294982777 shutdown_connections 2024-03-18T01:39:15.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:15.808+0000 7f34c7000640 1 -- 172.21.15.7:0/294982777 wait complete. 2024-03-18T01:39:15.823 INFO:tasks.workunit.client.0.smithi007.stdout:/bin/sh ./config.status --recheck 2024-03-18T01:39:15.830 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T01:39:15.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:39:15.981 INFO:tasks.workunit.client.0.smithi007.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-03-18T01:39:16.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 -- 172.21.15.7:0/1828870740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c38074240 msgr2=0x7f4c380746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/1828870740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c38074240 0x7f4c380746c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4c24009860 tx=0x7f4c2402bb70 comp rx=0 tx=0).stop 2024-03-18T01:39:16.059 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 -- 172.21.15.7:0/1828870740 shutdown_connections 2024-03-18T01:39:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/1828870740 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c3810ddd0 0x7f4c381101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/1828870740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c38074240 0x7f4c380746c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/1828870740 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c38075820 0x7f4c38073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 -- 172.21.15.7:0/1828870740 >> 172.21.15.7:0/1828870740 conn(0x7f4c380fcd70 msgr2=0x7f4c380ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 -- 172.21.15.7:0/1828870740 shutdown_connections 2024-03-18T01:39:16.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.059+0000 7f4c3fe00640 1 -- 172.21.15.7:0/1828870740 wait complete. 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.060+0000 7f4c3fe00640 1 Processor -- start 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.060+0000 7f4c3fe00640 1 -- start start 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.060+0000 7f4c3fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c38074240 0x7f4c3819e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.060+0000 7f4c3fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c38075820 0x7f4c3819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.060+0000 7f4c3fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c3810ddd0 0x7f4c381a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3fe00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4c38113a50 con 0x7f4c38075820 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3fe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c381138d0 con 0x7f4c38074240 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3fe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4c38113bd0 con 0x7f4c3810ddd0 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3ca00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c38075820 0x7f4c3819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3ca00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c38075820 0x7f4c3819e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46048/0 (socket says 172.21.15.7:46048) 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3ca00640 1 -- 172.21.15.7:0/2613781612 learned_addr learned my addr 172.21.15.7:0/2613781612 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3de00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c3810ddd0 0x7f4c381a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3d400640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c38074240 0x7f4c3819e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3ca00640 1 -- 172.21.15.7:0/2613781612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c3810ddd0 msgr2=0x7f4c381a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3ca00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c3810ddd0 0x7f4c381a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3ca00640 1 -- 172.21.15.7:0/2613781612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c38074240 msgr2=0x7f4c3819e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3ca00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c38074240 0x7f4c3819e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3ca00640 1 -- 172.21.15.7:0/2613781612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c381a5ce0 con 0x7f4c38075820 2024-03-18T01:39:16.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.061+0000 7f4c3d400640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c38074240 0x7f4c3819e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:16.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.062+0000 7f4c3ca00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c38075820 0x7f4c3819e600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4c24005940 tx=0x7f4c24034930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.062+0000 7f4c36000640 1 -- 172.21.15.7:0/2613781612 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24041020 con 0x7f4c38075820 2024-03-18T01:39:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.062+0000 7f4c36000640 1 -- 172.21.15.7:0/2613781612 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c2402c9f0 con 0x7f4c38075820 2024-03-18T01:39:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.062+0000 7f4c36000640 1 -- 172.21.15.7:0/2613781612 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24039040 con 0x7f4c38075820 2024-03-18T01:39:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.062+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c381a5f70 con 0x7f4c38075820 2024-03-18T01:39:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.062+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c381a6380 con 0x7f4c38075820 2024-03-18T01:39:16.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.064+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c38103fa0 con 0x7f4c38075820 2024-03-18T01:39:16.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.066+0000 7f4c36000640 1 -- 172.21.15.7:0/2613781612 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f4c24044020 con 0x7f4c38075820 2024-03-18T01:39:16.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.068+0000 7f4c36000640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c200a2800 0x7f4c200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.069+0000 7f4c36000640 1 -- 172.21.15.7:0/2613781612 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4c240f0ba0 con 0x7f4c38075820 2024-03-18T01:39:16.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.069+0000 7f4c3d400640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c200a2800 0x7f4c200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.070+0000 7f4c3d400640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c200a2800 0x7f4c200a4cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009b50 tx=0x7f4c2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.075 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.075+0000 7f4c36000640 1 -- 172.21.15.7:0/2613781612 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4c2404b020 con 0x7f4c38075820 2024-03-18T01:39:16.226 INFO:tasks.workunit.client.0.smithi007.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-18T01:39:16.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.365+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c3802d1b0 con 0x7f4c38075820 2024-03-18T01:39:16.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.367+0000 7f4c36000640 1 -- 172.21.15.7:0/2613781612 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11919 (secure 0 0 0) 0x7f4c240bd8e0 con 0x7f4c38075820 2024-03-18T01:39:16.367 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:39:16.367 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:39:15.247247+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24523,24544,24542,24562,24596]},"id":1}]} 2024-03-18T01:39:16.368 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 29 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.370+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c200a2800 msgr2=0x7f4c200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.370+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c200a2800 0x7f4c200a4cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c009b50 tx=0x7f4c2c005f90 comp rx=0 tx=0).stop 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.370+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c38075820 msgr2=0x7f4c3819e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.370+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c38075820 0x7f4c3819e600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4c24005940 tx=0x7f4c24034930 comp rx=0 tx=0).stop 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.371+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 shutdown_connections 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.371+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c3810ddd0 0x7f4c381a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.371+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c200a2800 0x7f4c200a4cc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.371+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c38075820 0x7f4c3819e600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.371+0000 7f4c3fe00640 1 --2- 172.21.15.7:0/2613781612 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c38074240 0x7f4c3819e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.371+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 >> 172.21.15.7:0/2613781612 conn(0x7f4c380fcd70 msgr2=0x7f4c38101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.371+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 shutdown_connections 2024-03-18T01:39:16.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.371+0000 7f4c3fe00640 1 -- 172.21.15.7:0/2613781612 wait complete. 2024-03-18T01:39:16.391 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T01:39:16.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 -- 172.21.15.7:0/3953364965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc102130 msgr2=0x7f21fc104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 --2- 172.21.15.7:0/3953364965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc102130 0x7f21fc104520 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009900 tx=0x7f21ec02bc30 comp rx=0 tx=0).stop 2024-03-18T01:39:16.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 -- 172.21.15.7:0/3953364965 shutdown_connections 2024-03-18T01:39:16.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 --2- 172.21.15.7:0/3953364965 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21fc107390 0x7f21fc109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 --2- 172.21.15.7:0/3953364965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21fc104a60 0x7f21fc106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.604 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 --2- 172.21.15.7:0/3953364965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc102130 0x7f21fc104520 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 -- 172.21.15.7:0/3953364965 >> 172.21.15.7:0/3953364965 conn(0x7f21fc0fbaa0 msgr2=0x7f21fc0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 -- 172.21.15.7:0/3953364965 shutdown_connections 2024-03-18T01:39:16.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.603+0000 7f2201200640 1 -- 172.21.15.7:0/3953364965 wait complete. 2024-03-18T01:39:16.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.604+0000 7f2201200640 1 Processor -- start 2024-03-18T01:39:16.605 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.604+0000 7f2201200640 1 -- start start 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.604+0000 7f2201200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21fc102130 0x7f21fc199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f2201200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21fc104a60 0x7f21fc19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f2201200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc107390 0x7f21fc1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f2201200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc10d270 con 0x7f21fc104a60 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f2201200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc10d0f0 con 0x7f21fc107390 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f2201200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc10d3f0 con 0x7f21fc102130 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21fc102130 0x7f21fc199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fb400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21fc102130 0x7f21fc199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38412/0 (socket says 172.21.15.7:38412) 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fb400640 1 -- 172.21.15.7:0/141986021 learned_addr learned my addr 172.21.15.7:0/141986021 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fbe00640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc107390 0x7f21fc1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21f3e00640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21fc104a60 0x7f21fc19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.607 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fb400640 1 -- 172.21.15.7:0/141986021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc107390 msgr2=0x7f21fc1a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fb400640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc107390 0x7f21fc1a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fb400640 1 -- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21fc104a60 msgr2=0x7f21fc19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fb400640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21fc104a60 0x7f21fc19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fb400640 1 -- 172.21.15.7:0/141986021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21fc1a1720 con 0x7f21fc102130 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.605+0000 7f21fbe00640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc107390 0x7f21fc1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.606+0000 7f21fb400640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21fc102130 0x7f21fc199b00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009230 tx=0x7f21ec02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.606+0000 7f21f8c00640 1 -- 172.21.15.7:0/141986021 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec041070 con 0x7f21fc102130 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.606+0000 7f21f8c00640 1 -- 172.21.15.7:0/141986021 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21ec004450 con 0x7f21fc102130 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.606+0000 7f2201200640 1 -- 172.21.15.7:0/141986021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21fc1a19b0 con 0x7f21fc102130 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.606+0000 7f21f8c00640 1 -- 172.21.15.7:0/141986021 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec003e20 con 0x7f21fc102130 2024-03-18T01:39:16.608 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.606+0000 7f2201200640 1 -- 172.21.15.7:0/141986021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21fc1a1ef0 con 0x7f21fc102130 2024-03-18T01:39:16.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.609+0000 7f220542c740 1 Processor -- start 2024-03-18T01:39:16.609 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.609+0000 7f220542c740 1 -- start start 2024-03-18T01:39:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f220542c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f223beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f220542c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f2240f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f220542c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2243a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f220542c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5625f22033b0 con 0x5625f223ba90 2024-03-18T01:39:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f220542c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5625f21a0450 con 0x5625f223c480 2024-03-18T01:39:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f220542c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5625f2148cb0 con 0x5625f22415f0 2024-03-18T01:39:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21fbe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f223beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.611 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2243a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21fbe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f223beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46068/0 (socket says 172.21.15.7:46068) 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21fbe00640 1 -- 172.21.15.7:0/3385333204 learned_addr learned my addr 172.21.15.7:0/3385333204 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21f3e00640 1 -- 172.21.15.7:0/3385333204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 msgr2=0x5625f2240f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21f3e00640 1 --2- 172.21.15.7:0/3385333204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f2240f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21f3e00640 1 -- 172.21.15.7:0/3385333204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 msgr2=0x5625f223beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21f3e00640 1 --2- 172.21.15.7:0/3385333204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f223beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21f3e00640 1 -- 172.21.15.7:0/3385333204 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625f216a1f0 con 0x5625f22415f0 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.610+0000 7f21fbe00640 1 --2- 172.21.15.7:0/3385333204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f223beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.611+0000 7f21f3e00640 1 --2- 172.21.15.7:0/3385333204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2243a80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f21fc069550 tx=0x7f21e4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38e7c40e8c52d55c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.611+0000 7f21f0c00640 1 -- 172.21.15.7:0/3385333204 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4009d70 con 0x5625f22415f0 2024-03-18T01:39:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.611+0000 7f21f0c00640 1 -- 172.21.15.7:0/3385333204 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21e4005020 con 0x5625f22415f0 2024-03-18T01:39:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.611+0000 7f21f0c00640 1 -- 172.21.15.7:0/3385333204 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4004b60 con 0x5625f22415f0 2024-03-18T01:39:16.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.611+0000 7f21f8c00640 1 -- 172.21.15.7:0/141986021 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f21ec044070 con 0x7f21fc102130 2024-03-18T01:39:16.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 -- 172.21.15.7:0/3385333204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 msgr2=0x5625f2243a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 --2- 172.21.15.7:0/3385333204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2243a80 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f21fc069550 tx=0x7f21e4011810 comp rx=0 tx=0).stop 2024-03-18T01:39:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 -- 172.21.15.7:0/3385333204 shutdown_connections 2024-03-18T01:39:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 --2- 172.21.15.7:0/3385333204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2243a80 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 --2- 172.21.15.7:0/3385333204 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f2240f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 --2- 172.21.15.7:0/3385333204 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f223beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 -- 172.21.15.7:0/3385333204 >> 172.21.15.7:0/3385333204 conn(0x5625f2237000 msgr2=0x5625f2237400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 -- 172.21.15.7:0/3385333204 shutdown_connections 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 -- 172.21.15.7:0/3385333204 wait complete. 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.612+0000 7f220542c740 1 -- start start 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.613+0000 7f220542c740 1 Processor -- start 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f220542c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f223ba90 0x5625f2245d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f220542c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f22462c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21f8c00640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f21e00a2780 0x7f21e00a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f220542c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f22415f0 0x5625f224d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f220542c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5625f215c110 con 0x5625f22415f0 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f220542c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5625f219c4f0 con 0x5625f223c480 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f220542c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5625f22445a0 con 0x5625f223ba90 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f220542c740 10 client.0 _ll_register_callbacks cb 0x5625f2149b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f22462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21f8c00640 1 -- 172.21.15.7:0/141986021 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f21ec0f08c0 con 0x7f21fc102130 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f223ba90 0x5625f2245d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f22462c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41452/0 (socket says 172.21.15.7:41452) 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fb400640 1 -- 172.21.15.7:0/2794137309 learned_addr learned my addr 172.21.15.7:0/2794137309 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fbe00640 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f22415f0 0x5625f224d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21f3e00640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f21e00a2780 0x7f21e00a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fb400640 1 -- 172.21.15.7:0/2794137309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f223ba90 msgr2=0x5625f2245d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fb400640 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f223ba90 0x5625f2245d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fb400640 1 -- 172.21.15.7:0/2794137309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f22415f0 msgr2=0x5625f224d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fb400640 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f22415f0 0x5625f224d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.614+0000 7f21fb400640 1 -- 172.21.15.7:0/2794137309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625f1eb43f0 con 0x5625f223c480 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f21fb400640 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f22462c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f21ec000c00 tx=0x7f21ec0b9310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d0f95c3e024f61e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f21e9600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f21e9600640 1 -- 172.21.15.7:0/2794137309 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec04b040 con 0x5625f223c480 2024-03-18T01:39:16.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f21e9600640 1 -- 172.21.15.7:0/2794137309 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f21ec0c5210 con 0x5625f223c480 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f220542c740 1 -- 172.21.15.7:0/2794137309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21e4005020 con 0x5625f223c480 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f21f3e00640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f21e00a2780 0x7f21e00a4c40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f21e40099b0 tx=0x7f21e4009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f21e9600640 1 -- 172.21.15.7:0/2794137309 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f21ec004c60 con 0x5625f223c480 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f220542c740 10 client.25799 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.615+0000 7f220542c740 1 -- 172.21.15.7:0/2794137309 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625f1f7e7d0 con 0x5625f223c480 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f21e9600640 1 -- 172.21.15.7:0/2794137309 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21e4005020 con 0x5625f223c480 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f21e9600640 1 -- 172.21.15.7:0/2794137309 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f21ec0bc400 con 0x5625f223c480 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f220542c740 10 client.25799 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f220542c740 10 client.25799 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f220542c740 20 client.25799 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:16.621 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f220542c740 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5625f2251b60 0x5625f2253f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f220542c740 4 client.25799 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f220542c740 1 -- 172.21.15.7:0/2794137309 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625f21ad990 con 0x5625f2251b60 2024-03-18T01:39:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.616+0000 7f21fbe00640 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5625f2251b60 0x5625f2253f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.617+0000 7f21fbe00640 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5625f2251b60 0x5625f2253f90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8b41ea4e7bf64094 server_cookie=fb2c14f129797616 in_seq=0 out_seq=0 2024-03-18T01:39:16.622 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.617+0000 7f21e9600640 10 client.25799 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:39:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.617+0000 7f21e9600640 1 -- 172.21.15.7:0/2794137309 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5625f21ad990 con 0x5625f2251b60 2024-03-18T01:39:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.617+0000 7f21e9600640 10 client.25799 handle_command_reply: tid=0 2024-03-18T01:39:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 client.25799 shutdown 2024-03-18T01:39:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 -- 172.21.15.7:0/2794137309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 msgr2=0x5625f22462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f22462c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f21ec000c00 tx=0x7f21ec0b9310 comp rx=0 tx=0).stop 2024-03-18T01:39:16.623 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 -- 172.21.15.7:0/2794137309 shutdown_connections 2024-03-18T01:39:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5625f2251b60 0x5625f2253f90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f22415f0 0x5625f224d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f21e9600640 0 client.25799 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:39:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f22462c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 --2- 172.21.15.7:0/2794137309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f223ba90 0x5625f2245d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 -- 172.21.15.7:0/2794137309 >> 172.21.15.7:0/2794137309 conn(0x5625f2237000 msgr2=0x5625f2236220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 -- 172.21.15.7:0/2794137309 shutdown_connections 2024-03-18T01:39:16.624 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 1 -- 172.21.15.7:0/2794137309 wait complete. 2024-03-18T01:39:16.625 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.618+0000 7f220542c740 20 client.25799 trim_cache size 0 max 16384 2024-03-18T01:39:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.635+0000 7f220542c740 1 Processor -- start 2024-03-18T01:39:16.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.635+0000 7f220542c740 1 -- start start 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.635+0000 7f220542c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f2258860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.635+0000 7f220542c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f224d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.635+0000 7f220542c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2258da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.635+0000 7f220542c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1ac2c0 con 0x5625f223ba90 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.635+0000 7f220542c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1ade30 con 0x5625f223c480 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.635+0000 7f220542c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1ac990 con 0x5625f22415f0 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2258da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f224d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2258da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38456/0 (socket says 172.21.15.7:38456) 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 -- 172.21.15.7:0/97997427 learned_addr learned my addr 172.21.15.7:0/97997427 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fb400640 1 --2- 172.21.15.7:0/97997427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f2258860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 -- 172.21.15.7:0/97997427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 msgr2=0x5625f224d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 --2- 172.21.15.7:0/97997427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f224d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 -- 172.21.15.7:0/97997427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 msgr2=0x5625f2258860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 --2- 172.21.15.7:0/97997427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f2258860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 -- 172.21.15.7:0/97997427 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625f22545e0 con 0x5625f22415f0 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21fbe00640 1 --2- 172.21.15.7:0/97997427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2258da0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5625f223cce0 tx=0x7f21f4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172c46ea5b19694 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21ebe00640 1 -- 172.21.15.7:0/97997427 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f40111b0 con 0x5625f22415f0 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.636+0000 7f21ebe00640 1 -- 172.21.15.7:0/97997427 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21f4022170 con 0x5625f22415f0 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f21ebe00640 1 -- 172.21.15.7:0/97997427 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f40053a0 con 0x5625f22415f0 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 -- 172.21.15.7:0/97997427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 msgr2=0x5625f2258da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 --2- 172.21.15.7:0/97997427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2258da0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x5625f223cce0 tx=0x7f21f4004a40 comp rx=0 tx=0).stop 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 -- 172.21.15.7:0/97997427 shutdown_connections 2024-03-18T01:39:16.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 --2- 172.21.15.7:0/97997427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f2258da0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 --2- 172.21.15.7:0/97997427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223c480 0x5625f224d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 --2- 172.21.15.7:0/97997427 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223ba90 0x5625f2258860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 -- 172.21.15.7:0/97997427 >> 172.21.15.7:0/97997427 conn(0x5625f2237000 msgr2=0x5625f223a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 -- 172.21.15.7:0/97997427 shutdown_connections 2024-03-18T01:39:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.637+0000 7f220542c740 1 -- 172.21.15.7:0/97997427 wait complete. 2024-03-18T01:39:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 1 -- start start 2024-03-18T01:39:16.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 1 Processor -- start 2024-03-18T01:39:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223ba90 0x5625f224cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223c480 0x5625f224b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f224baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1ad730 con 0x5625f223c480 2024-03-18T01:39:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1ad060 con 0x5625f223ba90 2024-03-18T01:39:16.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc1aa2a0 con 0x5625f22415f0 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f220542c740 10 client.0 _ll_register_callbacks cb 0x7f21ec0b8b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.638+0000 7f21fb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223ba90 0x5625f224cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21f3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223c480 0x5625f224b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f224baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223ba90 0x5625f224cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41470/0 (socket says 172.21.15.7:41470) 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fb400640 1 -- 172.21.15.7:0/1420314744 learned_addr learned my addr 172.21.15.7:0/1420314744 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fb400640 1 -- 172.21.15.7:0/1420314744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 msgr2=0x5625f224baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fb400640 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f224baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fb400640 1 -- 172.21.15.7:0/1420314744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223c480 msgr2=0x5625f224b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fb400640 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223c480 0x5625f224b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fb400640 1 -- 172.21.15.7:0/1420314744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625f224d6d0 con 0x5625f223ba90 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fbe00640 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f224baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:16.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21fb400640 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223ba90 0x5625f224cf50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f21ec006010 tx=0x7f21ec0bc630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7d674d761cbdd26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21e9600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21e9600640 1 -- 172.21.15.7:0/1420314744 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec04b040 con 0x5625f223ba90 2024-03-18T01:39:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.639+0000 7f21e9600640 1 -- 172.21.15.7:0/1420314744 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f21ec0c5410 con 0x5625f223ba90 2024-03-18T01:39:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 1 -- 172.21.15.7:0/1420314744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21f4022170 con 0x5625f223ba90 2024-03-18T01:39:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f21e9600640 1 -- 172.21.15.7:0/1420314744 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f21ec0bcfb0 con 0x5625f223ba90 2024-03-18T01:39:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 10 client.25805 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:16.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 1 -- 172.21.15.7:0/1420314744 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625f2246630 con 0x5625f223ba90 2024-03-18T01:39:16.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f21e9600640 1 -- 172.21.15.7:0/1420314744 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21f4022170 con 0x5625f223ba90 2024-03-18T01:39:16.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f21e9600640 1 -- 172.21.15.7:0/1420314744 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f21ec0bc0a0 con 0x5625f223ba90 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 10 client.25805 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 10 client.25805 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 20 client.25805 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5625f22b9280 0x5625f22bb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 4 client.25805 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.640+0000 7f220542c740 1 -- 172.21.15.7:0/1420314744 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625f21a7ce0 con 0x5625f22b9280 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.641+0000 7f21f3e00640 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5625f22b9280 0x5625f22bb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.641+0000 7f21f3e00640 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5625f22b9280 0x5625f22bb6a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cad61ede33b6774 server_cookie=20ce354b80e1af46 in_seq=0 out_seq=0 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.641+0000 7f21e9600640 10 client.25805 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.641+0000 7f21e9600640 1 -- 172.21.15.7:0/1420314744 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625f21a7ce0 con 0x5625f22b9280 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.641+0000 7f21e9600640 10 client.25805 handle_command_reply: tid=0 2024-03-18T01:39:16.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.641+0000 7f220542c740 1 client.25805 shutdown 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.641+0000 7f220542c740 1 -- 172.21.15.7:0/1420314744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223ba90 msgr2=0x5625f224cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.641+0000 7f220542c740 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223ba90 0x5625f224cf50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f21ec006010 tx=0x7f21ec0bc630 comp rx=0 tx=0).stop 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 1 -- 172.21.15.7:0/1420314744 shutdown_connections 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5625f22b9280 0x5625f22bb6a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5625f22415f0 0x5625f224baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5625f223c480 0x5625f224b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 1 --2- 172.21.15.7:0/1420314744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5625f223ba90 0x5625f224cf50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f21e9600640 0 client.25805 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 1 -- 172.21.15.7:0/1420314744 >> 172.21.15.7:0/1420314744 conn(0x5625f2237000 msgr2=0x5625f2235990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 1 -- 172.21.15.7:0/1420314744 shutdown_connections 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 1 -- 172.21.15.7:0/1420314744 wait complete. 2024-03-18T01:39:16.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f220542c740 20 client.25805 trim_cache size 0 max 16384 2024-03-18T01:39:16.648 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:16.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f2201200640 1 -- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f21e00a2780 msgr2=0x7f21e00a4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f2201200640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f21e00a2780 0x7f21e00a4c40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f21e40099b0 tx=0x7f21e4009680 comp rx=0 tx=0).stop 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f2201200640 1 -- 172.21.15.7:0/141986021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21fc102130 msgr2=0x7f21fc199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.642+0000 7f2201200640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21fc102130 0x7f21fc199b00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009230 tx=0x7f21ec02c650 comp rx=0 tx=0).stop 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.643+0000 7f2201200640 1 -- 172.21.15.7:0/141986021 shutdown_connections 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.643+0000 7f2201200640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21fc107390 0x7f21fc1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.643+0000 7f2201200640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f21e00a2780 0x7f21e00a4c40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.643+0000 7f2201200640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f21fc104a60 0x7f21fc19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.643+0000 7f2201200640 1 --2- 172.21.15.7:0/141986021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21fc102130 0x7f21fc199b00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.643+0000 7f2201200640 1 -- 172.21.15.7:0/141986021 >> 172.21.15.7:0/141986021 conn(0x7f21fc0fbaa0 msgr2=0x7f21fc1018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.643+0000 7f2201200640 1 -- 172.21.15.7:0/141986021 shutdown_connections 2024-03-18T01:39:16.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.643+0000 7f2201200640 1 -- 172.21.15.7:0/141986021 wait complete. 2024-03-18T01:39:16.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T01:39:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.873+0000 7f4c4c000640 1 -- 172.21.15.7:0/1055412396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c441055b0 msgr2=0x7f4c4410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.873+0000 7f4c4c000640 1 --2- 172.21.15.7:0/1055412396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c441055b0 0x7f4c4410be40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c009900 tx=0x7f4c3c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:39:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.873+0000 7f4c4c000640 1 -- 172.21.15.7:0/1055412396 shutdown_connections 2024-03-18T01:39:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.873+0000 7f4c4c000640 1 --2- 172.21.15.7:0/1055412396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c441055b0 0x7f4c4410be40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.873+0000 7f4c4c000640 1 --2- 172.21.15.7:0/1055412396 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c44104bf0 0x7f4c44105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.873+0000 7f4c4c000640 1 --2- 172.21.15.7:0/1055412396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c44103a00 0x7f4c44103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.873+0000 7f4c4c000640 1 -- 172.21.15.7:0/1055412396 >> 172.21.15.7:0/1055412396 conn(0x7f4c440fef30 msgr2=0x7f4c441013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.873+0000 7f4c4c000640 1 -- 172.21.15.7:0/1055412396 shutdown_connections 2024-03-18T01:39:16.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.874+0000 7f4c4c000640 1 -- 172.21.15.7:0/1055412396 wait complete. 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.874+0000 7f4c4c000640 1 Processor -- start 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c4c000640 1 -- start start 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c4c000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c44103a00 0x7f4c44199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c4c000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c44104bf0 0x7f4c4419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c4c000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c441055b0 0x7f4c441a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c4c000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410f6b0 con 0x7f4c44103a00 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c4c000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410f530 con 0x7f4c441055b0 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c4c000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410f830 con 0x7f4c44104bf0 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c48c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c44104bf0 0x7f4c4419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c48c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c44104bf0 0x7f4c4419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38488/0 (socket says 172.21.15.7:38488) 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c4a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c441055b0 0x7f4c441a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c48c00640 1 -- 172.21.15.7:0/3710345148 learned_addr learned my addr 172.21.15.7:0/3710345148 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.875+0000 7f4c49600640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c44103a00 0x7f4c44199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c48c00640 1 -- 172.21.15.7:0/3710345148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c441055b0 msgr2=0x7f4c441a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c48c00640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c441055b0 0x7f4c441a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c48c00640 1 -- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c44103a00 msgr2=0x7f4c44199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c48c00640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c44103a00 0x7f4c44199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c48c00640 1 -- 172.21.15.7:0/3710345148 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c441a18d0 con 0x7f4c44104bf0 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c48c00640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c44104bf0 0x7f4c4419a1d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000e9f0 tx=0x7f4c3000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c42000640 1 -- 172.21.15.7:0/3710345148 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3000cd80 con 0x7f4c44104bf0 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c42000640 1 -- 172.21.15.7:0/3710345148 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c30010910 con 0x7f4c44104bf0 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c42000640 1 -- 172.21.15.7:0/3710345148 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c30018990 con 0x7f4c44104bf0 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.876+0000 7f4c4c000640 1 -- 172.21.15.7:0/3710345148 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c441a1bc0 con 0x7f4c44104bf0 2024-03-18T01:39:16.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.877+0000 7f4c4c000640 1 -- 172.21.15.7:0/3710345148 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c441a20c0 con 0x7f4c44104bf0 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4edd7740 1 Processor -- start 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4edd7740 1 -- start start 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c42000640 1 -- 172.21.15.7:0/3710345148 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f4c30010ac0 con 0x7f4c44104bf0 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b385552db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b3855558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b3855153b0 con 0x55b38554d8c0 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b3854b2450 con 0x55b385553420 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b38545acb0 con 0x55b38554e2b0 2024-03-18T01:39:16.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c4a000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.880+0000 7f4c49600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b385552db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c4a000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46120/0 (socket says 172.21.15.7:46120) 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c4a000640 1 -- 172.21.15.7:0/1154452196 learned_addr learned my addr 172.21.15.7:0/1154452196 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c48c00640 1 --2- 172.21.15.7:0/1154452196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b3855558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c49600640 1 -- 172.21.15.7:0/1154452196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 msgr2=0x55b3855558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c49600640 1 --2- 172.21.15.7:0/1154452196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b3855558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c49600640 1 -- 172.21.15.7:0/1154452196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 msgr2=0x55b38554dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c49600640 1 --2- 172.21.15.7:0/1154452196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c49600640 1 -- 172.21.15.7:0/1154452196 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b38547c1f0 con 0x55b38554e2b0 2024-03-18T01:39:16.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c49600640 1 --2- 172.21.15.7:0/1154452196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b385552db0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c44104a50 tx=0x7f4c380118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=958eca6a4cdd673d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.881+0000 7f4c36600640 1 -- 172.21.15.7:0/1154452196 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38004d20 con 0x55b38554e2b0 2024-03-18T01:39:16.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c36600640 1 -- 172.21.15.7:0/1154452196 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c38003e50 con 0x55b38554e2b0 2024-03-18T01:39:16.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c36600640 1 -- 172.21.15.7:0/1154452196 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3800f740 con 0x55b38554e2b0 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c42000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c42000640 1 -- 172.21.15.7:0/3710345148 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4c300cf0b0 con 0x7f4c44104bf0 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4a000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4edd7740 1 -- 172.21.15.7:0/1154452196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 msgr2=0x55b385552db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/1154452196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b385552db0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4c44104a50 tx=0x7f4c380118d0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4edd7740 1 -- 172.21.15.7:0/1154452196 shutdown_connections 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/1154452196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b3855558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/1154452196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b385552db0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/1154452196 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4edd7740 1 -- 172.21.15.7:0/1154452196 >> 172.21.15.7:0/1154452196 conn(0x55b385548e30 msgr2=0x55b385549230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.882+0000 7f4c4edd7740 1 -- 172.21.15.7:0/1154452196 shutdown_connections 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.883+0000 7f4c4edd7740 1 -- 172.21.15.7:0/1154452196 wait complete. 2024-03-18T01:39:16.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.883+0000 7f4c4a000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x55b38554eb10 tx=0x7f4c3c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.883+0000 7f4c4edd7740 1 -- start start 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.883+0000 7f4c4edd7740 1 Processor -- start 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554d8c0 0x55b385557b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554e2b0 0x55b385558080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b38555f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b38546e110 con 0x55b38554e2b0 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b3854ae4f0 con 0x55b385553420 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b3855563d0 con 0x55b38554d8c0 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c4edd7740 10 client.0 _ll_register_callbacks cb 0x55b38545bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c49600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554d8c0 0x55b385557b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c4a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b38555f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c49600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554d8c0 0x55b385557b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38506/0 (socket says 172.21.15.7:38506) 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c49600640 1 -- 172.21.15.7:0/4211839123 learned_addr learned my addr 172.21.15.7:0/4211839123 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c48c00640 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554e2b0 0x55b385558080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c49600640 1 -- 172.21.15.7:0/4211839123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 msgr2=0x55b38555f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c49600640 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b38555f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c49600640 1 -- 172.21.15.7:0/4211839123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554e2b0 msgr2=0x55b385558080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c49600640 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554e2b0 0x55b385558080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c49600640 1 -- 172.21.15.7:0/4211839123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3851c63f0 con 0x55b38554d8c0 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.884+0000 7f4c48c00640 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554e2b0 0x55b385558080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.885+0000 7f4c49600640 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554d8c0 0x55b385557b40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c38006010 tx=0x7f4c38009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1743108a0ad9bf67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.885+0000 7f4c2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.885+0000 7f4c2aa00640 1 -- 172.21.15.7:0/4211839123 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38027020 con 0x55b38554d8c0 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.885+0000 7f4c4edd7740 1 -- 172.21.15.7:0/4211839123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c38003e50 con 0x55b38554d8c0 2024-03-18T01:39:16.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.885+0000 7f4c2aa00640 1 -- 172.21.15.7:0/4211839123 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4c38004210 con 0x55b38554d8c0 2024-03-18T01:39:16.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c2aa00640 1 -- 172.21.15.7:0/4211839123 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4c38032050 con 0x55b38554d8c0 2024-03-18T01:39:16.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c4edd7740 10 client.25798 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c4edd7740 1 -- 172.21.15.7:0/4211839123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b3852907d0 con 0x55b38554d8c0 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c2aa00640 1 -- 172.21.15.7:0/4211839123 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c38003e50 con 0x55b38554d8c0 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c2aa00640 1 -- 172.21.15.7:0/4211839123 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f4c38005040 con 0x55b38554d8c0 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c4edd7740 10 client.25798 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c4edd7740 10 client.25798 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c4edd7740 20 client.25798 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55b385563750 0x55b385565bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c4edd7740 4 client.25798 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.886+0000 7f4c4edd7740 1 -- 172.21.15.7:0/4211839123 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3854bf990 con 0x55b385563750 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.887+0000 7f4c48c00640 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55b385563750 0x55b385565bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.887+0000 7f4c48c00640 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55b385563750 0x55b385565bd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fe39ea569fac190f server_cookie=b1cdd600ba0522dd in_seq=0 out_seq=0 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.887+0000 7f4c2aa00640 10 client.25798 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c2aa00640 1 -- 172.21.15.7:0/4211839123 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b3854bf990 con 0x55b385563750 2024-03-18T01:39:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c2aa00640 10 client.25798 handle_command_reply: tid=0 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 client.25798 shutdown 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 -- 172.21.15.7:0/4211839123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554d8c0 msgr2=0x55b385557b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554d8c0 0x55b385557b40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c38006010 tx=0x7f4c38009230 comp rx=0 tx=0).stop 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 -- 172.21.15.7:0/4211839123 shutdown_connections 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55b385563750 0x55b385565bd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b38555f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554e2b0 0x55b385558080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c2aa00640 0 client.25798 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/4211839123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554d8c0 0x55b385557b40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.888+0000 7f4c4edd7740 1 -- 172.21.15.7:0/4211839123 >> 172.21.15.7:0/4211839123 conn(0x55b385548e30 msgr2=0x55b3855481f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.889+0000 7f4c4edd7740 1 -- 172.21.15.7:0/4211839123 shutdown_connections 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.889+0000 7f4c4edd7740 1 -- 172.21.15.7:0/4211839123 wait complete. 2024-03-18T01:39:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.889+0000 7f4c4edd7740 20 client.25798 trim_cache size 0 max 16384 2024-03-18T01:39:16.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c4edd7740 1 Processor -- start 2024-03-18T01:39:16.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c4edd7740 1 -- start start 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b38555f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b38556a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441ac5c0 con 0x55b38554d8c0 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441ae130 con 0x55b385553420 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441acc90 con 0x55b38554e2b0 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.904+0000 7f4c49600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c48c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b38555f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c4a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b38556a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c49600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46140/0 (socket says 172.21.15.7:46140) 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c49600640 1 -- 172.21.15.7:0/646919188 learned_addr learned my addr 172.21.15.7:0/646919188 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c48c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b38555f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38514/0 (socket says 172.21.15.7:38514) 2024-03-18T01:39:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c49600640 1 -- 172.21.15.7:0/646919188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 msgr2=0x55b38555f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c49600640 1 --2- 172.21.15.7:0/646919188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b38555f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c49600640 1 -- 172.21.15.7:0/646919188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 msgr2=0x55b38556a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c49600640 1 --2- 172.21.15.7:0/646919188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b38556a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c49600640 1 -- 172.21.15.7:0/646919188 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b385544c10 con 0x55b38554d8c0 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c48c00640 1 --2- 172.21.15.7:0/646919188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b38555f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c49600640 1 --2- 172.21.15.7:0/646919188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554c7a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c38000f80 tx=0x7f4c38005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d1c7b8f1d7bf89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.905+0000 7f4c35c00640 1 -- 172.21.15.7:0/646919188 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38027070 con 0x55b38554d8c0 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c35c00640 1 -- 172.21.15.7:0/646919188 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c38003e50 con 0x55b38554d8c0 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c35c00640 1 -- 172.21.15.7:0/646919188 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38004f60 con 0x55b38554d8c0 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 -- 172.21.15.7:0/646919188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 msgr2=0x55b38554c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/646919188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554c7a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c38000f80 tx=0x7f4c38005cf0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 -- 172.21.15.7:0/646919188 shutdown_connections 2024-03-18T01:39:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/646919188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b385553420 0x55b38556a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/646919188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b38554e2b0 0x55b38555f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/646919188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38554c7a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 -- 172.21.15.7:0/646919188 >> 172.21.15.7:0/646919188 conn(0x55b385548e30 msgr2=0x55b38554c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 -- 172.21.15.7:0/646919188 shutdown_connections 2024-03-18T01:39:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.906+0000 7f4c4edd7740 1 -- 172.21.15.7:0/646919188 wait complete. 2024-03-18T01:39:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.907+0000 7f4c4edd7740 1 -- start start 2024-03-18T01:39:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.907+0000 7f4c4edd7740 1 Processor -- start 2024-03-18T01:39:16.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.907+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38555c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.907+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b38554e2b0 0x55b38555cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c4edd7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b385553420 0x55b38555dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441ada30 con 0x55b38554d8c0 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441ad360 con 0x55b38554e2b0 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c4edd7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4c441aa5a0 con 0x55b385553420 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c4edd7740 10 client.0 _ll_register_callbacks cb 0x7f4c38011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c49600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38555c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c48c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b38554e2b0 0x55b38555cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c4a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b385553420 0x55b38555dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c49600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38555c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46142/0 (socket says 172.21.15.7:46142) 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c49600640 1 -- 172.21.15.7:0/2812488450 learned_addr learned my addr 172.21.15.7:0/2812488450 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c49600640 1 -- 172.21.15.7:0/2812488450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b385553420 msgr2=0x55b38555dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c49600640 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b385553420 0x55b38555dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c49600640 1 -- 172.21.15.7:0/2812488450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b38554e2b0 msgr2=0x55b38555cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c49600640 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b38554e2b0 0x55b38555cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c49600640 1 -- 172.21.15.7:0/2812488450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b38555e0c0 con 0x55b38554d8c0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.908+0000 7f4c4a000640 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b385553420 0x55b38555dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.909+0000 7f4c49600640 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38555c5d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c380054b0 tx=0x7f4c380049a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75501c6f67b7074f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.909+0000 7f4c2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.909+0000 7f4c2aa00640 1 -- 172.21.15.7:0/2812488450 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c38027040 con 0x55b38554d8c0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.909+0000 7f4c2aa00640 1 -- 172.21.15.7:0/2812488450 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4c38004620 con 0x55b38554d8c0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.909+0000 7f4c4edd7740 1 -- 172.21.15.7:0/2812488450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c38003e50 con 0x55b38554d8c0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.909+0000 7f4c2aa00640 1 -- 172.21.15.7:0/2812488450 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4c38047050 con 0x55b38554d8c0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.909+0000 7f4c4edd7740 10 client.15633 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.909+0000 7f4c4edd7740 1 -- 172.21.15.7:0/2812488450 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b38555f3c0 con 0x55b38554d8c0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c2aa00640 1 -- 172.21.15.7:0/2812488450 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c38003e50 con 0x55b38554d8c0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c2aa00640 1 -- 172.21.15.7:0/2812488450 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f4c3800fa10 con 0x55b38554d8c0 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c4edd7740 10 client.15633 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c4edd7740 10 client.15633 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c4edd7740 20 client.15633 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55b3855ca1c0 0x55b3855cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c4edd7740 4 client.15633 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c4edd7740 1 -- 172.21.15.7:0/2812488450 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b3854b9ce0 con 0x55b3855ca1c0 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.910+0000 7f4c48c00640 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55b3855ca1c0 0x55b3855cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c48c00640 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55b3855ca1c0 0x55b3855cc5e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cf7c1fa47e15b85e server_cookie=67324b2090fd0afe in_seq=0 out_seq=0 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c2aa00640 10 client.15633 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c2aa00640 1 -- 172.21.15.7:0/2812488450 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b3854b9ce0 con 0x55b3855ca1c0 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c2aa00640 10 client.15633 handle_command_reply: tid=0 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 client.15633 shutdown 2024-03-18T01:39:16.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 -- 172.21.15.7:0/2812488450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 msgr2=0x55b38555c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38555c5d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c380054b0 tx=0x7f4c380049a0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 -- 172.21.15.7:0/2812488450 shutdown_connections 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55b3855ca1c0 0x55b3855cc5e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b385553420 0x55b38555dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b38554e2b0 0x55b38555cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c2aa00640 0 client.15633 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 --2- 172.21.15.7:0/2812488450 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b38554d8c0 0x55b38555c5d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 -- 172.21.15.7:0/2812488450 >> 172.21.15.7:0/2812488450 conn(0x55b385548e30 msgr2=0x55b38554c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.911+0000 7f4c4edd7740 1 -- 172.21.15.7:0/2812488450 shutdown_connections 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4edd7740 1 -- 172.21.15.7:0/2812488450 wait complete. 2024-03-18T01:39:16.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4edd7740 20 client.15633 trim_cache size 0 max 16384 2024-03-18T01:39:16.917 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 -- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c2c0a2880 msgr2=0x7f4c2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x55b38554eb10 tx=0x7f4c3c02df90 comp rx=0 tx=0).stop 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 -- 172.21.15.7:0/3710345148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c44104bf0 msgr2=0x7f4c4419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c44104bf0 0x7f4c4419a1d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000e9f0 tx=0x7f4c3000eeb0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 -- 172.21.15.7:0/3710345148 shutdown_connections 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c441055b0 0x7f4c441a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4c44104bf0 0x7f4c4419a1d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4c2c0a2880 0x7f4c2c0a4d40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 --2- 172.21.15.7:0/3710345148 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4c44103a00 0x7f4c44199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.912+0000 7f4c4c000640 1 -- 172.21.15.7:0/3710345148 >> 172.21.15.7:0/3710345148 conn(0x7f4c440fef30 msgr2=0x7f4c44107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:16.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.913+0000 7f4c4c000640 1 -- 172.21.15.7:0/3710345148 shutdown_connections 2024-03-18T01:39:16.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:16.913+0000 7f4c4c000640 1 -- 172.21.15.7:0/3710345148 wait complete. 2024-03-18T01:39:16.931 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T01:39:17.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 -- 172.21.15.7:0/767530314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 msgr2=0x7fc3e4105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 --2- 172.21.15.7:0/767530314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 0x7fc3e4105090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009900 tx=0x7fc3d002bc40 comp rx=0 tx=0).stop 2024-03-18T01:39:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 -- 172.21.15.7:0/767530314 shutdown_connections 2024-03-18T01:39:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 --2- 172.21.15.7:0/767530314 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e41055d0 0x7fc3e410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 --2- 172.21.15.7:0/767530314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 0x7fc3e4105090 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 --2- 172.21.15.7:0/767530314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4103a20 0x7fc3e4103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 -- 172.21.15.7:0/767530314 >> 172.21.15.7:0/767530314 conn(0x7fc3e40fefd0 msgr2=0x7fc3e41013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 -- 172.21.15.7:0/767530314 shutdown_connections 2024-03-18T01:39:17.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.161+0000 7fc3ea400640 1 -- 172.21.15.7:0/767530314 wait complete. 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.162+0000 7fc3ea400640 1 Processor -- start 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.162+0000 7fc3ea400640 1 -- start start 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.162+0000 7fc3ea400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4103a20 0x7fc3e4199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.162+0000 7fc3ea400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 0x7fc3e419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.162+0000 7fc3ea400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e41055d0 0x7fc3e41a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.162+0000 7fc3ea400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e410f6d0 con 0x7fc3e41055d0 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.162+0000 7fc3ea400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e410f550 con 0x7fc3e4104c10 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.162+0000 7fc3ea400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e410f850 con 0x7fc3e4103a20 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 0x7fc3e419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4103a20 0x7fc3e4199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 0x7fc3e419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41558/0 (socket says 172.21.15.7:41558) 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 -- 172.21.15.7:0/867534267 learned_addr learned my addr 172.21.15.7:0/867534267 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e41055d0 0x7fc3e41a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 -- 172.21.15.7:0/867534267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4103a20 msgr2=0x7fc3e4199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4103a20 0x7fc3e4199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 -- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e41055d0 msgr2=0x7fc3e41a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e41055d0 0x7fc3e41a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 -- 172.21.15.7:0/867534267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e41a1940 con 0x7fc3e4104c10 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.163+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 0x7fc3e419a270 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc3d002c8f0 tx=0x7fc3d002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.164+0000 7fc3dbe00640 1 -- 172.21.15.7:0/867534267 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0041070 con 0x7fc3e4104c10 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.164+0000 7fc3dbe00640 1 -- 172.21.15.7:0/867534267 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d0004450 con 0x7fc3e4104c10 2024-03-18T01:39:17.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.164+0000 7fc3dbe00640 1 -- 172.21.15.7:0/867534267 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0034440 con 0x7fc3e4104c10 2024-03-18T01:39:17.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.164+0000 7fc3ea400640 1 -- 172.21.15.7:0/867534267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e41a1bd0 con 0x7fc3e4104c10 2024-03-18T01:39:17.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.164+0000 7fc3ea400640 1 -- 172.21.15.7:0/867534267 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e41a2110 con 0x7fc3e4104c10 2024-03-18T01:39:17.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.167+0000 7fc3ed1e3740 1 Processor -- start 2024-03-18T01:39:17.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.167+0000 7fc3ed1e3740 1 -- start start 2024-03-18T01:39:17.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.167+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d45ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.167+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d4adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.172 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.167+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d4d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.167+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5557b1d0d3b0 con 0x5557b1d462b0 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5557b1caa450 con 0x5557b1d458c0 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5557b1c52cb0 con 0x5557b1d4b420 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d45ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d45ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41568/0 (socket says 172.21.15.7:41568) 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3e00640 1 -- 172.21.15.7:0/1039392377 learned_addr learned my addr 172.21.15.7:0/1039392377 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3400640 1 --2- 172.21.15.7:0/1039392377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d4adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3dbe00640 1 -- 172.21.15.7:0/867534267 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fc3d0044070 con 0x7fc3e4104c10 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3e00640 1 -- 172.21.15.7:0/1039392377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 msgr2=0x5557b1d4d8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/1039392377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d4d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3e00640 1 -- 172.21.15.7:0/1039392377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 msgr2=0x5557b1d4adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/1039392377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d4adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.168+0000 7fc3e3e00640 1 -- 172.21.15.7:0/1039392377 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557b1c741f0 con 0x5557b1d458c0 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.169+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/1039392377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d45ce0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009fd0 tx=0x7fc3dc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ea2f709df863a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.169+0000 7fc3d8c00640 1 -- 172.21.15.7:0/1039392377 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc004d20 con 0x5557b1d458c0 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.169+0000 7fc3d8c00640 1 -- 172.21.15.7:0/1039392377 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3dc004050 con 0x5557b1d458c0 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.169+0000 7fc3d8c00640 1 -- 172.21.15.7:0/1039392377 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc005260 con 0x5557b1d458c0 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.169+0000 7fc3dbe00640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3c80a2880 0x7fc3c80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.169+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1039392377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 msgr2=0x5557b1d45ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.169+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1039392377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d45ce0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009fd0 tx=0x7fc3dc0118d0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3dbe00640 1 -- 172.21.15.7:0/867534267 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3d00f0970 con 0x7fc3e4104c10 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1039392377 shutdown_connections 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1039392377 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d4d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1039392377 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d4adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1039392377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d45ce0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1039392377 >> 172.21.15.7:0/1039392377 conn(0x5557b1d40e30 msgr2=0x5557b1d41230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3e3400640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3c80a2880 0x7fc3c80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1039392377 shutdown_connections 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1039392377 wait complete. 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.170+0000 7fc3ed1e3740 1 -- start start 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3e3400640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3c80a2880 0x7fc3c80a4d40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x5557b1d41da0 tx=0x7fc3d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3ed1e3740 1 Processor -- start 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d4fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d50190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5557b1c66110 con 0x5557b1d462b0 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5557b1ca64f0 con 0x5557b1d458c0 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5557b1d4e3d0 con 0x5557b1d4b420 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3ed1e3740 10 client.0 _ll_register_callbacks cb 0x5557b1c53b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3e3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d4fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3e3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3e2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d50190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3e3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d4fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41572/0 (socket says 172.21.15.7:41572) 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3e3400640 1 -- 172.21.15.7:0/1285242988 learned_addr learned my addr 172.21.15.7:0/1285242988 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.171+0000 7fc3e3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38584/0 (socket says 172.21.15.7:38584) 2024-03-18T01:39:17.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.172+0000 7fc3e3e00640 1 -- 172.21.15.7:0/1285242988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 msgr2=0x5557b1d4fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.172+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d4fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.172+0000 7fc3e3e00640 1 -- 172.21.15.7:0/1285242988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 msgr2=0x5557b1d50190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.172+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d50190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.172+0000 7fc3e3e00640 1 -- 172.21.15.7:0/1285242988 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557b19be3f0 con 0x5557b1d4b420 2024-03-18T01:39:17.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.172+0000 7fc3e3400640 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d4fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:17.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.172+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57190 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc01b040 tx=0x7fc3dc004cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ee0711526bfc693 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.172+0000 7fc3cd600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:17.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.173+0000 7fc3cd600640 1 -- 172.21.15.7:0/1285242988 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc028020 con 0x5557b1d4b420 2024-03-18T01:39:17.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.173+0000 7fc3cd600640 1 -- 172.21.15.7:0/1285242988 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3dc004410 con 0x5557b1d4b420 2024-03-18T01:39:17.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.173+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1285242988 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3dc004050 con 0x5557b1d4b420 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.173+0000 7fc3cd600640 1 -- 172.21.15.7:0/1285242988 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc3dc033050 con 0x5557b1d4b420 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.173+0000 7fc3ed1e3740 10 client.25813 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.173+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1285242988 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5557b1a887d0 con 0x5557b1d4b420 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.173+0000 7fc3cd600640 1 -- 172.21.15.7:0/1285242988 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3dc004050 con 0x5557b1d4b420 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.173+0000 7fc3cd600640 1 -- 172.21.15.7:0/1285242988 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fc3dc023570 con 0x5557b1d4b420 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.174+0000 7fc3ed1e3740 10 client.25813 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.174+0000 7fc3ed1e3740 10 client.25813 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.174+0000 7fc3ed1e3740 20 client.25813 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.174+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5557b1d5b930 0x5557b1d5dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.174+0000 7fc3ed1e3740 4 client.25813 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.174+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1285242988 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557b1cb7990 con 0x5557b1d5b930 2024-03-18T01:39:17.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.174+0000 7fc3e3400640 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5557b1d5b930 0x5557b1d5dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.175+0000 7fc3e3400640 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5557b1d5b930 0x5557b1d5dd40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a66818115e6a5fcb server_cookie=58f1faa80937ce65 in_seq=0 out_seq=0 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.175+0000 7fc3cd600640 10 client.25813 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.175+0000 7fc3cd600640 1 -- 172.21.15.7:0/1285242988 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5557b1cb7990 con 0x5557b1d5b930 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.175+0000 7fc3cd600640 10 client.25813 handle_command_reply: tid=0 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.175+0000 7fc3ed1e3740 1 client.25813 shutdown 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1285242988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 msgr2=0x5557b1d57190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57190 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc01b040 tx=0x7fc3dc004cf0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1285242988 shutdown_connections 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5557b1d5b930 0x5557b1d5dd40 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57190 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d50190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1285242988 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d4fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3cd600640 0 client.25813 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:39:17.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1285242988 >> 172.21.15.7:0/1285242988 conn(0x5557b1d40e30 msgr2=0x5557b1d401f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1285242988 shutdown_connections 2024-03-18T01:39:17.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1285242988 wait complete. 2024-03-18T01:39:17.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.176+0000 7fc3ed1e3740 20 client.25813 trim_cache size 0 max 16384 2024-03-18T01:39:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3ed1e3740 1 Processor -- start 2024-03-18T01:39:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3ed1e3740 1 -- start start 2024-03-18T01:39:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d458c0 0x5557b1d4ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d462b0 0x5557b1d4f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41ac510 con 0x5557b1d458c0 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41ae080 con 0x5557b1d462b0 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41acbe0 con 0x5557b1d4b420 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3e3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d458c0 0x5557b1d4ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d462b0 0x5557b1d4f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3e3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d462b0 0x5557b1d4f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41584/0 (socket says 172.21.15.7:41584) 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3e3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d458c0 0x5557b1d4ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:46196/0 (socket says 172.21.15.7:46196) 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.191+0000 7fc3e2a00640 1 -- 172.21.15.7:0/1578376177 learned_addr learned my addr 172.21.15.7:0/1578376177 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3e2a00640 1 -- 172.21.15.7:0/1578376177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 msgr2=0x5557b1d57150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3e2a00640 1 -- 172.21.15.7:0/1578376177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d458c0 msgr2=0x5557b1d4ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d458c0 0x5557b1d4ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3e2a00640 1 -- 172.21.15.7:0/1578376177 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557b1d3cc10 con 0x5557b1d462b0 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3e3400640 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d458c0 0x5557b1d4ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d462b0 0x5557b1d4f450 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00b9940 tx=0x7fc3d002bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd5d340e0e979930 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3cfe00640 1 -- 172.21.15.7:0/1578376177 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d004b040 con 0x5557b1d462b0 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3cfe00640 1 -- 172.21.15.7:0/1578376177 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d0004c60 con 0x5557b1d462b0 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.192+0000 7fc3cfe00640 1 -- 172.21.15.7:0/1578376177 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d00bd260 con 0x5557b1d462b0 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1578376177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d462b0 msgr2=0x5557b1d4f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d462b0 0x5557b1d4f450 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc3d00b9940 tx=0x7fc3d002bc40 comp rx=0 tx=0).stop 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1578376177 shutdown_connections 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d57150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d462b0 0x5557b1d4f450 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1578376177 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d458c0 0x5557b1d4ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1578376177 >> 172.21.15.7:0/1578376177 conn(0x5557b1d40e30 msgr2=0x5557b1d4e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1578376177 shutdown_connections 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.193+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1578376177 wait complete. 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 1 -- start start 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 1 Processor -- start 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d55610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d56620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41ad980 con 0x5557b1d462b0 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41ad2b0 con 0x5557b1d458c0 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e41aa4f0 con 0x5557b1d4b420 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.194+0000 7fc3ed1e3740 10 client.0 _ll_register_callbacks cb 0x7fc3d40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41592/0 (socket says 172.21.15.7:41592) 2024-03-18T01:39:17.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 -- 172.21.15.7:0/1026094618 learned_addr learned my addr 172.21.15.7:0/1026094618 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d55610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3e00640 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d56620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 -- 172.21.15.7:0/1026094618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 msgr2=0x5557b1d56620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d56620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 -- 172.21.15.7:0/1026094618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 msgr2=0x5557b1d55610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d55610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 -- 172.21.15.7:0/1026094618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557b1d50500 con 0x5557b1d458c0 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d55610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3e3400640 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d550d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x5557b1d5e5b0 tx=0x7fc3d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44a33fbae0ce2869 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.195+0000 7fc3cd600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3cd600640 1 -- 172.21.15.7:0/1026094618 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d402bd10 con 0x5557b1d458c0 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1026094618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3d0004c60 con 0x5557b1d458c0 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3cd600640 1 -- 172.21.15.7:0/1026094618 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3d402b3e0 con 0x5557b1d458c0 2024-03-18T01:39:17.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3cd600640 1 -- 172.21.15.7:0/1026094618 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc3d403b050 con 0x5557b1d458c0 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3ed1e3740 10 client.25841 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1026094618 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5557b1d45500 con 0x5557b1d458c0 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3cd600640 1 -- 172.21.15.7:0/1026094618 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d0004c60 con 0x5557b1d458c0 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3cd600640 1 -- 172.21.15.7:0/1026094618 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fc3d40048c0 con 0x5557b1d458c0 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3ed1e3740 10 client.25841 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3ed1e3740 10 client.25841 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3ed1e3740 20 client.25841 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.196+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5557b1dc2a80 0x5557b1dc4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.197+0000 7fc3ed1e3740 4 client.25841 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.197+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1026094618 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5557b1cb1ce0 con 0x5557b1dc2a80 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.197+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5557b1dc2a80 0x5557b1dc4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.197+0000 7fc3e2a00640 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5557b1dc2a80 0x5557b1dc4ea0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27bf7fbafbb84920 server_cookie=2c42d028fbd5d78d in_seq=0 out_seq=0 2024-03-18T01:39:17.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.197+0000 7fc3cd600640 10 client.25841 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.197+0000 7fc3cd600640 1 -- 172.21.15.7:0/1026094618 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5557b1cb1ce0 con 0x5557b1dc2a80 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.197+0000 7fc3cd600640 10 client.25841 handle_command_reply: tid=0 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 client.25841 shutdown 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1026094618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 msgr2=0x5557b1d550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d550d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x5557b1d5e5b0 tx=0x7fc3d40046c0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1026094618 shutdown_connections 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5557b1dc2a80 0x5557b1dc4ea0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5557b1d4b420 0x5557b1d56620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5557b1d462b0 0x5557b1d55610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3cd600640 0 client.25841 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 --2- 172.21.15.7:0/1026094618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557b1d458c0 0x5557b1d550d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1026094618 >> 172.21.15.7:0/1026094618 conn(0x5557b1d40e30 msgr2=0x5557b1d4f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1026094618 shutdown_connections 2024-03-18T01:39:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 1 -- 172.21.15.7:0/1026094618 wait complete. 2024-03-18T01:39:17.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.198+0000 7fc3ed1e3740 20 client.25841 trim_cache size 0 max 16384 2024-03-18T01:39:17.205 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 -- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3c80a2880 msgr2=0x7fc3c80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3c80a2880 0x7fc3c80a4d40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x5557b1d41da0 tx=0x7fc3d4005f90 comp rx=0 tx=0).stop 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 -- 172.21.15.7:0/867534267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 msgr2=0x7fc3e419a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 0x7fc3e419a270 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc3d002c8f0 tx=0x7fc3d002c660 comp rx=0 tx=0).stop 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 -- 172.21.15.7:0/867534267 shutdown_connections 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc3c80a2880 0x7fc3c80a4d40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc3e41055d0 0x7fc3e41a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3e4104c10 0x7fc3e419a270 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 --2- 172.21.15.7:0/867534267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3e4103a20 0x7fc3e4199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 -- 172.21.15.7:0/867534267 >> 172.21.15.7:0/867534267 conn(0x7fc3e40fefd0 msgr2=0x7fc3e4107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 -- 172.21.15.7:0/867534267 shutdown_connections 2024-03-18T01:39:17.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.199+0000 7fc3ea400640 1 -- 172.21.15.7:0/867534267 wait complete. 2024-03-18T01:39:17.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:16 smithi007 ceph-mon[31996]: pgmap v450: 129 pgs: 129 active+clean; 4.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 320 KiB/s wr, 40 op/s 2024-03-18T01:39:17.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:16 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:17.214 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:16 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2613781612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:39:17.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T01:39:17.240 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether build environment is sane... yes 2024-03-18T01:39:17.253 INFO:tasks.workunit.client.0.smithi007.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-18T01:39:17.253 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gawk... gawk 2024-03-18T01:39:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:16 smithi046 ceph-mon[36108]: pgmap v450: 129 pgs: 129 active+clean; 4.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 320 KiB/s wr, 40 op/s 2024-03-18T01:39:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:16 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:16 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2613781612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:39:17.281 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether make sets $(MAKE)... yes 2024-03-18T01:39:17.355 INFO:tasks.workunit.client.0.smithi007.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-18T01:39:17.355 INFO:tasks.workunit.client.0.smithi007.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-18T01:39:17.374 INFO:tasks.workunit.client.0.smithi007.stdout:checking for style of include used by make... GNU 2024-03-18T01:39:17.380 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc... gcc 2024-03-18T01:39:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:16 smithi203 ceph-mon[36345]: pgmap v450: 129 pgs: 129 active+clean; 4.1 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 320 KiB/s wr, 40 op/s 2024-03-18T01:39:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:16 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:16 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2613781612' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:39:17.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.418+0000 7f560de00640 1 -- 172.21.15.7:0/3983084940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5608075820 msgr2=0x7f5608073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.418+0000 7f560de00640 1 --2- 172.21.15.7:0/3983084940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5608075820 0x7f5608073c70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009900 tx=0x7f55fc02bc40 comp rx=0 tx=0).stop 2024-03-18T01:39:17.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.418+0000 7f560de00640 1 -- 172.21.15.7:0/3983084940 shutdown_connections 2024-03-18T01:39:17.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.418+0000 7f560de00640 1 --2- 172.21.15.7:0/3983084940 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f560810dbf0 0x7f560810ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.418+0000 7f560de00640 1 --2- 172.21.15.7:0/3983084940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5608074240 0x7f56080746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.419+0000 7f560de00640 1 --2- 172.21.15.7:0/3983084940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5608075820 0x7f5608073c70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.419+0000 7f560de00640 1 -- 172.21.15.7:0/3983084940 >> 172.21.15.7:0/3983084940 conn(0x7f56080fcb90 msgr2=0x7f56080fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.419+0000 7f560de00640 1 -- 172.21.15.7:0/3983084940 shutdown_connections 2024-03-18T01:39:17.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.419+0000 7f560de00640 1 -- 172.21.15.7:0/3983084940 wait complete. 2024-03-18T01:39:17.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560de00640 1 Processor -- start 2024-03-18T01:39:17.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560de00640 1 -- start start 2024-03-18T01:39:17.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5608074240 0x7f560819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560de00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5608075820 0x7f560819e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560de00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560810dbf0 0x7f56081a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560de00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f56081138c0 con 0x7f5608075820 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560de00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5608113740 con 0x7f5608074240 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560de00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5608113a40 con 0x7f560810dbf0 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.420+0000 7f560ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5608074240 0x7f560819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5608074240 0x7f560819ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41614/0 (socket says 172.21.15.7:41614) 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560ca00640 1 -- 172.21.15.7:0/230266185 learned_addr learned my addr 172.21.15.7:0/230266185 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f5607e00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5608075820 0x7f560819e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560d400640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560810dbf0 0x7f56081a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560ca00640 1 -- 172.21.15.7:0/230266185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560810dbf0 msgr2=0x7f56081a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560ca00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560810dbf0 0x7f56081a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560ca00640 1 -- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5608075820 msgr2=0x7f560819e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560ca00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5608075820 0x7f560819e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560ca00640 1 -- 172.21.15.7:0/230266185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56081a5af0 con 0x7f5608074240 2024-03-18T01:39:17.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f5607e00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5608075820 0x7f560819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:17.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560d400640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560810dbf0 0x7f56081a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:17.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.421+0000 7f560ca00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5608074240 0x7f560819ded0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f55fc02c920 tx=0x7f55fc005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.422+0000 7f5605600640 1 -- 172.21.15.7:0/230266185 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc041070 con 0x7f5608074240 2024-03-18T01:39:17.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.422+0000 7f5605600640 1 -- 172.21.15.7:0/230266185 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55fc0044d0 con 0x7f5608074240 2024-03-18T01:39:17.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.422+0000 7f560de00640 1 -- 172.21.15.7:0/230266185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56081a5d80 con 0x7f5608074240 2024-03-18T01:39:17.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.422+0000 7f5605600640 1 -- 172.21.15.7:0/230266185 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc034440 con 0x7f5608074240 2024-03-18T01:39:17.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.422+0000 7f560de00640 1 -- 172.21.15.7:0/230266185 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56081a6200 con 0x7f5608074240 2024-03-18T01:39:17.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.426+0000 7f561216a740 1 Processor -- start 2024-03-18T01:39:17.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.426+0000 7f561216a740 1 -- start start 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f561216a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633ccea1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f561216a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633ccea6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f561216a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633ccea9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f561216a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5633cce693b0 con 0x5633ccea75f0 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f5605600640 1 -- 172.21.15.7:0/230266185 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f55fc044070 con 0x7f5608074240 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f561216a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5633cce06450 con 0x5633ccea2480 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f561216a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5633ccdaecb0 con 0x5633ccea1a90 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f560d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633ccea1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f560d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633ccea1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38618/0 (socket says 172.21.15.7:38618) 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f560d400640 1 -- 172.21.15.7:0/1340408368 learned_addr learned my addr 172.21.15.7:0/1340408368 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.427+0000 7f560ca00640 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633ccea6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.428+0000 7f5607e00640 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633ccea9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.428+0000 7f560d400640 1 -- 172.21.15.7:0/1340408368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 msgr2=0x5633ccea6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.428+0000 7f560d400640 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633ccea6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.428+0000 7f560d400640 1 -- 172.21.15.7:0/1340408368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 msgr2=0x5633ccea9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.428+0000 7f560d400640 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633ccea9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.428+0000 7f560d400640 1 -- 172.21.15.7:0/1340408368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633ccdd01f0 con 0x5633ccea1a90 2024-03-18T01:39:17.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.428+0000 7f5607e00640 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633ccea9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.428+0000 7f560d400640 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633ccea1eb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5600009fb0 tx=0x7f5600010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3141a6a0d57ac1d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.429+0000 7f55f9c00640 1 -- 172.21.15.7:0/1340408368 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600016070 con 0x5633ccea1a90 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.429+0000 7f55f9c00640 1 -- 172.21.15.7:0/1340408368 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5600012bf0 con 0x5633ccea1a90 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.429+0000 7f55f9c00640 1 -- 172.21.15.7:0/1340408368 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f560001bd20 con 0x5633ccea1a90 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.429+0000 7f5605600640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55f00a2870 0x7f55f00a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.429+0000 7f5605600640 1 -- 172.21.15.7:0/230266185 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f55fc0f0a60 con 0x7f5608074240 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f560ca00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55f00a2870 0x7f55f00a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 -- 172.21.15.7:0/1340408368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 msgr2=0x5633ccea1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633ccea1eb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5600009fb0 tx=0x7f5600010800 comp rx=0 tx=0).stop 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 -- 172.21.15.7:0/1340408368 shutdown_connections 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633ccea9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633ccea6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 --2- 172.21.15.7:0/1340408368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633ccea1eb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 -- 172.21.15.7:0/1340408368 >> 172.21.15.7:0/1340408368 conn(0x5633cce9d000 msgr2=0x5633cce9d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 -- 172.21.15.7:0/1340408368 shutdown_connections 2024-03-18T01:39:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f561216a740 1 -- 172.21.15.7:0/1340408368 wait complete. 2024-03-18T01:39:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.430+0000 7f560ca00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55f00a2870 0x7f55f00a4d30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x5633ccea2880 tx=0x7f55fc0dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.431+0000 7f561216a740 1 -- start start 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.431+0000 7f561216a740 1 Processor -- start 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f561216a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea1a90 0x5633cceabe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f561216a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633cceac370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f561216a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea75f0 0x5633cceb3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f561216a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5633ccdc2110 con 0x5633ccea1a90 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f561216a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5633cce024f0 con 0x5633ccea2480 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f561216a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5633cceaa5a0 con 0x5633ccea75f0 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f561216a740 10 client.0 _ll_register_callbacks cb 0x5633ccdafb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f5607e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633cceac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f5607e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633cceac370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41640/0 (socket says 172.21.15.7:41640) 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f5607e00640 1 -- 172.21.15.7:0/3143701588 learned_addr learned my addr 172.21.15.7:0/3143701588 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f560d400640 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea75f0 0x5633cceb3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.432+0000 7f560d400640 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea1a90 0x5633cceabe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f5607e00640 1 -- 172.21.15.7:0/3143701588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea75f0 msgr2=0x5633cceb3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f5607e00640 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea75f0 0x5633cceb3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f5607e00640 1 -- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea1a90 msgr2=0x5633cceabe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f5607e00640 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea1a90 0x5633cceabe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f5607e00640 1 -- 172.21.15.7:0/3143701588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633ccb1a3f0 con 0x5633ccea2480 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f560d400640 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea1a90 0x5633cceabe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f5607e00640 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633cceac370 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f55f4013790 tx=0x7f55f4013c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b77f9e2758dd2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f55ee000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f55ee000640 1 -- 172.21.15.7:0/3143701588 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4020060 con 0x5633ccea2480 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f561216a740 1 -- 172.21.15.7:0/3143701588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5600012bf0 con 0x5633ccea2480 2024-03-18T01:39:17.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.433+0000 7f55ee000640 1 -- 172.21.15.7:0/3143701588 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f55f4011dd0 con 0x5633ccea2480 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f55ee000640 1 -- 172.21.15.7:0/3143701588 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f55f4027050 con 0x5633ccea2480 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f561216a740 10 client.25853 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f561216a740 1 -- 172.21.15.7:0/3143701588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5633ccbe47d0 con 0x5633ccea2480 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f55ee000640 1 -- 172.21.15.7:0/3143701588 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5600012bf0 con 0x5633ccea2480 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f55ee000640 1 -- 172.21.15.7:0/3143701588 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f55f401a030 con 0x5633ccea2480 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f561216a740 10 client.25853 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f561216a740 10 client.25853 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f561216a740 20 client.25853 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f561216a740 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5633cceb7c80 0x5633cceba0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f561216a740 4 client.25853 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.434+0000 7f561216a740 1 -- 172.21.15.7:0/3143701588 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633cce13990 con 0x5633cceb7c80 2024-03-18T01:39:17.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.435+0000 7f560d400640 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5633cceb7c80 0x5633cceba0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.435+0000 7f560d400640 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5633cceb7c80 0x5633cceba0e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e2a21ccef79dfb71 server_cookie=17de0a378569b4ad in_seq=0 out_seq=0 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.435+0000 7f55ee000640 10 client.25853 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f55ee000640 1 -- 172.21.15.7:0/3143701588 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633cce13990 con 0x5633cceb7c80 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f55ee000640 10 client.25853 handle_command_reply: tid=0 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 client.25853 shutdown 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 -- 172.21.15.7:0/3143701588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 msgr2=0x5633cceac370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633cceac370 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f55f4013790 tx=0x7f55f4013c50 comp rx=0 tx=0).stop 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 -- 172.21.15.7:0/3143701588 shutdown_connections 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5633cceb7c80 0x5633cceba0e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea75f0 0x5633cceb3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea2480 0x5633cceac370 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 --2- 172.21.15.7:0/3143701588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea1a90 0x5633cceabe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f55ee000640 0 client.25853 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:39:17.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 -- 172.21.15.7:0/3143701588 >> 172.21.15.7:0/3143701588 conn(0x5633cce9d000 msgr2=0x5633cce9c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 -- 172.21.15.7:0/3143701588 shutdown_connections 2024-03-18T01:39:17.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.437+0000 7f561216a740 1 -- 172.21.15.7:0/3143701588 wait complete. 2024-03-18T01:39:17.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.438+0000 7f561216a740 20 client.25853 trim_cache size 0 max 16384 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.455+0000 7f561216a740 1 Processor -- start 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.455+0000 7f561216a740 1 -- start start 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.455+0000 7f561216a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633cce9b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.455+0000 7f561216a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea2480 0x5633cceb3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.455+0000 7f561216a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea75f0 0x5633cce9bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f561216a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f56081b0710 con 0x5633ccea2480 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f561216a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56081b2280 con 0x5633ccea75f0 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f561216a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f56081b0de0 con 0x5633ccea1a90 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f560d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea75f0 0x5633cce9bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f5607e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea2480 0x5633cceb3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f560d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea75f0 0x5633cce9bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41642/0 (socket says 172.21.15.7:41642) 2024-03-18T01:39:17.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f560d400640 1 -- 172.21.15.7:0/421105330 learned_addr learned my addr 172.21.15.7:0/421105330 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f560d400640 1 -- 172.21.15.7:0/421105330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 msgr2=0x5633cce9b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f560d400640 1 --2- 172.21.15.7:0/421105330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633cce9b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f560d400640 1 -- 172.21.15.7:0/421105330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea2480 msgr2=0x5633cceb3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f560d400640 1 --2- 172.21.15.7:0/421105330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea2480 0x5633cceb3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.456+0000 7f560d400640 1 -- 172.21.15.7:0/421105330 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633cce9ce30 con 0x5633ccea75f0 2024-03-18T01:39:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.457+0000 7f560d400640 1 --2- 172.21.15.7:0/421105330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea75f0 0x5633cce9bed0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5600000f80 tx=0x7f560001b6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7ce144315f3f4b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.457+0000 7f55f9200640 1 -- 172.21.15.7:0/421105330 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600016040 con 0x5633ccea75f0 2024-03-18T01:39:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.457+0000 7f55f9200640 1 -- 172.21.15.7:0/421105330 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f560001b7d0 con 0x5633ccea75f0 2024-03-18T01:39:17.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.457+0000 7f55f9200640 1 -- 172.21.15.7:0/421105330 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5600012770 con 0x5633ccea75f0 2024-03-18T01:39:17.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 -- 172.21.15.7:0/421105330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea75f0 msgr2=0x5633cce9bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 --2- 172.21.15.7:0/421105330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea75f0 0x5633cce9bed0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5600000f80 tx=0x7f560001b6c0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 -- 172.21.15.7:0/421105330 shutdown_connections 2024-03-18T01:39:17.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 --2- 172.21.15.7:0/421105330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea75f0 0x5633cce9bed0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 --2- 172.21.15.7:0/421105330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea2480 0x5633cceb3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 --2- 172.21.15.7:0/421105330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea1a90 0x5633cce9b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 -- 172.21.15.7:0/421105330 >> 172.21.15.7:0/421105330 conn(0x5633cce9d000 msgr2=0x5633cceabe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 -- 172.21.15.7:0/421105330 shutdown_connections 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 -- 172.21.15.7:0/421105330 wait complete. 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.458+0000 7f561216a740 1 -- start start 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f561216a740 1 Processor -- start 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f561216a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea1a90 0x5633cceb0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f561216a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea2480 0x5633cceb1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f561216a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633cceb2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f561216a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f56081b1b80 con 0x5633ccea75f0 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f561216a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f56081b14b0 con 0x5633ccea1a90 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f561216a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f56081ae6f0 con 0x5633ccea2480 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f561216a740 10 client.0 _ll_register_callbacks cb 0x7f55f4009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:17.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f5607e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea2480 0x5633cceb1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f560d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea1a90 0x5633cceb0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f5607e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea2480 0x5633cceb1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38628/0 (socket says 172.21.15.7:38628) 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f5607e00640 1 -- 172.21.15.7:0/565772911 learned_addr learned my addr 172.21.15.7:0/565772911 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.459+0000 7f560d400640 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633cceb2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f5607e00640 1 -- 172.21.15.7:0/565772911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea1a90 msgr2=0x5633cceb0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f5607e00640 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea1a90 0x5633cceb0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f5607e00640 1 -- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 msgr2=0x5633cceb2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f5607e00640 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633cceb2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f5607e00640 1 -- 172.21.15.7:0/565772911 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633cceb2620 con 0x5633ccea2480 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f560d400640 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633cceb2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f5607e00640 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea2480 0x5633cceb1020 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f55f4016040 tx=0x7f55f40118f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1540d27250a7b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f55ee000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f55ee000640 1 -- 172.21.15.7:0/565772911 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f4000e60 con 0x5633ccea2480 2024-03-18T01:39:17.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f561216a740 1 -- 172.21.15.7:0/565772911 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f560001b7d0 con 0x5633ccea2480 2024-03-18T01:39:17.463 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.460+0000 7f55ee000640 1 -- 172.21.15.7:0/565772911 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f55f401baf0 con 0x5633ccea2480 2024-03-18T01:39:17.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f55ee000640 1 -- 172.21.15.7:0/565772911 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f55f4034050 con 0x5633ccea2480 2024-03-18T01:39:17.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f561216a740 10 client.25834 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:17.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f561216a740 1 -- 172.21.15.7:0/565772911 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5633ccf1e0e0 con 0x5633ccea2480 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f55ee000640 1 -- 172.21.15.7:0/565772911 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f560001b7d0 con 0x5633ccea2480 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f55ee000640 1 -- 172.21.15.7:0/565772911 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f55f401a040 con 0x5633ccea2480 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f561216a740 10 client.25834 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f561216a740 10 client.25834 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f561216a740 20 client.25834 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f561216a740 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5633ccf1eab0 0x5633ccf20ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f561216a740 4 client.25834 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.461+0000 7f561216a740 1 -- 172.21.15.7:0/565772911 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5633cce0dce0 con 0x5633ccf1eab0 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.462+0000 7f560d400640 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5633ccf1eab0 0x5633ccf20ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.462+0000 7f560d400640 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5633ccf1eab0 0x5633ccf20ed0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=743aef47c17cc248 server_cookie=4a2510cce65114e9 in_seq=0 out_seq=0 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.462+0000 7f55ee000640 10 client.25834 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f55ee000640 1 -- 172.21.15.7:0/565772911 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5633cce0dce0 con 0x5633ccf1eab0 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f55ee000640 10 client.25834 handle_command_reply: tid=0 2024-03-18T01:39:17.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 client.25834 shutdown 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 -- 172.21.15.7:0/565772911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea2480 msgr2=0x5633cceb1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea2480 0x5633cceb1020 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f55f4016040 tx=0x7f55f40118f0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 -- 172.21.15.7:0/565772911 shutdown_connections 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5633ccf1eab0 0x5633ccf20ed0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5633ccea75f0 0x5633cceb2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5633ccea2480 0x5633cceb1020 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 --2- 172.21.15.7:0/565772911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5633ccea1a90 0x5633cceb0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f55ee000640 0 client.25834 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 -- 172.21.15.7:0/565772911 >> 172.21.15.7:0/565772911 conn(0x5633cce9d000 msgr2=0x5633ccea0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 -- 172.21.15.7:0/565772911 shutdown_connections 2024-03-18T01:39:17.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 1 -- 172.21.15.7:0/565772911 wait complete. 2024-03-18T01:39:17.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.463+0000 7f561216a740 20 client.25834 trim_cache size 0 max 16384 2024-03-18T01:39:17.467 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 -- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55f00a2870 msgr2=0x7f55f00a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55f00a2870 0x7f55f00a4d30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x5633ccea2880 tx=0x7f55fc0dd040 comp rx=0 tx=0).stop 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 -- 172.21.15.7:0/230266185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5608074240 msgr2=0x7f560819ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5608074240 0x7f560819ded0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f55fc02c920 tx=0x7f55fc005d50 comp rx=0 tx=0).stop 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 -- 172.21.15.7:0/230266185 shutdown_connections 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f560810dbf0 0x7f56081a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55f00a2870 0x7f55f00a4d30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5608075820 0x7f560819e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 --2- 172.21.15.7:0/230266185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5608074240 0x7f560819ded0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 -- 172.21.15.7:0/230266185 >> 172.21.15.7:0/230266185 conn(0x7f56080fcb90 msgr2=0x7f5608100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 -- 172.21.15.7:0/230266185 shutdown_connections 2024-03-18T01:39:17.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.464+0000 7f560de00640 1 -- 172.21.15.7:0/230266185 wait complete. 2024-03-18T01:39:17.482 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:39:17.544 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether the C compiler works... yes 2024-03-18T01:39:17.544 INFO:tasks.workunit.client.0.smithi007.stdout:checking for C compiler default output file name... a.out 2024-03-18T01:39:17.621 INFO:tasks.workunit.client.0.smithi007.stdout:checking for suffix of executables... 2024-03-18T01:39:17.699 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are cross compiling... no 2024-03-18T01:39:17.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 -- 172.21.15.7:0/2528580146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103a00 msgr2=0x7f5350103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 --2- 172.21.15.7:0/2528580146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103a00 0x7f5350103e20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5340009900 tx=0x7f534002bc50 comp rx=0 tx=0).stop 2024-03-18T01:39:17.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 -- 172.21.15.7:0/2528580146 shutdown_connections 2024-03-18T01:39:17.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 --2- 172.21.15.7:0/2528580146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f53501055b0 0x7f535010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 --2- 172.21.15.7:0/2528580146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5350104bf0 0x7f5350105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 --2- 172.21.15.7:0/2528580146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350103a00 0x7f5350103e20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 -- 172.21.15.7:0/2528580146 >> 172.21.15.7:0/2528580146 conn(0x7f53500fef30 msgr2=0x7f53501013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 -- 172.21.15.7:0/2528580146 shutdown_connections 2024-03-18T01:39:17.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.718+0000 7f5356a00640 1 -- 172.21.15.7:0/2528580146 wait complete. 2024-03-18T01:39:17.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.719+0000 7f5356a00640 1 Processor -- start 2024-03-18T01:39:17.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.719+0000 7f5356a00640 1 -- start start 2024-03-18T01:39:17.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.719+0000 7f5356a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5350103a00 0x7f5350199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.719+0000 7f5356a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350104bf0 0x7f535019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.719+0000 7f5356a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501055b0 0x7f53501a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.719+0000 7f5356a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f535010f6d0 con 0x7f53501055b0 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.719+0000 7f5356a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f535010f550 con 0x7f5350103a00 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.719+0000 7f5356a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f535010f850 con 0x7f5350104bf0 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5350103a00 0x7f5350199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350104bf0 0x7f535019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350104bf0 0x7f535019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38638/0 (socket says 172.21.15.7:38638) 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5350103a00 0x7f5350199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41672/0 (socket says 172.21.15.7:41672) 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534f400640 1 -- 172.21.15.7:0/1294150133 learned_addr learned my addr 172.21.15.7:0/1294150133 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f5354a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501055b0 0x7f53501a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534f400640 1 -- 172.21.15.7:0/1294150133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5350103a00 msgr2=0x7f5350199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534f400640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5350103a00 0x7f5350199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534f400640 1 -- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501055b0 msgr2=0x7f53501a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534f400640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501055b0 0x7f53501a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.720+0000 7f534f400640 1 -- 172.21.15.7:0/1294150133 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53501a1960 con 0x7f5350104bf0 2024-03-18T01:39:17.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.721+0000 7f534f400640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350104bf0 0x7f535019a290 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f533c00e9f0 tx=0x7f533c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.721+0000 7f534cc00640 1 -- 172.21.15.7:0/1294150133 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c00cd80 con 0x7f5350104bf0 2024-03-18T01:39:17.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.721+0000 7f534cc00640 1 -- 172.21.15.7:0/1294150133 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c010910 con 0x7f5350104bf0 2024-03-18T01:39:17.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.721+0000 7f534cc00640 1 -- 172.21.15.7:0/1294150133 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c018990 con 0x7f5350104bf0 2024-03-18T01:39:17.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.721+0000 7f5356a00640 1 -- 172.21.15.7:0/1294150133 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53501a1c50 con 0x7f5350104bf0 2024-03-18T01:39:17.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.721+0000 7f5356a00640 1 -- 172.21.15.7:0/1294150133 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53501a2120 con 0x7f5350104bf0 2024-03-18T01:39:17.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.724+0000 7f53597bd740 1 Processor -- start 2024-03-18T01:39:17.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.724+0000 7f53597bd740 1 -- start start 2024-03-18T01:39:17.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f53597bd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f534cc00640 1 -- 172.21.15.7:0/1294150133 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f533c010ac0 con 0x7f5350104bf0 2024-03-18T01:39:17.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f53597bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd13db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f53597bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd168b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f53597bd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55c11fcd63b0 con 0x55c11fd0e8c0 2024-03-18T01:39:17.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f53597bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c11fc73450 con 0x55c11fd0f2b0 2024-03-18T01:39:17.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f53597bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c11fc1bcb0 con 0x55c11fd14420 2024-03-18T01:39:17.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f534f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f534fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd13db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f534fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd13db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41674/0 (socket says 172.21.15.7:41674) 2024-03-18T01:39:17.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f534fe00640 1 -- 172.21.15.7:0/1237557561 learned_addr learned my addr 172.21.15.7:0/1237557561 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.725+0000 7f5354a00640 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f534fe00640 1 -- 172.21.15.7:0/1237557561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 msgr2=0x55c11fd168b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f534fe00640 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd168b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f534fe00640 1 -- 172.21.15.7:0/1237557561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 msgr2=0x55c11fd0ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f534fe00640 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f534fe00640 1 -- 172.21.15.7:0/1237557561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c11fc3d1f0 con 0x55c11fd0f2b0 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f534f400640 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f534fe00640 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd13db0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5340005ab0 tx=0x7f5340009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e72255bce1cffd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f5345200640 1 -- 172.21.15.7:0/1237557561 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5340042070 con 0x55c11fd0f2b0 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f5345200640 1 -- 172.21.15.7:0/1237557561 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f534002cba0 con 0x55c11fd0f2b0 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.726+0000 7f5345200640 1 -- 172.21.15.7:0/1237557561 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5340005130 con 0x55c11fd0f2b0 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.727+0000 7f534cc00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53340a2880 0x7f53340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.727+0000 7f534cc00640 1 -- 172.21.15.7:0/1294150133 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f533c0cf100 con 0x7f5350104bf0 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.727+0000 7f5354a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53340a2880 0x7f53340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.727+0000 7f53597bd740 1 -- 172.21.15.7:0/1237557561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 msgr2=0x55c11fd13db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.727+0000 7f53597bd740 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd13db0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5340005ab0 tx=0x7f5340009560 comp rx=0 tx=0).stop 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f53597bd740 1 -- 172.21.15.7:0/1237557561 shutdown_connections 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f53597bd740 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f53597bd740 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd13db0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f53597bd740 1 --2- 172.21.15.7:0/1237557561 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f53597bd740 1 -- 172.21.15.7:0/1237557561 >> 172.21.15.7:0/1237557561 conn(0x55c11fd09e30 msgr2=0x55c11fd0a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f53597bd740 1 -- 172.21.15.7:0/1237557561 shutdown_connections 2024-03-18T01:39:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f5354a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53340a2880 0x7f53340a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55c11fd0fb10 tx=0x7f5348009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f53597bd740 1 -- 172.21.15.7:0/1237557561 wait complete. 2024-03-18T01:39:17.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.728+0000 7f53597bd740 1 -- start start 2024-03-18T01:39:17.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f53597bd740 1 Processor -- start 2024-03-18T01:39:17.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f53597bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0e8c0 0x55c11fd18b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f53597bd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0f2b0 0x55c11fd190a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f53597bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd14420 0x55c11fd200e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f53597bd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55c11fc2f110 con 0x55c11fd0f2b0 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f53597bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55c11fc6f4f0 con 0x55c11fd14420 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f53597bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55c11fd17370 con 0x55c11fd0e8c0 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f53597bd740 10 client.0 _ll_register_callbacks cb 0x55c11fc1cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f534fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0e8c0 0x55c11fd18b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f534fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0e8c0 0x55c11fd18b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38654/0 (socket says 172.21.15.7:38654) 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f534fe00640 1 -- 172.21.15.7:0/3895227366 learned_addr learned my addr 172.21.15.7:0/3895227366 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.729+0000 7f5354a00640 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd14420 0x55c11fd200e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f534f400640 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0f2b0 0x55c11fd190a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f534fe00640 1 -- 172.21.15.7:0/3895227366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd14420 msgr2=0x55c11fd200e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f534fe00640 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd14420 0x55c11fd200e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f534fe00640 1 -- 172.21.15.7:0/3895227366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0f2b0 msgr2=0x55c11fd190a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f534fe00640 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0f2b0 0x55c11fd190a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f534fe00640 1 -- 172.21.15.7:0/3895227366 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11f9873f0 con 0x55c11fd0e8c0 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f534f400640 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0f2b0 0x55c11fd190a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f534fe00640 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0e8c0 0x55c11fd18b60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5340036040 tx=0x7f5340005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d0216fab45ede07 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.730+0000 7f5339600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.731+0000 7f5339600640 1 -- 172.21.15.7:0/3895227366 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534004b070 con 0x55c11fd0e8c0 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.731+0000 7f5339600640 1 -- 172.21.15.7:0/3895227366 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f53400052e0 con 0x55c11fd0e8c0 2024-03-18T01:39:17.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.731+0000 7f53597bd740 1 -- 172.21.15.7:0/3895227366 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f534002cba0 con 0x55c11fd0e8c0 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.731+0000 7f5339600640 1 -- 172.21.15.7:0/3895227366 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5340042040 con 0x55c11fd0e8c0 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.731+0000 7f53597bd740 10 client.25846 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.731+0000 7f53597bd740 1 -- 172.21.15.7:0/3895227366 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c11fa517d0 con 0x55c11fd0e8c0 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.731+0000 7f5339600640 1 -- 172.21.15.7:0/3895227366 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f534002cba0 con 0x55c11fd0e8c0 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.731+0000 7f5339600640 1 -- 172.21.15.7:0/3895227366 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f5340047460 con 0x55c11fd0e8c0 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f53597bd740 10 client.25846 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f53597bd740 10 client.25846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f53597bd740 20 client.25846 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f53597bd740 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55c11fd245d0 0x55c11fd269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f53597bd740 4 client.25846 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f53597bd740 1 -- 172.21.15.7:0/3895227366 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c11fc80990 con 0x55c11fd245d0 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f534f400640 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55c11fd245d0 0x55c11fd269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f534f400640 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55c11fd245d0 0x55c11fd269c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7be8b5ffe6686358 server_cookie=f56ed1a6a756a8c in_seq=0 out_seq=0 2024-03-18T01:39:17.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.732+0000 7f5339600640 10 client.25846 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f5339600640 1 -- 172.21.15.7:0/3895227366 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c11fc80990 con 0x55c11fd245d0 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f5339600640 10 client.25846 handle_command_reply: tid=0 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 client.25846 shutdown 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 -- 172.21.15.7:0/3895227366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0e8c0 msgr2=0x55c11fd18b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0e8c0 0x55c11fd18b60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5340036040 tx=0x7f5340005850 comp rx=0 tx=0).stop 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 -- 172.21.15.7:0/3895227366 shutdown_connections 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55c11fd245d0 0x55c11fd269c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd14420 0x55c11fd200e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0f2b0 0x55c11fd190a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 --2- 172.21.15.7:0/3895227366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0e8c0 0x55c11fd18b60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f5339600640 0 client.25846 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.733+0000 7f53597bd740 1 -- 172.21.15.7:0/3895227366 >> 172.21.15.7:0/3895227366 conn(0x55c11fd09e30 msgr2=0x55c11fd091f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.734+0000 7f53597bd740 1 -- 172.21.15.7:0/3895227366 shutdown_connections 2024-03-18T01:39:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.734+0000 7f53597bd740 1 -- 172.21.15.7:0/3895227366 wait complete. 2024-03-18T01:39:17.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.734+0000 7f53597bd740 20 client.25846 trim_cache size 0 max 16384 2024-03-18T01:39:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.748+0000 7f53597bd740 1 Processor -- start 2024-03-18T01:39:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f53597bd740 1 -- start start 2024-03-18T01:39:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f53597bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f53597bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0f2b0 0x55c11fd200a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f53597bd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd14420 0x55c11fd2b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f53597bd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f53501ac550 con 0x55c11fd14420 2024-03-18T01:39:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f53597bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53501ae0c0 con 0x55c11fd0e8c0 2024-03-18T01:39:17.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f53597bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53501acc20 con 0x55c11fd0f2b0 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f534fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f534f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0f2b0 0x55c11fd200a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f5354a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd14420 0x55c11fd2b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f534f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0f2b0 0x55c11fd200a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38662/0 (socket says 172.21.15.7:38662) 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f534f400640 1 -- 172.21.15.7:0/3479145485 learned_addr learned my addr 172.21.15.7:0/3479145485 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.749+0000 7f534fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41700/0 (socket says 172.21.15.7:41700) 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f534f400640 1 -- 172.21.15.7:0/3479145485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0e8c0 msgr2=0x55c11fd0d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f534f400640 1 --2- 172.21.15.7:0/3479145485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f534f400640 1 -- 172.21.15.7:0/3479145485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd14420 msgr2=0x55c11fd2b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f534f400640 1 --2- 172.21.15.7:0/3479145485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd14420 0x55c11fd2b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f534f400640 1 -- 172.21.15.7:0/3479145485 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c11fd0aba0 con 0x55c11fd0f2b0 2024-03-18T01:39:17.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f534fe00640 1 --2- 172.21.15.7:0/3479145485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:17.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f534f400640 1 --2- 172.21.15.7:0/3479145485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0f2b0 0x55c11fd200a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f533c00c6a0 tx=0x7f533c0cc770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ec1e8479deb649c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f533be00640 1 -- 172.21.15.7:0/3479145485 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c09b330 con 0x55c11fd0f2b0 2024-03-18T01:39:17.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f533be00640 1 -- 172.21.15.7:0/3479145485 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c09b7f0 con 0x55c11fd0f2b0 2024-03-18T01:39:17.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.750+0000 7f533be00640 1 -- 172.21.15.7:0/3479145485 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c099930 con 0x55c11fd0f2b0 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 -- 172.21.15.7:0/3479145485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0f2b0 msgr2=0x55c11fd200a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 --2- 172.21.15.7:0/3479145485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0f2b0 0x55c11fd200a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f533c00c6a0 tx=0x7f533c0cc770 comp rx=0 tx=0).stop 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 -- 172.21.15.7:0/3479145485 shutdown_connections 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 --2- 172.21.15.7:0/3479145485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd14420 0x55c11fd2b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 --2- 172.21.15.7:0/3479145485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd0f2b0 0x55c11fd200a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 --2- 172.21.15.7:0/3479145485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0e8c0 0x55c11fd0d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 -- 172.21.15.7:0/3479145485 >> 172.21.15.7:0/3479145485 conn(0x55c11fd09e30 msgr2=0x55c11fd0d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 -- 172.21.15.7:0/3479145485 shutdown_connections 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.751+0000 7f53597bd740 1 -- 172.21.15.7:0/3479145485 wait complete. 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 1 -- start start 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 1 Processor -- start 2024-03-18T01:39:17.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd1dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd1e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd1f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f53501ad9c0 con 0x55c11fd0e8c0 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f53501ad2f0 con 0x55c11fd0f2b0 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f53501aa530 con 0x55c11fd14420 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.752+0000 7f53597bd740 10 client.0 _ll_register_callbacks cb 0x7f5348009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f534fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd1dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd1f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f534f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd1e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd1f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38664/0 (socket says 172.21.15.7:38664) 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 -- 172.21.15.7:0/2339804291 learned_addr learned my addr 172.21.15.7:0/2339804291 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:17.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 -- 172.21.15.7:0/2339804291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 msgr2=0x55c11fd1e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd1e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 -- 172.21.15.7:0/2339804291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 msgr2=0x55c11fd1dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd1dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 -- 172.21.15.7:0/2339804291 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11fd1f6e0 con 0x55c11fd14420 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f534fe00640 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd1dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5354a00640 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd1f0f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f53480040c0 tx=0x7f5348004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a81e227608298a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.753+0000 7f5339600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f5339600640 1 -- 172.21.15.7:0/2339804291 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5348004b20 con 0x55c11fd14420 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f53597bd740 1 -- 172.21.15.7:0/2339804291 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f533c09b7f0 con 0x55c11fd14420 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f5339600640 1 -- 172.21.15.7:0/2339804291 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5348004cc0 con 0x55c11fd14420 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f5339600640 1 -- 172.21.15.7:0/2339804291 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5348034050 con 0x55c11fd14420 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f53597bd740 10 client.25852 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:17.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f53597bd740 1 -- 172.21.15.7:0/2339804291 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c11fd8ab50 con 0x55c11fd14420 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f5339600640 1 -- 172.21.15.7:0/2339804291 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c09b7f0 con 0x55c11fd14420 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f5339600640 1 -- 172.21.15.7:0/2339804291 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f53480056c0 con 0x55c11fd14420 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f53597bd740 10 client.25852 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f53597bd740 10 client.25852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f53597bd740 20 client.25852 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.754+0000 7f53597bd740 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55c11fd8b5e0 0x55c11fd8da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.755+0000 7f53597bd740 4 client.25852 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.755+0000 7f53597bd740 1 -- 172.21.15.7:0/2339804291 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c11fc7ace0 con 0x55c11fd8b5e0 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.755+0000 7f534fe00640 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55c11fd8b5e0 0x55c11fd8da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.755+0000 7f534fe00640 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55c11fd8b5e0 0x55c11fd8da00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e61e08ad6322c72f server_cookie=11e154b4797b2dab in_seq=0 out_seq=0 2024-03-18T01:39:17.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.755+0000 7f5339600640 10 client.25852 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:17.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.757+0000 7f5339600640 1 -- 172.21.15.7:0/2339804291 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c11fc7ace0 con 0x55c11fd8b5e0 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.757+0000 7f5339600640 10 client.25852 handle_command_reply: tid=0 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.757+0000 7f53597bd740 1 client.25852 shutdown 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 -- 172.21.15.7:0/2339804291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 msgr2=0x55c11fd1f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd1f0f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f53480040c0 tx=0x7f5348004a40 comp rx=0 tx=0).stop 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 -- 172.21.15.7:0/2339804291 shutdown_connections 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55c11fd8b5e0 0x55c11fd8da00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55c11fd14420 0x55c11fd1f0f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f5339600640 0 client.25852 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c11fd0f2b0 0x55c11fd1e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 --2- 172.21.15.7:0/2339804291 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55c11fd0e8c0 0x55c11fd1dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 -- 172.21.15.7:0/2339804291 >> 172.21.15.7:0/2339804291 conn(0x55c11fd09e30 msgr2=0x55c11fd0d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 -- 172.21.15.7:0/2339804291 shutdown_connections 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 1 -- 172.21.15.7:0/2339804291 wait complete. 2024-03-18T01:39:17.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.758+0000 7f53597bd740 20 client.25852 trim_cache size 0 max 16384 2024-03-18T01:39:17.762 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 -- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53340a2880 msgr2=0x7f53340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53340a2880 0x7f53340a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55c11fd0fb10 tx=0x7f5348009500 comp rx=0 tx=0).stop 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 -- 172.21.15.7:0/1294150133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350104bf0 msgr2=0x7f535019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350104bf0 0x7f535019a290 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f533c00e9f0 tx=0x7f533c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 -- 172.21.15.7:0/1294150133 shutdown_connections 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f53340a2880 0x7f53340a4d40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f53501055b0 0x7f53501a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5350104bf0 0x7f535019a290 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 --2- 172.21.15.7:0/1294150133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5350103a00 0x7f5350199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 -- 172.21.15.7:0/1294150133 >> 172.21.15.7:0/1294150133 conn(0x7f53500fef30 msgr2=0x7f5350107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 -- 172.21.15.7:0/1294150133 shutdown_connections 2024-03-18T01:39:17.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:17.759+0000 7f5356a00640 1 -- 172.21.15.7:0/1294150133 wait complete. 2024-03-18T01:39:17.777 INFO:tasks.workunit.client.0.smithi007.stdout:checking for suffix of object files... o 2024-03-18T01:39:17.827 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are using the GNU C compiler... yes 2024-03-18T01:39:17.887 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether gcc accepts -g... yes 2024-03-18T01:39:17.976 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-18T01:39:18.177 INFO:tasks.workunit.client.0.smithi007.stdout:checking dependency style of gcc... gcc3 2024-03-18T01:39:18.290 INFO:tasks.workunit.client.0.smithi007.stdout:checking how to run the C preprocessor... gcc -E 2024-03-18T01:39:18.394 INFO:tasks.workunit.client.0.smithi007.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-18T01:39:18.400 INFO:tasks.workunit.client.0.smithi007.stdout:checking for egrep... /usr/bin/grep -E 2024-03-18T01:39:18.678 INFO:tasks.workunit.client.0.smithi007.stdout:checking for ANSI C header files... yes 2024-03-18T01:39:18.752 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/types.h... yes 2024-03-18T01:39:18.777 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T01:39:18.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:39:18.778 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T01:39:18.836 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/stat.h... yes 2024-03-18T01:39:18.911 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdlib.h... yes 2024-03-18T01:39:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.983+0000 7f9854600640 1 -- 172.21.15.7:0/1377326394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 msgr2=0x7f984c105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.983+0000 7f9854600640 1 --2- 172.21.15.7:0/1377326394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 0x7f984c105090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9838009900 tx=0x7f983802bc30 comp rx=0 tx=0).stop 2024-03-18T01:39:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.984+0000 7f9854600640 1 -- 172.21.15.7:0/1377326394 shutdown_connections 2024-03-18T01:39:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.984+0000 7f9854600640 1 --2- 172.21.15.7:0/1377326394 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f984c1055d0 0x7f984c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.984+0000 7f9854600640 1 --2- 172.21.15.7:0/1377326394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 0x7f984c105090 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:18.984 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.984+0000 7f9854600640 1 --2- 172.21.15.7:0/1377326394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f984c103a20 0x7f984c103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.984+0000 7f9854600640 1 -- 172.21.15.7:0/1377326394 >> 172.21.15.7:0/1377326394 conn(0x7f984c0fef90 msgr2=0x7f984c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.984+0000 7f9854600640 1 -- 172.21.15.7:0/1377326394 shutdown_connections 2024-03-18T01:39:18.985 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.984+0000 7f9854600640 1 -- 172.21.15.7:0/1377326394 wait complete. 2024-03-18T01:39:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9854600640 1 Processor -- start 2024-03-18T01:39:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9854600640 1 -- start start 2024-03-18T01:39:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9854600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f984c103a20 0x7f984c199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:18.986 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9854600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 0x7f984c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9854600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f984c1055d0 0x7f984c1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9854600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f984c10f6d0 con 0x7f984c1055d0 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9854600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f984c10f550 con 0x7f984c104c10 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9854600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f984c10f850 con 0x7f984c103a20 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9851c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f984c103a20 0x7f984c199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9851200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 0x7f984c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9851c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f984c103a20 0x7f984c199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38676/0 (socket says 172.21.15.7:38676) 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9851c00640 1 -- 172.21.15.7:0/943340855 learned_addr learned my addr 172.21.15.7:0/943340855 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.985+0000 7f9851200640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 0x7f984c19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41742/0 (socket says 172.21.15.7:41742) 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.986+0000 7f9852600640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f984c1055d0 0x7f984c1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.986+0000 7f9851200640 1 -- 172.21.15.7:0/943340855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f984c103a20 msgr2=0x7f984c199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.986+0000 7f9851200640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f984c103a20 0x7f984c199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.986+0000 7f9851200640 1 -- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f984c1055d0 msgr2=0x7f984c1a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:18.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.986+0000 7f9851200640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f984c1055d0 0x7f984c1a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.986+0000 7f9851200640 1 -- 172.21.15.7:0/943340855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984c1a1940 con 0x7f984c104c10 2024-03-18T01:39:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.986+0000 7f9852600640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f984c1055d0 0x7f984c1a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.986+0000 7f9851200640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 0x7f984c19a270 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9838036040 tx=0x7f983802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:18.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.987+0000 7f984a000640 1 -- 172.21.15.7:0/943340855 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838043020 con 0x7f984c104c10 2024-03-18T01:39:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.987+0000 7f984a000640 1 -- 172.21.15.7:0/943340855 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98380054e0 con 0x7f984c104c10 2024-03-18T01:39:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.987+0000 7f9854600640 1 -- 172.21.15.7:0/943340855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984c1a1bd0 con 0x7f984c104c10 2024-03-18T01:39:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.987+0000 7f984a000640 1 -- 172.21.15.7:0/943340855 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838003c30 con 0x7f984c104c10 2024-03-18T01:39:18.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.987+0000 7f9854600640 1 -- 172.21.15.7:0/943340855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f984c1a2140 con 0x7f984c104c10 2024-03-18T01:39:18.990 INFO:tasks.workunit.client.0.smithi007.stdout:checking for string.h... yes 2024-03-18T01:39:18.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.990+0000 7f98573ef740 1 Processor -- start 2024-03-18T01:39:18.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f98573ef740 1 -- start start 2024-03-18T01:39:18.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f98573ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5ea7e0 0x560cfd5eac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:18.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f98573ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:18.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f98573ef740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:18.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f98573ef740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x560cfd54df80 con 0x560cfd5f0340 2024-03-18T01:39:18.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f98573ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560cfd4f6ad0 con 0x560cfd5ea7e0 2024-03-18T01:39:18.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f98573ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560cfd54a020 con 0x560cfd5eb1d0 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f984a000640 1 -- 172.21.15.7:0/943340855 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9838046020 con 0x7f984c104c10 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f9852600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5ea7e0 0x560cfd5eac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f9852600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5ea7e0 0x560cfd5eac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41752/0 (socket says 172.21.15.7:41752) 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f9852600640 1 -- 172.21.15.7:0/3968561205 learned_addr learned my addr 172.21.15.7:0/3968561205 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f9851c00640 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f9852600640 1 -- 172.21.15.7:0/3968561205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 msgr2=0x560cfd5efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9851200640 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.991+0000 7f9852600640 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9852600640 1 -- 172.21.15.7:0/3968561205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 msgr2=0x560cfd5f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9852600640 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:18.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9852600640 1 -- 172.21.15.7:0/3968561205 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cfd50e6f0 con 0x560cfd5ea7e0 2024-03-18T01:39:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9851200640 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9852600640 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5ea7e0 0x560cfd5eac00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9844009cc0 tx=0x7f98440117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6b186b487430c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9842a00640 1 -- 172.21.15.7:0/3968561205 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9844012450 con 0x560cfd5ea7e0 2024-03-18T01:39:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9842a00640 1 -- 172.21.15.7:0/3968561205 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9844012ad0 con 0x560cfd5ea7e0 2024-03-18T01:39:18.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.992+0000 7f9842a00640 1 -- 172.21.15.7:0/3968561205 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984400fb80 con 0x560cfd5ea7e0 2024-03-18T01:39:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.993+0000 7f984a000640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f98300a2880 0x7f98300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.993+0000 7f98573ef740 1 -- 172.21.15.7:0/3968561205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5ea7e0 msgr2=0x560cfd5eac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.993+0000 7f98573ef740 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5ea7e0 0x560cfd5eac00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9844009cc0 tx=0x7f98440117b0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f984a000640 1 -- 172.21.15.7:0/943340855 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f98380f0e70 con 0x7f984c104c10 2024-03-18T01:39:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f98573ef740 1 -- 172.21.15.7:0/3968561205 shutdown_connections 2024-03-18T01:39:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f98573ef740 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f98573ef740 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f98573ef740 1 --2- 172.21.15.7:0/3968561205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5ea7e0 0x560cfd5eac00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f98573ef740 1 -- 172.21.15.7:0/3968561205 >> 172.21.15.7:0/3968561205 conn(0x560cfd5b2930 msgr2=0x560cfd5e6150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f9851c00640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f98300a2880 0x7f98300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f98573ef740 1 -- 172.21.15.7:0/3968561205 shutdown_connections 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f98573ef740 1 -- 172.21.15.7:0/3968561205 wait complete. 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.994+0000 7f98573ef740 1 -- start start 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f9851c00640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f98300a2880 0x7f98300a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x560cfd5e70a0 tx=0x7f983c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f98573ef740 1 Processor -- start 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f98573ef740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 0x560cfd5f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f98573ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5f5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f98573ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5f0340 0x560cfd5fc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f98573ef740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x560cfd5f32f0 con 0x560cfd5ea7e0 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f98573ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560cfd509f30 con 0x560cfd5eb1d0 2024-03-18T01:39:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f98573ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560cfd5f3470 con 0x560cfd5f0340 2024-03-18T01:39:19.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f98573ef740 10 client.0 _ll_register_callbacks cb 0x560cfd4f7bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f9851200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5f5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f9851c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 0x560cfd5f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f9852600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5f0340 0x560cfd5fc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.995+0000 7f9851200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5f5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41764/0 (socket says 172.21.15.7:41764) 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.996+0000 7f9851200640 1 -- 172.21.15.7:0/2002126975 learned_addr learned my addr 172.21.15.7:0/2002126975 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.996+0000 7f9852600640 1 -- 172.21.15.7:0/2002126975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 msgr2=0x560cfd5f5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.996+0000 7f9852600640 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5f5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.996+0000 7f9852600640 1 -- 172.21.15.7:0/2002126975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 msgr2=0x560cfd5f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.996+0000 7f9852600640 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 0x560cfd5f4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.996+0000 7f9852600640 1 -- 172.21.15.7:0/2002126975 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cfd32a7e0 con 0x560cfd5f0340 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.996+0000 7f9852600640 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5f0340 0x560cfd5fc070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9844011cb0 tx=0x7f9844009f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fede48a7eb72d021 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.005 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.996+0000 7f9837400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.997+0000 7f9837400640 1 -- 172.21.15.7:0/2002126975 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9844029020 con 0x560cfd5f0340 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.997+0000 7f98573ef740 1 -- 172.21.15.7:0/2002126975 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9844012ad0 con 0x560cfd5f0340 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.997+0000 7f9837400640 1 -- 172.21.15.7:0/2002126975 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f984400fad0 con 0x560cfd5f0340 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.997+0000 7f9837400640 1 -- 172.21.15.7:0/2002126975 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9844034050 con 0x560cfd5f0340 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.997+0000 7f98573ef740 10 client.25864 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.997+0000 7f98573ef740 1 -- 172.21.15.7:0/2002126975 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560cfd38af70 con 0x560cfd5f0340 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f9837400640 1 -- 172.21.15.7:0/2002126975 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9844012ad0 con 0x560cfd5f0340 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f9837400640 1 -- 172.21.15.7:0/2002126975 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f98440046c0 con 0x560cfd5f0340 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f98573ef740 10 client.25864 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f98573ef740 10 client.25864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f98573ef740 20 client.25864 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f98573ef740 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560cfd600630 0x560cfd602a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f98573ef740 4 client.25864 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:19.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f98573ef740 1 -- 172.21.15.7:0/2002126975 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cfd55b4c0 con 0x560cfd600630 2024-03-18T01:39:19.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.998+0000 7f9851c00640 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560cfd600630 0x560cfd602a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.999+0000 7f9851c00640 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560cfd600630 0x560cfd602a20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc8cc52a2998de72 server_cookie=7d01140baf947cf in_seq=0 out_seq=0 2024-03-18T01:39:19.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:18.999+0000 7f9837400640 10 client.25864 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:19.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.001+0000 7f9837400640 1 -- 172.21.15.7:0/2002126975 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560cfd55b4c0 con 0x560cfd600630 2024-03-18T01:39:19.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.001+0000 7f9837400640 10 client.25864 handle_command_reply: tid=0 2024-03-18T01:39:19.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.001+0000 7f98573ef740 1 client.25864 shutdown 2024-03-18T01:39:19.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 -- 172.21.15.7:0/2002126975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5f0340 msgr2=0x560cfd5fc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5f0340 0x560cfd5fc070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9844011cb0 tx=0x7f9844009f20 comp rx=0 tx=0).stop 2024-03-18T01:39:19.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 -- 172.21.15.7:0/2002126975 shutdown_connections 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560cfd600630 0x560cfd602a20 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5f0340 0x560cfd5fc070 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5f5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 --2- 172.21.15.7:0/2002126975 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 0x560cfd5f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f9837400640 0 client.25864 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 -- 172.21.15.7:0/2002126975 >> 172.21.15.7:0/2002126975 conn(0x560cfd5b2930 msgr2=0x560cfd5e5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 -- 172.21.15.7:0/2002126975 shutdown_connections 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 1 -- 172.21.15.7:0/2002126975 wait complete. 2024-03-18T01:39:19.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.002+0000 7f98573ef740 20 client.25864 trim_cache size 0 max 16384 2024-03-18T01:39:19.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f98573ef740 1 Processor -- start 2024-03-18T01:39:19.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f98573ef740 1 -- start start 2024-03-18T01:39:19.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f98573ef740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 0x560cfd5e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f98573ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f98573ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5f0340 0x560cfd5e5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f98573ef740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1ac4b0 con 0x560cfd5ea7e0 2024-03-18T01:39:19.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f98573ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1ae020 con 0x560cfd5f0340 2024-03-18T01:39:19.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f98573ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1acb80 con 0x560cfd5eb1d0 2024-03-18T01:39:19.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f9852600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5f0340 0x560cfd5e5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f9851200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f9851c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 0x560cfd5e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f9851200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38708/0 (socket says 172.21.15.7:38708) 2024-03-18T01:39:19.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f9852600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5f0340 0x560cfd5e5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41772/0 (socket says 172.21.15.7:41772) 2024-03-18T01:39:19.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.022+0000 7f9851200640 1 -- 172.21.15.7:0/196410608 learned_addr learned my addr 172.21.15.7:0/196410608 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9851200640 1 -- 172.21.15.7:0/196410608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5f0340 msgr2=0x560cfd5e5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9851200640 1 --2- 172.21.15.7:0/196410608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5f0340 0x560cfd5e5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9851200640 1 -- 172.21.15.7:0/196410608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 msgr2=0x560cfd5e51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9851200640 1 --2- 172.21.15.7:0/196410608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 0x560cfd5e51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9851200640 1 -- 172.21.15.7:0/196410608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cfd222170 con 0x560cfd5eb1d0 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9852600640 1 --2- 172.21.15.7:0/196410608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5f0340 0x560cfd5e5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9851200640 1 --2- 172.21.15.7:0/196410608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fc030 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f98380b9d90 tx=0x7f98380f32c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=667e7cad5a5e4262 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9842000640 1 -- 172.21.15.7:0/196410608 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98380bcdb0 con 0x560cfd5eb1d0 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9842000640 1 -- 172.21.15.7:0/196410608 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9838005690 con 0x560cfd5eb1d0 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.023+0000 7f9842000640 1 -- 172.21.15.7:0/196410608 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98380bb590 con 0x560cfd5eb1d0 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 -- 172.21.15.7:0/196410608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 msgr2=0x560cfd5fc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 --2- 172.21.15.7:0/196410608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fc030 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f98380b9d90 tx=0x7f98380f32c0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 -- 172.21.15.7:0/196410608 shutdown_connections 2024-03-18T01:39:19.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 --2- 172.21.15.7:0/196410608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5f0340 0x560cfd5e5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 --2- 172.21.15.7:0/196410608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fc030 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 --2- 172.21.15.7:0/196410608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5ea7e0 0x560cfd5e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 -- 172.21.15.7:0/196410608 >> 172.21.15.7:0/196410608 conn(0x560cfd5b2930 msgr2=0x560cfd5e4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 -- 172.21.15.7:0/196410608 shutdown_connections 2024-03-18T01:39:19.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.024+0000 7f98573ef740 1 -- 172.21.15.7:0/196410608 wait complete. 2024-03-18T01:39:19.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 1 -- start start 2024-03-18T01:39:19.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 1 Processor -- start 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5ea7e0 0x560cfd5fb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1ad920 con 0x560cfd5f0340 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1ad250 con 0x560cfd5eb1d0 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1aa490 con 0x560cfd5ea7e0 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.025+0000 7f98573ef740 10 client.0 _ll_register_callbacks cb 0x7f9844009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5ea7e0 0x560cfd5fb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5ea7e0 0x560cfd5fb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38724/0 (socket says 172.21.15.7:38724) 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 -- 172.21.15.7:0/492099586 learned_addr learned my addr 172.21.15.7:0/492099586 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9852600640 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 -- 172.21.15.7:0/492099586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 msgr2=0x560cfd5fbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 -- 172.21.15.7:0/492099586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 msgr2=0x560cfd5f9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 -- 172.21.15.7:0/492099586 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cfd5fa710 con 0x560cfd5ea7e0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9851c00640 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5ea7e0 0x560cfd5fb940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f983c009fd0 tx=0x7f983c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19b6f89c1044d34 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9837400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.026+0000 7f9837400640 1 -- 172.21.15.7:0/492099586 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c004660 con 0x560cfd5ea7e0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f98573ef740 1 -- 172.21.15.7:0/492099586 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560cfd5f48e0 con 0x560cfd5ea7e0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f9837400640 1 -- 172.21.15.7:0/492099586 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f983c018030 con 0x560cfd5ea7e0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f9837400640 1 -- 172.21.15.7:0/492099586 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f983c03a050 con 0x560cfd5ea7e0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f98573ef740 10 client.25870 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f98573ef740 1 -- 172.21.15.7:0/492099586 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560cfd66fce0 con 0x560cfd5ea7e0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f9837400640 1 -- 172.21.15.7:0/492099586 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560cfd5f48e0 con 0x560cfd5ea7e0 2024-03-18T01:39:19.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f9837400640 1 -- 172.21.15.7:0/492099586 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7f983c01fa40 con 0x560cfd5ea7e0 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f98573ef740 10 client.25870 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f98573ef740 10 client.25870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.027+0000 7f98573ef740 20 client.25870 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.028+0000 7f98573ef740 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560cfd6706f0 0x560cfd672b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.028+0000 7f98573ef740 4 client.25870 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.028+0000 7f98573ef740 1 -- 172.21.15.7:0/492099586 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560cfd555810 con 0x560cfd6706f0 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.028+0000 7f9851200640 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560cfd6706f0 0x560cfd672b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.028+0000 7f9851200640 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560cfd6706f0 0x560cfd672b10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd8d04787410ca8c server_cookie=5dc3b47cbfef89e9 in_seq=0 out_seq=0 2024-03-18T01:39:19.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.028+0000 7f9837400640 10 client.25870 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:19.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.029+0000 7f9837400640 1 -- 172.21.15.7:0/492099586 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560cfd555810 con 0x560cfd6706f0 2024-03-18T01:39:19.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.029+0000 7f9837400640 10 client.25870 handle_command_reply: tid=0 2024-03-18T01:39:19.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.029+0000 7f98573ef740 1 client.25870 shutdown 2024-03-18T01:39:19.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 -- 172.21.15.7:0/492099586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5ea7e0 msgr2=0x560cfd5fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5ea7e0 0x560cfd5fb940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f983c009fd0 tx=0x7f983c004ab0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 -- 172.21.15.7:0/492099586 shutdown_connections 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560cfd6706f0 0x560cfd672b10 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560cfd5f0340 0x560cfd5f9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560cfd5eb1d0 0x560cfd5fbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f9837400640 0 client.25870 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 --2- 172.21.15.7:0/492099586 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560cfd5ea7e0 0x560cfd5fb940 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 -- 172.21.15.7:0/492099586 >> 172.21.15.7:0/492099586 conn(0x560cfd5b2930 msgr2=0x560cfd5e5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 -- 172.21.15.7:0/492099586 shutdown_connections 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 1 -- 172.21.15.7:0/492099586 wait complete. 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.030+0000 7f98573ef740 20 client.25870 trim_cache size 0 max 16384 2024-03-18T01:39:19.033 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:39:19.034 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-03-18T01:39:19.034 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "49dbb29a-31e4-466b-8ac7-08fd41527508", 2024-03-18T01:39:19.034 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-03-18T01:39:19.034 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:39:19.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 -- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f98300a2880 msgr2=0x7f98300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f98300a2880 0x7f98300a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x560cfd5e70a0 tx=0x7f983c009500 comp rx=0 tx=0).stop 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 -- 172.21.15.7:0/943340855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 msgr2=0x7f984c19a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 0x7f984c19a270 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9838036040 tx=0x7f983802c650 comp rx=0 tx=0).stop 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 -- 172.21.15.7:0/943340855 shutdown_connections 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f98300a2880 0x7f98300a4d40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f984c1055d0 0x7f984c1a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f984c104c10 0x7f984c19a270 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 --2- 172.21.15.7:0/943340855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f984c103a20 0x7f984c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 -- 172.21.15.7:0/943340855 >> 172.21.15.7:0/943340855 conn(0x7f984c0fef90 msgr2=0x7f984c107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 -- 172.21.15.7:0/943340855 shutdown_connections 2024-03-18T01:39:19.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.031+0000 7f9854600640 1 -- 172.21.15.7:0/943340855 wait complete. 2024-03-18T01:39:19.053 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:39:19.054 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:39:19.085 INFO:tasks.workunit.client.0.smithi007.stdout:checking for memory.h... yes 2024-03-18T01:39:19.136 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:18 smithi007 ceph-mon[31996]: pgmap v451: 129 pgs: 129 active+clean; 3.9 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 324 KiB/s wr, 42 op/s 2024-03-18T01:39:19.179 INFO:tasks.workunit.client.0.smithi007.stdout:checking for strings.h... yes 2024-03-18T01:39:19.266 INFO:tasks.workunit.client.0.smithi007.stdout:checking for inttypes.h... yes 2024-03-18T01:39:19.268 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:18 smithi046 ceph-mon[36108]: pgmap v451: 129 pgs: 129 active+clean; 3.9 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 324 KiB/s wr, 42 op/s 2024-03-18T01:39:19.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 -- 172.21.15.7:0/3428313409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1017f0 msgr2=0x7fb29c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 --2- 172.21.15.7:0/3428313409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1017f0 0x7fb29c101c10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009900 tx=0x7fb28c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:39:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 -- 172.21.15.7:0/3428313409 shutdown_connections 2024-03-18T01:39:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 --2- 172.21.15.7:0/3428313409 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb29c1033a0 0x7fb29c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 --2- 172.21.15.7:0/3428313409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb29c1029e0 0x7fb29c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 --2- 172.21.15.7:0/3428313409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1017f0 0x7fb29c101c10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 -- 172.21.15.7:0/3428313409 >> 172.21.15.7:0/3428313409 conn(0x7fb29c0fcd60 msgr2=0x7fb29c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 -- 172.21.15.7:0/3428313409 shutdown_connections 2024-03-18T01:39:19.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.284+0000 7fb2a1200640 1 -- 172.21.15.7:0/3428313409 wait complete. 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.285+0000 7fb2a1200640 1 Processor -- start 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.285+0000 7fb2a1200640 1 -- start start 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb2a1200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb29c1017f0 0x7fb29c19e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb2a1200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb29c1029e0 0x7fb29c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb2a1200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1033a0 0x7fb29c1a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb2a1200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c075630 con 0x7fb29c1017f0 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb2a1200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0754b0 con 0x7fb29c1029e0 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb2a1200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0757b0 con 0x7fb29c1033a0 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb299400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb29c1029e0 0x7fb29c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb29a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1033a0 0x7fb29c1a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb299e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb29c1017f0 0x7fb29c19e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb299400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb29c1029e0 0x7fb29c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41814/0 (socket says 172.21.15.7:41814) 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb299400640 1 -- 172.21.15.7:0/861577190 learned_addr learned my addr 172.21.15.7:0/861577190 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb29a800640 1 -- 172.21.15.7:0/861577190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb29c1029e0 msgr2=0x7fb29c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb29a800640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb29c1029e0 0x7fb29c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb29a800640 1 -- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb29c1017f0 msgr2=0x7fb29c19e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb29a800640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb29c1017f0 0x7fb29c19e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.286+0000 7fb29a800640 1 -- 172.21.15.7:0/861577190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29c1a5db0 con 0x7fb29c1033a0 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.287+0000 7fb299400640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb29c1029e0 0x7fb29c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:19.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.287+0000 7fb29a800640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1033a0 0x7fb29c1a5740 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb29400da10 tx=0x7fb29400ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.287+0000 7fb292a00640 1 -- 172.21.15.7:0/861577190 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29400bd70 con 0x7fb29c1033a0 2024-03-18T01:39:19.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.287+0000 7fb292a00640 1 -- 172.21.15.7:0/861577190 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb294003e50 con 0x7fb29c1033a0 2024-03-18T01:39:19.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.287+0000 7fb2a1200640 1 -- 172.21.15.7:0/861577190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29c1a60a0 con 0x7fb29c1033a0 2024-03-18T01:39:19.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.287+0000 7fb292a00640 1 -- 172.21.15.7:0/861577190 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2940106a0 con 0x7fb29c1033a0 2024-03-18T01:39:19.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.287+0000 7fb2a1200640 1 -- 172.21.15.7:0/861577190 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29c1a65f0 con 0x7fb29c1033a0 2024-03-18T01:39:19.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.290+0000 7fb2a3df9740 1 Processor -- start 2024-03-18T01:39:19.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.290+0000 7fb292a00640 1 -- 172.21.15.7:0/861577190 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fb294003970 con 0x7fb29c1033a0 2024-03-18T01:39:19.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb2a3df9740 1 -- start start 2024-03-18T01:39:19.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a349020c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349025d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 0x55a349028800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a348f851f0 con 0x55a349026370 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a348f2da80 con 0x55a349021200 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a348f81290 con 0x55a349020810 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb299e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349025d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb299e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349025d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41818/0 (socket says 172.21.15.7:41818) 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb299e00640 1 -- 172.21.15.7:0/3997222585 learned_addr learned my addr 172.21.15.7:0/3997222585 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.291+0000 7fb299400640 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a349020c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb299e00640 1 -- 172.21.15.7:0/3997222585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 msgr2=0x55a349020c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb299e00640 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a349020c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb299e00640 1 -- 172.21.15.7:0/3997222585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 msgr2=0x55a349028800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb299e00640 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 0x55a349028800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb299e00640 1 -- 172.21.15.7:0/3997222585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a348d637d0 con 0x55a349021200 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb299400640 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a349020c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb299e00640 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349025d00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009900 tx=0x7fb28c0095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8948f6370517a943 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb292a00640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2800a2880 0x7fb2800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.292+0000 7fb287400640 1 -- 172.21.15.7:0/3997222585 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c02cb90 con 0x55a349021200 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.293+0000 7fb287400640 1 -- 172.21.15.7:0/3997222585 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28c005040 con 0x55a349021200 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.293+0000 7fb287400640 1 -- 172.21.15.7:0/3997222585 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c004900 con 0x55a349021200 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.293+0000 7fb292a00640 1 -- 172.21.15.7:0/861577190 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb2940cf120 con 0x7fb29c1033a0 2024-03-18T01:39:19.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.293+0000 7fb299400640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2800a2880 0x7fb2800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.293+0000 7fb2a3df9740 1 -- 172.21.15.7:0/3997222585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 msgr2=0x55a349025d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.293+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349025d00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009900 tx=0x7fb28c0095e0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.293+0000 7fb299400640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2800a2880 0x7fb2800a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb288009640 tx=0x7fb288013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.294+0000 7fb2a3df9740 1 -- 172.21.15.7:0/3997222585 shutdown_connections 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.294+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 0x55a349028800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.294+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349025d00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.294+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/3997222585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a349020c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.294+0000 7fb2a3df9740 1 -- 172.21.15.7:0/3997222585 >> 172.21.15.7:0/3997222585 conn(0x55a34901bd80 msgr2=0x55a34901c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.294+0000 7fb2a3df9740 1 -- 172.21.15.7:0/3997222585 shutdown_connections 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.294+0000 7fb2a3df9740 1 -- 172.21.15.7:0/3997222585 wait complete. 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.294+0000 7fb2a3df9740 1 -- start start 2024-03-18T01:39:19.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.295+0000 7fb2a3df9740 1 Processor -- start 2024-03-18T01:39:19.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.295+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a34902ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.295+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.295+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349026370 0x55a349032290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.295+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55a3490292c0 con 0x55a349021200 2024-03-18T01:39:19.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.295+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55a348f40ed0 con 0x55a349026370 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.295+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a349029440 con 0x55a349020810 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.295+0000 7fb2a3df9740 10 client.0 _ll_register_callbacks cb 0x55a348f7c220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb299e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a34902ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb299e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a34902ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38756/0 (socket says 172.21.15.7:38756) 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb299e00640 1 -- 172.21.15.7:0/2532626280 learned_addr learned my addr 172.21.15.7:0/2532626280 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb29a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349026370 0x55a349032290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb299400640 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb29a800640 1 -- 172.21.15.7:0/2532626280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 msgr2=0x55a34902ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb29a800640 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a34902ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb29a800640 1 -- 172.21.15.7:0/2532626280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 msgr2=0x55a34902b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb29a800640 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb29a800640 1 -- 172.21.15.7:0/2532626280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a348c993f0 con 0x55a349026370 2024-03-18T01:39:19.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb299e00640 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a34902ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb29a800640 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349026370 0x55a349032290 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb2940ccc30 tx=0x7fb29409be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecf0e77944199927 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb27be00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.296+0000 7fb27be00640 1 -- 172.21.15.7:0/2532626280 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2940a3f00 con 0x55a349026370 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb27be00640 1 -- 172.21.15.7:0/2532626280 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb2940042e0 con 0x55a349026370 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 1 -- 172.21.15.7:0/2532626280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb28c005040 con 0x55a349026370 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb27be00640 1 -- 172.21.15.7:0/2532626280 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb294003c60 con 0x55a349026370 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 10 client.25907 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 1 -- 172.21.15.7:0/2532626280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a348c590f0 con 0x55a349026370 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb27be00640 1 -- 172.21.15.7:0/2532626280 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28c005040 con 0x55a349026370 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb27be00640 1 -- 172.21.15.7:0/2532626280 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fb294024060 con 0x55a349026370 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 10 client.25907 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 10 client.25907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:19.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 20 client.25907 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55a349036bc0 0x55a349039080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 4 client.25907 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.297+0000 7fb2a3df9740 1 -- 172.21.15.7:0/2532626280 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a348f98be0 con 0x55a349036bc0 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.298+0000 7fb299e00640 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55a349036bc0 0x55a349039080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.298+0000 7fb299e00640 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55a349036bc0 0x55a349039080 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1c7e9fd50a3164 server_cookie=bdabd4b44186a96c in_seq=0 out_seq=0 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.298+0000 7fb27be00640 10 client.25907 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.299+0000 7fb27be00640 1 -- 172.21.15.7:0/2532626280 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a348f98be0 con 0x55a349036bc0 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.299+0000 7fb27be00640 10 client.25907 handle_command_reply: tid=0 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.299+0000 7fb2a3df9740 1 client.25907 shutdown 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 -- 172.21.15.7:0/2532626280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349026370 msgr2=0x55a349032290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349026370 0x55a349032290 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb2940ccc30 tx=0x7fb29409be60 comp rx=0 tx=0).stop 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 -- 172.21.15.7:0/2532626280 shutdown_connections 2024-03-18T01:39:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55a349036bc0 0x55a349039080 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349026370 0x55a349032290 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/2532626280 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a34902ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb27be00640 0 client.25907 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:19.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 -- 172.21.15.7:0/2532626280 >> 172.21.15.7:0/2532626280 conn(0x55a34901bd80 msgr2=0x55a34901aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 -- 172.21.15.7:0/2532626280 shutdown_connections 2024-03-18T01:39:19.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 1 -- 172.21.15.7:0/2532626280 wait complete. 2024-03-18T01:39:19.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.300+0000 7fb2a3df9740 20 client.25907 trim_cache size 0 max 16384 2024-03-18T01:39:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.332+0000 7fb2a3df9740 1 Processor -- start 2024-03-18T01:39:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.332+0000 7fb2a3df9740 1 -- start start 2024-03-18T01:39:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.332+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a3490227a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.332+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349022ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 0x55a349032160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c1b07d0 con 0x55a349026370 2024-03-18T01:39:19.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c1b2340 con 0x55a349021200 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c1b0ea0 con 0x55a349020810 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349022ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a3490227a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349022ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41842/0 (socket says 172.21.15.7:41842) 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299400640 1 -- 172.21.15.7:0/4283708577 learned_addr learned my addr 172.21.15.7:0/4283708577 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb29a800640 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 0x55a349032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299400640 1 -- 172.21.15.7:0/4283708577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 msgr2=0x55a3490227a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299400640 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a3490227a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299400640 1 -- 172.21.15.7:0/4283708577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 msgr2=0x55a349032160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299400640 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 0x55a349032160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299400640 1 -- 172.21.15.7:0/4283708577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3490215e0 con 0x55a349021200 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.333+0000 7fb299e00640 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a3490227a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.334+0000 7fb299400640 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349022ce0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb28800f910 tx=0x7fb28801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd2df66fb469aa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.334+0000 7fb286a00640 1 -- 172.21.15.7:0/4283708577 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28802c070 con 0x55a349021200 2024-03-18T01:39:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.334+0000 7fb286a00640 1 -- 172.21.15.7:0/4283708577 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28801bb80 con 0x55a349021200 2024-03-18T01:39:19.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.334+0000 7fb286a00640 1 -- 172.21.15.7:0/4283708577 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb288026630 con 0x55a349021200 2024-03-18T01:39:19.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 -- 172.21.15.7:0/4283708577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 msgr2=0x55a349022ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349022ce0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb28800f910 tx=0x7fb28801a000 comp rx=0 tx=0).stop 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 -- 172.21.15.7:0/4283708577 shutdown_connections 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349026370 0x55a349032160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349021200 0x55a349022ce0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/4283708577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349020810 0x55a3490227a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 -- 172.21.15.7:0/4283708577 >> 172.21.15.7:0/4283708577 conn(0x55a34901bd80 msgr2=0x55a34901c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 -- 172.21.15.7:0/4283708577 shutdown_connections 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.335+0000 7fb2a3df9740 1 -- 172.21.15.7:0/4283708577 wait complete. 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.336+0000 7fb2a3df9740 1 -- start start 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.336+0000 7fb2a3df9740 1 Processor -- start 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349020810 0x55a34902f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb2a3df9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349026370 0x55a3490308c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c1b1c40 con 0x55a349021200 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c1b1570 con 0x55a349020810 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb2a3df9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c1ae7b0 con 0x55a349026370 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb2a3df9740 10 client.0 _ll_register_callbacks cb 0x7fb294004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb299e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349020810 0x55a34902f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb29a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349026370 0x55a3490308c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb299e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349020810 0x55a34902f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:41848/0 (socket says 172.21.15.7:41848) 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb29a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349026370 0x55a3490308c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38770/0 (socket says 172.21.15.7:38770) 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb299e00640 1 -- 172.21.15.7:0/1006890073 learned_addr learned my addr 172.21.15.7:0/1006890073 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.337+0000 7fb299400640 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb29a800640 1 -- 172.21.15.7:0/1006890073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349020810 msgr2=0x55a34902f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb29a800640 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349020810 0x55a34902f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb29a800640 1 -- 172.21.15.7:0/1006890073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 msgr2=0x55a34902f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:19.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb29a800640 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb29a800640 1 -- 172.21.15.7:0/1006890073 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a349030eb0 con 0x55a349026370 2024-03-18T01:39:19.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb299e00640 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349020810 0x55a34902f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:19.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb299400640 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:19.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb29a800640 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349026370 0x55a3490308c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb294004af0 tx=0x7fb2940046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a9453f895d90de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:19.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb27be00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:19.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb27be00640 1 -- 172.21.15.7:0/1006890073 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2940a3f00 con 0x55a349026370 2024-03-18T01:39:19.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb27be00640 1 -- 172.21.15.7:0/1006890073 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb294097dd0 con 0x55a349026370 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb2a3df9740 1 -- 172.21.15.7:0/1006890073 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb28801bb80 con 0x55a349026370 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb27be00640 1 -- 172.21.15.7:0/1006890073 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb294004b40 con 0x55a349026370 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb2a3df9740 10 client.25888 fetch_fsmap learned FSMap version 29 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.338+0000 7fb2a3df9740 1 -- 172.21.15.7:0/1006890073 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a349032230 con 0x55a349026370 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb27be00640 1 -- 172.21.15.7:0/1006890073 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28801bb80 con 0x55a349026370 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb27be00640 1 -- 172.21.15.7:0/1006890073 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8366+0+0 (secure 0 0 0) 0x7fb294004ce0 con 0x55a349026370 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb2a3df9740 10 client.25888 fetch_fsmap finished waiting for FSMap version 29 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb2a3df9740 10 client.25888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb2a3df9740 20 client.25888 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55a34909d2f0 0x55a34909f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb2a3df9740 4 client.25888 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb2a3df9740 1 -- 172.21.15.7:0/1006890073 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a348f92730 con 0x55a34909d2f0 2024-03-18T01:39:19.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.339+0000 7fb299e00640 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55a34909d2f0 0x55a34909f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb299e00640 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55a34909d2f0 0x55a34909f710 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db6505199cd3390d server_cookie=c8c7d1b44a151878 in_seq=0 out_seq=0 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb27be00640 10 client.25888 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb27be00640 1 -- 172.21.15.7:0/1006890073 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a348f92730 con 0x55a34909d2f0 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb27be00640 10 client.25888 handle_command_reply: tid=0 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 client.25888 shutdown 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 -- 172.21.15.7:0/1006890073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349026370 msgr2=0x55a3490308c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349026370 0x55a3490308c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb294004af0 tx=0x7fb2940046e0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 -- 172.21.15.7:0/1006890073 shutdown_connections 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55a34909d2f0 0x55a34909f710 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb27be00640 0 client.25888 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a349026370 0x55a3490308c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55a349021200 0x55a34902f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 --2- 172.21.15.7:0/1006890073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a349020810 0x55a34902f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.340+0000 7fb2a3df9740 1 -- 172.21.15.7:0/1006890073 >> 172.21.15.7:0/1006890073 conn(0x55a34901bd80 msgr2=0x55a34902ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.341+0000 7fb2a3df9740 1 -- 172.21.15.7:0/1006890073 shutdown_connections 2024-03-18T01:39:19.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.341+0000 7fb2a3df9740 1 -- 172.21.15.7:0/1006890073 wait complete. 2024-03-18T01:39:19.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.341+0000 7fb2a3df9740 20 client.25888 trim_cache size 0 max 16384 2024-03-18T01:39:19.347 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:39:19.347 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T01:39:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:39:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "49dbb29a-31e4-466b-8ac7-08fd41527508": { 2024-03-18T01:39:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:39:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "49dbb29a-31e4-466b-8ac7-08fd41527508", 2024-03-18T01:39:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:39:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:39:19.348 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:39:19.348 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:39:19.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.341+0000 7fb2a1200640 1 -- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2800a2880 msgr2=0x7fb2800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.341+0000 7fb2a1200640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2800a2880 0x7fb2800a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb288009640 tx=0x7fb288013040 comp rx=0 tx=0).stop 2024-03-18T01:39:19.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.341+0000 7fb2a1200640 1 -- 172.21.15.7:0/861577190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1033a0 msgr2=0x7fb29c1a5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:19.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.341+0000 7fb2a1200640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1033a0 0x7fb29c1a5740 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb29400da10 tx=0x7fb29400ded0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.342+0000 7fb2a1200640 1 -- 172.21.15.7:0/861577190 shutdown_connections 2024-03-18T01:39:19.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.342+0000 7fb2a1200640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb29c1033a0 0x7fb29c1a5740 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.342+0000 7fb2a1200640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb29c1029e0 0x7fb29c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.342+0000 7fb2a1200640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2800a2880 0x7fb2800a4d40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.342+0000 7fb2a1200640 1 --2- 172.21.15.7:0/861577190 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb29c1017f0 0x7fb29c19e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:19.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.342+0000 7fb2a1200640 1 -- 172.21.15.7:0/861577190 >> 172.21.15.7:0/861577190 conn(0x7fb29c0fcd60 msgr2=0x7fb29c10ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:19.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.342+0000 7fb2a1200640 1 -- 172.21.15.7:0/861577190 shutdown_connections 2024-03-18T01:39:19.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:19.342+0000 7fb2a1200640 1 -- 172.21.15.7:0/861577190 wait complete. 2024-03-18T01:39:19.353 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdint.h... yes 2024-03-18T01:39:19.360 INFO:tasks.cephfs.filesystem:scrub status for tag:49dbb29a-31e4-466b-8ac7-08fd41527508 - {'path': '/', 'tag': '49dbb29a-31e4-466b-8ac7-08fd41527508', 'options': 'recursive,force'} 2024-03-18T01:39:19.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:18 smithi203 ceph-mon[36345]: pgmap v451: 129 pgs: 129 active+clean; 3.9 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 324 KiB/s wr, 42 op/s 2024-03-18T01:39:19.442 INFO:tasks.workunit.client.0.smithi007.stdout:checking for unistd.h... yes 2024-03-18T01:39:19.526 INFO:tasks.workunit.client.0.smithi007.stdout:checking minix/config.h usability... no 2024-03-18T01:39:19.582 INFO:tasks.workunit.client.0.smithi007.stdout:checking minix/config.h presence... no 2024-03-18T01:39:19.582 INFO:tasks.workunit.client.0.smithi007.stdout:checking for minix/config.h... no 2024-03-18T01:39:19.665 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-03-18T01:39:19.666 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc... (cached) gcc 2024-03-18T01:39:19.746 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-18T01:39:19.746 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-18T01:39:19.746 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-18T01:39:19.747 INFO:tasks.workunit.client.0.smithi007.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-18T01:39:19.850 INFO:tasks.workunit.client.0.smithi007.stdout:checking for main in -lm... yes 2024-03-18T01:39:19.951 INFO:tasks.workunit.client.0.smithi007.stdout:checking for main in -lpthread... yes 2024-03-18T01:39:19.955 INFO:tasks.workunit.client.0.smithi007.stdout:checking for ANSI C header files... (cached) yes 2024-03-18T01:39:20.027 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-18T01:39:20.145 INFO:tasks.workunit.client.0.smithi007.stdout:checking pthread.h usability... yes 2024-03-18T01:39:20.194 INFO:tasks.workunit.client.0.smithi007.stdout:checking pthread.h presence... yes 2024-03-18T01:39:20.194 INFO:tasks.workunit.client.0.smithi007.stdout:checking for pthread.h... yes 2024-03-18T01:39:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:19 smithi046 ceph-mon[36108]: scrub queued for path: / 2024-03-18T01:39:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:19 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:19 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:20.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:19 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:20.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:19 smithi007 ceph-mon[31996]: scrub queued for path: / 2024-03-18T01:39:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:19 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:19 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:20.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:19 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:20.305 INFO:tasks.workunit.client.0.smithi007.stdout:checking fcntl.h usability... yes 2024-03-18T01:39:20.351 INFO:tasks.workunit.client.0.smithi007.stdout:checking fcntl.h presence... yes 2024-03-18T01:39:20.352 INFO:tasks.workunit.client.0.smithi007.stdout:checking for fcntl.h... yes 2024-03-18T01:39:20.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:19 smithi203 ceph-mon[36345]: scrub queued for path: / 2024-03-18T01:39:20.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:19 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:20.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:19 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:20.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:19 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:20.470 INFO:tasks.workunit.client.0.smithi007.stdout:checking limits.h usability... yes 2024-03-18T01:39:20.503 INFO:tasks.workunit.client.0.smithi007.stdout:checking limits.h presence... yes 2024-03-18T01:39:20.503 INFO:tasks.workunit.client.0.smithi007.stdout:checking for limits.h... yes 2024-03-18T01:39:20.537 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stdint.h... (cached) yes 2024-03-18T01:39:20.623 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/time.h usability... yes 2024-03-18T01:39:20.668 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/time.h presence... yes 2024-03-18T01:39:20.668 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/time.h... yes 2024-03-18T01:39:20.695 INFO:tasks.workunit.client.0.smithi007.stdout:checking for unistd.h... (cached) yes 2024-03-18T01:39:20.776 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/vfs.h usability... yes 2024-03-18T01:39:20.818 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/vfs.h presence... yes 2024-03-18T01:39:20.818 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/vfs.h... yes 2024-03-18T01:39:20.935 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/limits.h usability... no 2024-03-18T01:39:21.010 INFO:tasks.workunit.client.0.smithi007.stdout:checking sys/limits.h presence... no 2024-03-18T01:39:21.010 INFO:tasks.workunit.client.0.smithi007.stdout:checking for sys/limits.h... no 2024-03-18T01:39:21.061 INFO:tasks.workunit.client.0.smithi007.stdout:checking for an ANSI C-conforming const... yes 2024-03-18T01:39:21.127 INFO:tasks.workunit.client.0.smithi007.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-18T01:39:21.239 INFO:tasks.workunit.client.0.smithi007.stdout:checking for system... yes 2024-03-18T01:39:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.273 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:20 smithi046 ceph-mon[36108]: pgmap v452: 129 pgs: 129 active+clean; 5.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 461 KiB/s wr, 55 op/s 2024-03-18T01:39:21.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.305 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:21.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:20 smithi007 ceph-mon[31996]: pgmap v452: 129 pgs: 129 active+clean; 5.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 461 KiB/s wr, 55 op/s 2024-03-18T01:39:21.365 INFO:tasks.workunit.client.0.smithi007.stdout:checking for gettimeofday... yes 2024-03-18T01:39:21.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:21.394 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:20 smithi203 ceph-mon[36345]: pgmap v452: 129 pgs: 129 active+clean; 5.4 MiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 461 KiB/s wr, 55 op/s 2024-03-18T01:39:21.516 INFO:tasks.workunit.client.0.smithi007.stdout:checking for mkdir... yes 2024-03-18T01:39:21.642 INFO:tasks.workunit.client.0.smithi007.stdout:checking for strerror... yes 2024-03-18T01:39:21.772 INFO:tasks.workunit.client.0.smithi007.stdout:checking for open64... yes 2024-03-18T01:39:21.902 INFO:tasks.workunit.client.0.smithi007.stdout:checking for stat64... yes 2024-03-18T01:39:22.027 INFO:tasks.workunit.client.0.smithi007.stdout:checking for fseeko64... yes 2024-03-18T01:39:22.158 INFO:tasks.workunit.client.0.smithi007.stdout:checking for lrand48_r... yes 2024-03-18T01:39:22.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:21 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:22.284 INFO:tasks.workunit.client.0.smithi007.stdout:checking for srand48_r... yes 2024-03-18T01:39:22.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:21 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:22.332 INFO:tasks.workunit.client.0.smithi007.stdout:configure: creating ./config.status 2024-03-18T01:39:22.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:21 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:22.482 INFO:tasks.workunit.client.0.smithi007.stdout: /bin/sh ./config.status 2024-03-18T01:39:22.629 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating Makefile 2024-03-18T01:39:22.672 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating config.h 2024-03-18T01:39:22.720 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: executing depfiles commands 2024-03-18T01:39:22.772 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: executing default commands 2024-03-18T01:39:22.813 INFO:tasks.workunit.client.0.smithi007.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-03-18T01:39:23.121 INFO:tasks.workunit.client.0.smithi007.stdout:rm -f stamp-h1 2024-03-18T01:39:23.124 INFO:tasks.workunit.client.0.smithi007.stdout:touch config.h.in 2024-03-18T01:39:23.129 INFO:tasks.workunit.client.0.smithi007.stdout:cd . && /bin/sh ./config.status config.h 2024-03-18T01:39:23.238 INFO:tasks.workunit.client.0.smithi007.stdout:config.status: creating config.h 2024-03-18T01:39:23.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:22 smithi046 ceph-mon[36108]: pgmap v453: 129 pgs: 129 active+clean; 5.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 342 KiB/s wr, 50 op/s 2024-03-18T01:39:23.295 INFO:tasks.workunit.client.0.smithi007.stdout:make all-am 2024-03-18T01:39:23.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:22 smithi007 ceph-mon[31996]: pgmap v453: 129 pgs: 129 active+clean; 5.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 342 KiB/s wr, 50 op/s 2024-03-18T01:39:23.304 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-18T01:39:23.304 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-03-18T01:39:23.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:22 smithi203 ceph-mon[36345]: pgmap v453: 129 pgs: 129 active+clean; 5.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 342 KiB/s wr, 50 op/s 2024-03-18T01:39:23.576 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-03-18T01:39:23.583 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-03-18T01:39:23.725 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-03-18T01:39:23.732 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-03-18T01:39:23.902 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-03-18T01:39:23.909 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-03-18T01:39:24.049 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-03-18T01:39:24.056 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-03-18T01:39:24.245 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-03-18T01:39:24.252 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-03-18T01:39:24.346 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-03-18T01:39:24.348 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-03-18T01:39:24.424 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-03-18T01:39:24.429 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-03-18T01:39:24.520 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-03-18T01:39:24.524 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-03-18T01:39:24.803 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-03-18T01:39:24.811 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-03-18T01:39:24.999 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-03-18T01:39:25.004 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-03-18T01:39:25.224 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-03-18T01:39:25.234 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-03-18T01:39:25.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:24 smithi046 ceph-mon[36108]: pgmap v454: 129 pgs: 129 active+clean; 5.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 347 KiB/s wr, 48 op/s 2024-03-18T01:39:25.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:24 smithi007 ceph-mon[31996]: pgmap v454: 129 pgs: 129 active+clean; 5.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 347 KiB/s wr, 48 op/s 2024-03-18T01:39:25.382 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-03-18T01:39:25.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:24 smithi203 ceph-mon[36345]: pgmap v454: 129 pgs: 129 active+clean; 5.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 347 KiB/s wr, 48 op/s 2024-03-18T01:39:25.394 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-03-18T01:39:25.503 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-03-18T01:39:25.510 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-03-18T01:39:25.672 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-03-18T01:39:25.679 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-03-18T01:39:25.737 INFO:tasks.workunit.client.0.smithi007.stderr:In file included from parser.h:22, 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr: from parser.c:25: 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c: In function ‘init_filesys’: 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2024-03-18T01:39:25.738 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T01:39:25.739 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T01:39:25.740 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T01:39:25.741 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-18T01:39:25.742 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.743 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c: In function ‘init_tg_stats’: 2024-03-18T01:39:25.743 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T01:39:25.743 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T01:39:25.743 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T01:39:25.744 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T01:39:25.745 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2024-03-18T01:39:25.746 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: | ^ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T01:39:25.747 INFO:tasks.workunit.client.0.smithi007.stderr: 36 | container_of(ptr, type, member) 2024-03-18T01:39:25.748 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~ 2024-03-18T01:39:25.748 INFO:tasks.workunit.client.0.smithi007.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T01:39:25.748 INFO:tasks.workunit.client.0.smithi007.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T01:39:25.748 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~ 2024-03-18T01:39:25.748 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T01:39:25.748 INFO:tasks.workunit.client.0.smithi007.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-18T01:39:25.748 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.795 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c: In function ‘get_optstr’: 2024-03-18T01:39:25.796 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-03-18T01:39:25.796 INFO:tasks.workunit.client.0.smithi007.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-03-18T01:39:25.796 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:25.796 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:200:14: note: source object declared here 2024-03-18T01:39:25.796 INFO:tasks.workunit.client.0.smithi007.stderr: 200 | char temp[BUFSIZE]; 2024-03-18T01:39:25.796 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~ 2024-03-18T01:39:25.803 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c: In function ‘size64_convert’: 2024-03-18T01:39:25.803 INFO:tasks.workunit.client.0.smithi007.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-03-18T01:39:25.803 INFO:tasks.workunit.client.0.smithi007.stderr: 119 | memcpy(unit, "\0", 3); 2024-03-18T01:39:25.803 INFO:tasks.workunit.client.0.smithi007.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-03-18T01:39:26.238 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-03-18T01:39:26.245 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-03-18T01:39:26.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:25 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:39:26.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:25 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:39:26.354 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-03-18T01:39:26.360 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-03-18T01:39:26.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:25 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:39:26.489 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-03-18T01:39:26.504 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-03-18T01:39:26.570 INFO:tasks.workunit.client.0.smithi007.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-03-18T01:39:26.582 INFO:tasks.workunit.client.0.smithi007.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-03-18T01:39:26.654 INFO:tasks.workunit.client.0.smithi007.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-18T01:39:26.655 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd .. 2024-03-18T01:39:26.655 INFO:tasks.workunit.client.0.smithi007.stderr:+ mkdir tmp 2024-03-18T01:39:26.661 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd tmp 2024-03-18T01:39:26.661 INFO:tasks.workunit.client.0.smithi007.stderr:+ for f in $mydir/*.ffsb 2024-03-18T01:39:26.661 INFO:tasks.workunit.client.0.smithi007.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-18T01:39:26.663 INFO:tasks.workunit.client.0.smithi007.stdout:FFSB version 6.0-RC2 started 2024-03-18T01:39:26.663 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout:benchmark time = 300 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout:ThreadGroup 0 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout:================ 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout: num_threads = 32 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout: read_random = off 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout: read_size = 0 (0B) 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout: read_blocksize = 0 (0B) 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout: read_skip = off 2024-03-18T01:39:26.672 INFO:tasks.workunit.client.0.smithi007.stdout: read_skipsize = 0 (0B) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: write_random = on 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: write_size = 5242880 (5MB) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: fsync_file = 0 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: write_blocksize = 4096 (4KB) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: wait time = 0 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: op weights 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: read = 0 (0.00%) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: readall = 0 (0.00%) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: write = 1 (100.00%) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: create = 0 (0.00%) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: append = 0 (0.00%) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: delete = 0 (0.00%) 2024-03-18T01:39:26.673 INFO:tasks.workunit.client.0.smithi007.stdout: metaop = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: createdir = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: stat = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: writeall = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: writeall_fsync = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: open_close = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: write_fsync = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: create_fsync = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: append_fsync = 0 (0.00%) 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout:FileSystem . 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout:========== 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: num_dirs = 0 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: starting files = 128 2024-03-18T01:39:26.674 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: min file size = 104857600 (100MB) 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: max file size = 104857600 (100MB) 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: directio = off 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: alignedio = on 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: bufferedio = off 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: aging is off 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: current utilization = 0.03% 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:39:26.675 INFO:tasks.workunit.client.0.smithi007.stdout:checking existing fs: . 2024-03-18T01:39:26.676 INFO:tasks.workunit.client.0.smithi007.stdout:recreating new fileset 2024-03-18T01:39:26.676 INFO:tasks.workunit.client.0.smithi007.stderr:opendir: No such file or directory 2024-03-18T01:39:27.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:26 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:39:27.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:26 smithi046 ceph-mon[36108]: pgmap v455: 129 pgs: 129 active+clean; 7.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 361 KiB/s wr, 51 op/s 2024-03-18T01:39:27.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:26 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:39:27.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:26 smithi007 ceph-mon[31996]: pgmap v455: 129 pgs: 129 active+clean; 7.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 361 KiB/s wr, 51 op/s 2024-03-18T01:39:27.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:26 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:39:27.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:26 smithi203 ceph-mon[36345]: pgmap v455: 129 pgs: 129 active+clean; 7.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 361 KiB/s wr, 51 op/s 2024-03-18T01:39:29.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:28 smithi046 ceph-mon[36108]: pgmap v456: 129 pgs: 129 active+clean; 7.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 339 KiB/s wr, 44 op/s 2024-03-18T01:39:29.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:28 smithi007 ceph-mon[31996]: pgmap v456: 129 pgs: 129 active+clean; 7.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 339 KiB/s wr, 44 op/s 2024-03-18T01:39:29.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:28 smithi203 ceph-mon[36345]: pgmap v456: 129 pgs: 129 active+clean; 7.5 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 339 KiB/s wr, 44 op/s 2024-03-18T01:39:31.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:30 smithi046 ceph-mon[36108]: pgmap v457: 129 pgs: 129 active+clean; 9.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 491 KiB/s wr, 62 op/s 2024-03-18T01:39:31.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:30 smithi007 ceph-mon[31996]: pgmap v457: 129 pgs: 129 active+clean; 9.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 491 KiB/s wr, 62 op/s 2024-03-18T01:39:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:30 smithi203 ceph-mon[36345]: pgmap v457: 129 pgs: 129 active+clean; 9.0 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 491 KiB/s wr, 62 op/s 2024-03-18T01:39:33.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:32 smithi046 ceph-mon[36108]: pgmap v458: 129 pgs: 129 active+clean; 9.6 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 472 KiB/s wr, 78 op/s 2024-03-18T01:39:33.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:32 smithi007 ceph-mon[31996]: pgmap v458: 129 pgs: 129 active+clean; 9.6 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 472 KiB/s wr, 78 op/s 2024-03-18T01:39:33.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:32 smithi203 ceph-mon[36345]: pgmap v458: 129 pgs: 129 active+clean; 9.6 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 472 KiB/s wr, 78 op/s 2024-03-18T01:39:35.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:34 smithi046 ceph-mon[36108]: pgmap v459: 129 pgs: 129 active+clean; 50 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 3.8 MiB/s wr, 78 op/s 2024-03-18T01:39:35.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:34 smithi007 ceph-mon[31996]: pgmap v459: 129 pgs: 129 active+clean; 50 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 3.8 MiB/s wr, 78 op/s 2024-03-18T01:39:35.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:34 smithi203 ceph-mon[36345]: pgmap v459: 129 pgs: 129 active+clean; 50 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 3.8 MiB/s wr, 78 op/s 2024-03-18T01:39:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:37 smithi046 ceph-mon[36108]: pgmap v460: 129 pgs: 129 active+clean; 91 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 7.2 MiB/s wr, 91 op/s 2024-03-18T01:39:37.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:37 smithi007 ceph-mon[31996]: pgmap v460: 129 pgs: 129 active+clean; 91 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 7.2 MiB/s wr, 91 op/s 2024-03-18T01:39:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:37 smithi203 ceph-mon[36345]: pgmap v460: 129 pgs: 129 active+clean; 91 MiB data, 546 MiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 7.2 MiB/s wr, 91 op/s 2024-03-18T01:39:39.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:39 smithi046 ceph-mon[36108]: pgmap v461: 129 pgs: 129 active+clean; 195 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 16 MiB/s wr, 83 op/s 2024-03-18T01:39:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:39 smithi007 ceph-mon[31996]: pgmap v461: 129 pgs: 129 active+clean; 195 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 16 MiB/s wr, 83 op/s 2024-03-18T01:39:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:39 smithi203 ceph-mon[36345]: pgmap v461: 129 pgs: 129 active+clean; 195 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 16 MiB/s wr, 83 op/s 2024-03-18T01:39:41.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:41 smithi007 ceph-mon[31996]: pgmap v462: 129 pgs: 129 active+clean; 591 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 49 MiB/s wr, 98 op/s 2024-03-18T01:39:41.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:41 smithi203 ceph-mon[36345]: pgmap v462: 129 pgs: 129 active+clean; 591 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 49 MiB/s wr, 98 op/s 2024-03-18T01:39:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:41 smithi046 ceph-mon[36108]: pgmap v462: 129 pgs: 129 active+clean; 591 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 49 MiB/s wr, 98 op/s 2024-03-18T01:39:43.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:43 smithi007 ceph-mon[31996]: pgmap v463: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 113 MiB/s wr, 94 op/s 2024-03-18T01:39:43.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:43 smithi203 ceph-mon[36345]: pgmap v463: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 113 MiB/s wr, 94 op/s 2024-03-18T01:39:43.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:43 smithi046 ceph-mon[36108]: pgmap v463: 129 pgs: 129 active+clean; 1.3 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 113 MiB/s wr, 94 op/s 2024-03-18T01:39:44.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:44 smithi203 ceph-mon[36345]: pgmap v464: 129 pgs: 129 active+clean; 1.6 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 140 MiB/s wr, 73 op/s 2024-03-18T01:39:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:44 smithi046 ceph-mon[36108]: pgmap v464: 129 pgs: 129 active+clean; 1.6 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 140 MiB/s wr, 73 op/s 2024-03-18T01:39:44.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:44 smithi007 ceph-mon[31996]: pgmap v464: 129 pgs: 129 active+clean; 1.6 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 140 MiB/s wr, 73 op/s 2024-03-18T01:39:46.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:46 smithi203 ceph-mon[36345]: pgmap v465: 129 pgs: 129 active+clean; 2.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 183 MiB/s wr, 91 op/s 2024-03-18T01:39:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:46 smithi046 ceph-mon[36108]: pgmap v465: 129 pgs: 129 active+clean; 2.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 183 MiB/s wr, 91 op/s 2024-03-18T01:39:46.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:46 smithi007 ceph-mon[31996]: pgmap v465: 129 pgs: 129 active+clean; 2.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 183 MiB/s wr, 91 op/s 2024-03-18T01:39:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:48 smithi203 ceph-mon[36345]: pgmap v466: 129 pgs: 129 active+clean; 2.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 215 MiB/s wr, 87 op/s 2024-03-18T01:39:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:48 smithi046 ceph-mon[36108]: pgmap v466: 129 pgs: 129 active+clean; 2.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 215 MiB/s wr, 87 op/s 2024-03-18T01:39:48.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:48 smithi007 ceph-mon[31996]: pgmap v466: 129 pgs: 129 active+clean; 2.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 215 MiB/s wr, 87 op/s 2024-03-18T01:39:49.361 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:39:49.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:39:49.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.685+0000 7f7638c00640 1 -- 172.21.15.7:0/3774683666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f763410db00 msgr2=0x7f763410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:49.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.685+0000 7f7638c00640 1 --2- 172.21.15.7:0/3774683666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f763410db00 0x7f763410df80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f762c009be0 tx=0x7f762c02bf30 comp rx=0 tx=0).stop 2024-03-18T01:39:49.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.686+0000 7f7638c00640 1 -- 172.21.15.7:0/3774683666 shutdown_connections 2024-03-18T01:39:49.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.686+0000 7f7638c00640 1 --2- 172.21.15.7:0/3774683666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f763410db00 0x7f763410df80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.686+0000 7f7638c00640 1 --2- 172.21.15.7:0/3774683666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f763410b150 0x7f763410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.686+0000 7f7638c00640 1 --2- 172.21.15.7:0/3774683666 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f76341087e0 0x7f763410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.686+0000 7f7638c00640 1 -- 172.21.15.7:0/3774683666 >> 172.21.15.7:0/3774683666 conn(0x7f763406e780 msgr2=0x7f7634070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:49.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.686+0000 7f7638c00640 1 -- 172.21.15.7:0/3774683666 shutdown_connections 2024-03-18T01:39:49.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.686+0000 7f7638c00640 1 -- 172.21.15.7:0/3774683666 wait complete. 2024-03-18T01:39:49.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.689+0000 7f7638c00640 1 Processor -- start 2024-03-18T01:39:49.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.689+0000 7f7638c00640 1 -- start start 2024-03-18T01:39:49.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.691+0000 7f7638c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76341087e0 0x7f763419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.691 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.691+0000 7f7638c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f763410b150 0x7f763419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.692+0000 7f7638c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f763410db00 0x7f76341a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.692+0000 7f7638c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7634111a90 con 0x7f763410b150 2024-03-18T01:39:49.693 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.692+0000 7f7638c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7634111910 con 0x7f76341087e0 2024-03-18T01:39:49.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.692+0000 7f7638c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7634111c10 con 0x7f763410db00 2024-03-18T01:39:49.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.692+0000 7f7631400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f763410b150 0x7f763419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.694 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.692+0000 7f7631400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f763410b150 0x7f763419e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49028/0 (socket says 172.21.15.7:49028) 2024-03-18T01:39:49.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.692+0000 7f7631400640 1 -- 172.21.15.7:0/1213318230 learned_addr learned my addr 172.21.15.7:0/1213318230 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:49.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.693+0000 7f7631400640 1 -- 172.21.15.7:0/1213318230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f763410db00 msgr2=0x7f76341a5570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:39:49.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.693+0000 7f7631400640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f763410db00 0x7f76341a5570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.693+0000 7f7631400640 1 -- 172.21.15.7:0/1213318230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76341087e0 msgr2=0x7f763419dfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:39:49.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.693+0000 7f7631400640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76341087e0 0x7f763419dfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.695 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.693+0000 7f7631400640 1 -- 172.21.15.7:0/1213318230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76341a5be0 con 0x7f763410b150 2024-03-18T01:39:49.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.697+0000 7f7631400640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f763410b150 0x7f763419e500 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f762000d950 tx=0x7f762000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:49.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.698+0000 7f762aa00640 1 -- 172.21.15.7:0/1213318230 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620017030 con 0x7f763410b150 2024-03-18T01:39:49.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.698+0000 7f762aa00640 1 -- 172.21.15.7:0/1213318230 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7620004020 con 0x7f763410b150 2024-03-18T01:39:49.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.698+0000 7f762aa00640 1 -- 172.21.15.7:0/1213318230 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620003c20 con 0x7f763410b150 2024-03-18T01:39:49.698 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.698+0000 7f7638c00640 1 -- 172.21.15.7:0/1213318230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76341a5ed0 con 0x7f763410b150 2024-03-18T01:39:49.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.699+0000 7f7638c00640 1 -- 172.21.15.7:0/1213318230 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76341a6340 con 0x7f763410b150 2024-03-18T01:39:49.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.709+0000 7f763b994740 1 Processor -- start 2024-03-18T01:39:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.710+0000 7f763b994740 1 -- start start 2024-03-18T01:39:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.711+0000 7f763b994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 0x5560fa09ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.711+0000 7f763b994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.711+0000 7f763b994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 0x5560fa0a6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.711+0000 7f763b994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5560fa0031f0 con 0x5560fa09e810 2024-03-18T01:39:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.711+0000 7f763b994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5560f9faba80 con 0x5560fa09f200 2024-03-18T01:39:49.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.711+0000 7f763b994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5560f9fff290 con 0x5560fa0a4370 2024-03-18T01:39:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.712+0000 7f762aa00640 1 -- 172.21.15.7:0/1213318230 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f762001a070 con 0x7f763410b150 2024-03-18T01:39:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.714+0000 7f7631e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.714+0000 7f7631e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0a3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54864/0 (socket says 172.21.15.7:54864) 2024-03-18T01:39:49.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.714+0000 7f7631e00640 1 -- 172.21.15.7:0/2106627485 learned_addr learned my addr 172.21.15.7:0/2106627485 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.715+0000 7f7631e00640 1 -- 172.21.15.7:0/2106627485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 msgr2=0x5560fa0a6800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:39:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.715+0000 7f7631e00640 1 --2- 172.21.15.7:0/2106627485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 0x5560fa0a6800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.715+0000 7f7631e00640 1 -- 172.21.15.7:0/2106627485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 msgr2=0x5560fa09ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.715+0000 7f7631e00640 1 --2- 172.21.15.7:0/2106627485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 0x5560fa09ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.715+0000 7f7631e00640 1 -- 172.21.15.7:0/2106627485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560f9de17d0 con 0x5560fa09f200 2024-03-18T01:39:49.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.717+0000 7f7631e00640 1 --2- 172.21.15.7:0/2106627485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0a3d00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f762401ab90 tx=0x7f762401f6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2104dfb009e05e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:49.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.718+0000 7f761f000640 1 -- 172.21.15.7:0/2106627485 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624027070 con 0x5560fa09f200 2024-03-18T01:39:49.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.718+0000 7f762aa00640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f76180a2990 0x7f76180a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.718+0000 7f762aa00640 1 -- 172.21.15.7:0/1213318230 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f76200ce810 con 0x7f763410b150 2024-03-18T01:39:49.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.719+0000 7f761f000640 1 -- 172.21.15.7:0/2106627485 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7624005500 con 0x5560fa09f200 2024-03-18T01:39:49.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.719+0000 7f761f000640 1 -- 172.21.15.7:0/2106627485 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624022480 con 0x5560fa09f200 2024-03-18T01:39:49.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.720+0000 7f7632800640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f76180a2990 0x7f76180a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.722+0000 7f763b994740 1 -- 172.21.15.7:0/2106627485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 msgr2=0x5560fa0a3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:49.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.723+0000 7f763b994740 1 --2- 172.21.15.7:0/2106627485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0a3d00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f762401ab90 tx=0x7f762401f6d0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.723+0000 7f7632800640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f76180a2990 0x7f76180a4e50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f762c02c540 tx=0x7f762c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:49.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.723+0000 7f763b994740 1 -- 172.21.15.7:0/2106627485 shutdown_connections 2024-03-18T01:39:49.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.724+0000 7f763b994740 1 --2- 172.21.15.7:0/2106627485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 0x5560fa0a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.724+0000 7f763b994740 1 --2- 172.21.15.7:0/2106627485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0a3d00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.724+0000 7f763b994740 1 --2- 172.21.15.7:0/2106627485 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 0x5560fa09ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.724+0000 7f763b994740 1 -- 172.21.15.7:0/2106627485 >> 172.21.15.7:0/2106627485 conn(0x5560fa099d80 msgr2=0x5560fa09a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:49.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.724+0000 7f763b994740 1 -- 172.21.15.7:0/2106627485 shutdown_connections 2024-03-18T01:39:49.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.724+0000 7f763b994740 1 -- 172.21.15.7:0/2106627485 wait complete. 2024-03-18T01:39:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.725+0000 7f763b994740 1 -- start start 2024-03-18T01:39:49.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.725+0000 7f763b994740 1 Processor -- start 2024-03-18T01:39:49.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.727+0000 7f763b994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09e810 0x5560fa0a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.727+0000 7f763b994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09f200 0x5560fa0a9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.727+0000 7f763b994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa0a4370 0x5560fa0b0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.728+0000 7f7631e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09e810 0x5560fa0a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.728+0000 7f7631e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09e810 0x5560fa0a8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44968/0 (socket says 172.21.15.7:44968) 2024-03-18T01:39:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.728+0000 7f7632800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa0a4370 0x5560fa0b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.728+0000 7f7632800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa0a4370 0x5560fa0b0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54876/0 (socket says 172.21.15.7:54876) 2024-03-18T01:39:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.728+0000 7f7631400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09f200 0x5560fa0a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.729+0000 7f7631400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09f200 0x5560fa0a9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49050/0 (socket says 172.21.15.7:49050) 2024-03-18T01:39:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.729+0000 7f763b994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5560fa0a7320 con 0x5560fa09f200 2024-03-18T01:39:49.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.729+0000 7f763b994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5560f9fbeed0 con 0x5560fa0a4370 2024-03-18T01:39:49.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.729+0000 7f763b994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5560fa0a74a0 con 0x5560fa09e810 2024-03-18T01:39:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.729+0000 7f7631e00640 1 -- 172.21.15.7:0/460114904 learned_addr learned my addr 172.21.15.7:0/460114904 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.730+0000 7f7632800640 1 -- 172.21.15.7:0/460114904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09e810 msgr2=0x5560fa0a8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.730+0000 7f7632800640 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09e810 0x5560fa0a8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.730+0000 7f7632800640 1 -- 172.21.15.7:0/460114904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09f200 msgr2=0x5560fa0a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:49.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.730+0000 7f7632800640 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09f200 0x5560fa0a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.730+0000 7f7632800640 1 -- 172.21.15.7:0/460114904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560f9d173f0 con 0x5560fa0a4370 2024-03-18T01:39:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.731+0000 7f763b994740 10 client.0 _ll_register_callbacks cb 0x5560f9ffa220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:49.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.731+0000 7f763b994740 1 -- 172.21.15.7:0/460114904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7624005500 con 0x5560fa0a4370 2024-03-18T01:39:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.732+0000 7f7632800640 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa0a4370 0x5560fa0b0360 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f762c036040 tx=0x7f762c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53ced06b0e71e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.732+0000 7f7613400640 10 client.25922 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.732+0000 7f7613400640 1 -- 172.21.15.7:0/460114904 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c046040 con 0x5560fa0a4370 2024-03-18T01:39:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.732+0000 7f7613400640 1 -- 172.21.15.7:0/460114904 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f762c02cd60 con 0x5560fa0a4370 2024-03-18T01:39:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.733+0000 7f7613400640 1 -- 172.21.15.7:0/460114904 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f762c033e70 con 0x5560fa0a4370 2024-03-18T01:39:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.733+0000 7f763b994740 10 client.25922 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:49.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.733+0000 7f763b994740 1 -- 172.21.15.7:0/460114904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5560f9cd70f0 con 0x5560fa0a4370 2024-03-18T01:39:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.733+0000 7f7613400640 1 -- 172.21.15.7:0/460114904 <== mon.2 v2:172.21.15.46:3300/0 4 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f762c034900 con 0x5560fa0a4370 2024-03-18T01:39:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.734+0000 7f763b994740 10 client.25922 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.734+0000 7f763b994740 10 client.25922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.734+0000 7f763b994740 20 client.25922 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.734+0000 7f763b994740 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5560fa0b4950 0x5560fa0b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.734+0000 7f763b994740 4 client.25922 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:49.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.734+0000 7f763b994740 1 -- 172.21.15.7:0/460114904 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560fa016be0 con 0x5560fa0b4950 2024-03-18T01:39:49.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.734+0000 7f7631e00640 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5560fa0b4950 0x5560fa0b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.735+0000 7f7613400640 10 client.25922 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:49.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.735+0000 7f7631e00640 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5560fa0b4950 0x5560fa0b6df0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23e6925b5d8b876d server_cookie=153e0beae4090a5f in_seq=0 out_seq=0 2024-03-18T01:39:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.736+0000 7f7613400640 1 -- 172.21.15.7:0/460114904 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5560fa016be0 con 0x5560fa0b4950 2024-03-18T01:39:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.736+0000 7f7613400640 10 client.25922 handle_command_reply: tid=0 2024-03-18T01:39:49.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.736+0000 7f763b994740 1 client.25922 shutdown 2024-03-18T01:39:49.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.737+0000 7f763b994740 1 -- 172.21.15.7:0/460114904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa0a4370 msgr2=0x5560fa0b0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:49.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.737+0000 7f763b994740 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa0a4370 0x5560fa0b0360 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f762c036040 tx=0x7f762c009500 comp rx=0 tx=0).stop 2024-03-18T01:39:49.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.738+0000 7f763b994740 1 -- 172.21.15.7:0/460114904 shutdown_connections 2024-03-18T01:39:49.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.738+0000 7f763b994740 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5560fa0b4950 0x5560fa0b6df0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.738+0000 7f763b994740 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa0a4370 0x5560fa0b0360 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.738+0000 7f763b994740 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09f200 0x5560fa0a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.738+0000 7f763b994740 1 --2- 172.21.15.7:0/460114904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09e810 0x5560fa0a8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.738+0000 7f763b994740 1 -- 172.21.15.7:0/460114904 >> 172.21.15.7:0/460114904 conn(0x5560fa099d80 msgr2=0x5560fa098ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:49.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.738+0000 7f7613400640 0 client.25922 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:49.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.738+0000 7f763b994740 1 -- 172.21.15.7:0/460114904 shutdown_connections 2024-03-18T01:39:49.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.739+0000 7f763b994740 1 -- 172.21.15.7:0/460114904 wait complete. 2024-03-18T01:39:49.740 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.740+0000 7f763b994740 20 client.25922 trim_cache size 0 max 16384 2024-03-18T01:39:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.825+0000 7f763b994740 1 Processor -- start 2024-03-18T01:39:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.825+0000 7f763b994740 1 -- start start 2024-03-18T01:39:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.825+0000 7f763b994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09e810 0x5560fa098d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.825+0000 7f763b994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09f200 0x5560fa0b0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.825+0000 7f763b994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa0a4370 0x5560fa0bb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.825+0000 7f763b994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b0880 con 0x5560fa0a4370 2024-03-18T01:39:49.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.825+0000 7f763b994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b2420 con 0x5560fa09e810 2024-03-18T01:39:49.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.825+0000 7f763b994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b0f50 con 0x5560fa09f200 2024-03-18T01:39:49.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.827+0000 7f7631e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09e810 0x5560fa098d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.827+0000 7f7631400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09f200 0x5560fa0b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.827+0000 7f7631400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09f200 0x5560fa0b0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44982/0 (socket says 172.21.15.7:44982) 2024-03-18T01:39:49.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.827+0000 7f7631400640 1 -- 172.21.15.7:0/4227478491 learned_addr learned my addr 172.21.15.7:0/4227478491 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:49.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.827+0000 7f7632800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa0a4370 0x5560fa0bb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.828+0000 7f7631400640 1 -- 172.21.15.7:0/4227478491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09e810 msgr2=0x5560fa098d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:49.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.828+0000 7f7631400640 1 --2- 172.21.15.7:0/4227478491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09e810 0x5560fa098d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.828+0000 7f7631400640 1 -- 172.21.15.7:0/4227478491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa0a4370 msgr2=0x5560fa0bb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:49.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.828+0000 7f7631400640 1 --2- 172.21.15.7:0/4227478491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa0a4370 0x5560fa0bb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.828+0000 7f7631400640 1 -- 172.21.15.7:0/4227478491 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560fa094370 con 0x5560fa09f200 2024-03-18T01:39:49.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.829+0000 7f7631400640 1 --2- 172.21.15.7:0/4227478491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09f200 0x5560fa0b0280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f762009ade0 tx=0x7f762009ad50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8077c22a6b2e724 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:49.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.829+0000 7f761e600640 1 -- 172.21.15.7:0/4227478491 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200a35f0 con 0x5560fa09f200 2024-03-18T01:39:49.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.829+0000 7f761e600640 1 -- 172.21.15.7:0/4227478491 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f762009d070 con 0x5560fa09f200 2024-03-18T01:39:49.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.829+0000 7f761e600640 1 -- 172.21.15.7:0/4227478491 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200a3160 con 0x5560fa09f200 2024-03-18T01:39:49.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.830+0000 7f763b994740 1 -- 172.21.15.7:0/4227478491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09f200 msgr2=0x5560fa0b0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:49.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.830+0000 7f763b994740 1 --2- 172.21.15.7:0/4227478491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09f200 0x5560fa0b0280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f762009ade0 tx=0x7f762009ad50 comp rx=0 tx=0).stop 2024-03-18T01:39:49.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.831+0000 7f763b994740 1 -- 172.21.15.7:0/4227478491 shutdown_connections 2024-03-18T01:39:49.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.831+0000 7f763b994740 1 --2- 172.21.15.7:0/4227478491 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa0a4370 0x5560fa0bb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.831+0000 7f763b994740 1 --2- 172.21.15.7:0/4227478491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa09f200 0x5560fa0b0280 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.831+0000 7f763b994740 1 --2- 172.21.15.7:0/4227478491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09e810 0x5560fa098d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.831+0000 7f763b994740 1 -- 172.21.15.7:0/4227478491 >> 172.21.15.7:0/4227478491 conn(0x5560fa099d80 msgr2=0x5560fa098630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:49.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.831+0000 7f763b994740 1 -- 172.21.15.7:0/4227478491 shutdown_connections 2024-03-18T01:39:49.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.831+0000 7f763b994740 1 -- 172.21.15.7:0/4227478491 wait complete. 2024-03-18T01:39:49.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.832+0000 7f763b994740 1 -- start start 2024-03-18T01:39:49.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.832+0000 7f763b994740 1 Processor -- start 2024-03-18T01:39:49.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.833+0000 7f763b994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 0x5560fa0afa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.834+0000 7f763b994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0aff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.834+0000 7f763b994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 0x5560fa0ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.834+0000 7f763b994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b1cf0 con 0x5560fa09e810 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f763b994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f76341b1620 con 0x5560fa09f200 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f7632800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 0x5560fa0ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f7632800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 0x5560fa0ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44996/0 (socket says 172.21.15.7:44996) 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f763b994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f76341ae860 con 0x5560fa0a4370 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f763b994740 10 client.0 _ll_register_callbacks cb 0x7f762c043040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f7631e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 0x5560fa0afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f7631e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 0x5560fa0afa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49068/0 (socket says 172.21.15.7:49068) 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f7631e00640 1 -- 172.21.15.7:0/2667618341 learned_addr learned my addr 172.21.15.7:0/2667618341 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:49.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.835+0000 7f7631400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.837+0000 7f7631400640 1 -- 172.21.15.7:0/2667618341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 msgr2=0x5560fa0ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.837+0000 7f7631400640 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 0x5560fa0ae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.837+0000 7f7631400640 1 -- 172.21.15.7:0/2667618341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 msgr2=0x5560fa0afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:49.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.837+0000 7f7631400640 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 0x5560fa0afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.837+0000 7f7631400640 1 -- 172.21.15.7:0/2667618341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560fa0ae6f0 con 0x5560fa09f200 2024-03-18T01:39:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.838+0000 7f7631400640 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0aff90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f762009ade0 tx=0x7f7620097530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4bfb3f6f8a7a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.838+0000 7f7613400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.838+0000 7f7613400640 1 -- 172.21.15.7:0/2667618341 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200b9070 con 0x5560fa09f200 2024-03-18T01:39:49.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.838+0000 7f7613400640 1 -- 172.21.15.7:0/2667618341 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7620004240 con 0x5560fa09f200 2024-03-18T01:39:49.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.838+0000 7f763b994740 1 -- 172.21.15.7:0/2667618341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f762009d070 con 0x5560fa09f200 2024-03-18T01:39:49.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.839+0000 7f7613400640 1 -- 172.21.15.7:0/2667618341 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f76200a3160 con 0x5560fa09f200 2024-03-18T01:39:49.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.841+0000 7f763b994740 10 client.25928 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:49.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.841+0000 7f763b994740 1 -- 172.21.15.7:0/2667618341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5560fa11b680 con 0x5560fa09f200 2024-03-18T01:39:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f7613400640 1 -- 172.21.15.7:0/2667618341 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76200a8070 con 0x5560fa09f200 2024-03-18T01:39:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f7613400640 1 -- 172.21.15.7:0/2667618341 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7620024060 con 0x5560fa09f200 2024-03-18T01:39:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f763b994740 10 client.25928 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f763b994740 10 client.25928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f763b994740 20 client.25928 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:49.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f763b994740 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5560fa11c020 0x5560fa11e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:49.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f763b994740 4 client.25928 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:39:49.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f763b994740 1 -- 172.21.15.7:0/2667618341 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f762009d070 con 0x5560fa11c020 2024-03-18T01:39:49.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.842+0000 7f7631e00640 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5560fa11c020 0x5560fa11e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.844+0000 7f7631e00640 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5560fa11c020 0x5560fa11e440 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7ee515dfe8a5f36 server_cookie=6f65e6c3d88cbb2b in_seq=0 out_seq=0 2024-03-18T01:39:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.845+0000 7f7613400640 10 client.25928 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.845+0000 7f7613400640 1 -- 172.21.15.7:0/2667618341 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f762009d070 con 0x5560fa11c020 2024-03-18T01:39:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.845+0000 7f7613400640 10 client.25928 handle_command_reply: tid=0 2024-03-18T01:39:49.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.845+0000 7f763b994740 1 client.25928 shutdown 2024-03-18T01:39:49.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.845+0000 7f763b994740 1 -- 172.21.15.7:0/2667618341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 msgr2=0x5560fa0aff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:49.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.845+0000 7f763b994740 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0aff90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f762009ade0 tx=0x7f7620097530 comp rx=0 tx=0).stop 2024-03-18T01:39:49.846 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.845+0000 7f763b994740 1 -- 172.21.15.7:0/2667618341 shutdown_connections 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.845+0000 7f763b994740 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5560fa11c020 0x5560fa11e440 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.846+0000 7f763b994740 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5560fa0a4370 0x5560fa0ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.846+0000 7f763b994740 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5560fa09f200 0x5560fa0aff90 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f762009ade0 tx=0x7f7620097530 comp rx=0 tx=0).stop 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.846+0000 7f763b994740 1 --2- 172.21.15.7:0/2667618341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5560fa09e810 0x5560fa0afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.846+0000 7f763b994740 1 -- 172.21.15.7:0/2667618341 >> 172.21.15.7:0/2667618341 conn(0x5560fa099d80 msgr2=0x5560fa098ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.846+0000 7f7613400640 0 client.25928 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.846+0000 7f763b994740 1 -- 172.21.15.7:0/2667618341 shutdown_connections 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.846+0000 7f763b994740 1 -- 172.21.15.7:0/2667618341 wait complete. 2024-03-18T01:39:49.847 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.846+0000 7f763b994740 20 client.25928 trim_cache size 0 max 16384 2024-03-18T01:39:49.848 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:39:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-03-18T01:39:49.848 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-03-18T01:39:49.848 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:39:49.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.852+0000 7f7638c00640 1 -- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f76180a2990 msgr2=0x7f76180a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:49.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.852+0000 7f7638c00640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f76180a2990 0x7f76180a4e50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f762c02c540 tx=0x7f762c02e040 comp rx=0 tx=0).stop 2024-03-18T01:39:49.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.852+0000 7f7638c00640 1 -- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f763410b150 msgr2=0x7f763419e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:49.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.852+0000 7f7638c00640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f763410b150 0x7f763419e500 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f762000d950 tx=0x7f762000de10 comp rx=0 tx=0).stop 2024-03-18T01:39:49.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.853+0000 7f7638c00640 1 -- 172.21.15.7:0/1213318230 shutdown_connections 2024-03-18T01:39:49.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.853+0000 7f7638c00640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f763410db00 0x7f76341a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.853+0000 7f7638c00640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f76180a2990 0x7f76180a4e50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.853+0000 7f7638c00640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f763410b150 0x7f763419e500 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.853+0000 7f7638c00640 1 --2- 172.21.15.7:0/1213318230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76341087e0 0x7f763419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:49.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.853+0000 7f7638c00640 1 -- 172.21.15.7:0/1213318230 >> 172.21.15.7:0/1213318230 conn(0x7f763406e780 msgr2=0x7f763410b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:49.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.853+0000 7f7638c00640 1 -- 172.21.15.7:0/1213318230 shutdown_connections 2024-03-18T01:39:49.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:49.853+0000 7f7638c00640 1 -- 172.21.15.7:0/1213318230 wait complete. 2024-03-18T01:39:49.885 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T01:39:49.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:39:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 -- 172.21.15.7:0/2414210599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c01029d0 msgr2=0x7fe7c0102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 --2- 172.21.15.7:0/2414210599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c01029d0 0x7fe7c0102e50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009900 tx=0x7fe7b002bc60 comp rx=0 tx=0).stop 2024-03-18T01:39:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 -- 172.21.15.7:0/2414210599 shutdown_connections 2024-03-18T01:39:50.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 --2- 172.21.15.7:0/2414210599 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c0103390 0x7fe7c0109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 --2- 172.21.15.7:0/2414210599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c01029d0 0x7fe7c0102e50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 --2- 172.21.15.7:0/2414210599 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7c01017e0 0x7fe7c0101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 -- 172.21.15.7:0/2414210599 >> 172.21.15.7:0/2414210599 conn(0x7fe7c00fcd50 msgr2=0x7fe7c00ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:50.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 -- 172.21.15.7:0/2414210599 shutdown_connections 2024-03-18T01:39:50.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.236+0000 7fe7c5800640 1 -- 172.21.15.7:0/2414210599 wait complete. 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.237+0000 7fe7c5800640 1 Processor -- start 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.238+0000 7fe7c5800640 1 -- start start 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.238+0000 7fe7c5800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7c01017e0 0x7fe7c0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.238+0000 7fe7bea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7c01017e0 0x7fe7c0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7bea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7c01017e0 0x7fe7c0199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54898/0 (socket says 172.21.15.7:54898) 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7c5800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c01029d0 0x7fe7c019a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7bea00640 1 -- 172.21.15.7:0/509475889 learned_addr learned my addr 172.21.15.7:0/509475889 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7c5800640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c0103390 0x7fe7c01a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c010d490 con 0x7fe7c01029d0 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c010d310 con 0x7fe7c01017e0 2024-03-18T01:39:50.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c010d610 con 0x7fe7c0103390 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7bf400640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c0103390 0x7fe7c01a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7be000640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c01029d0 0x7fe7c019a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7bf400640 1 -- 172.21.15.7:0/509475889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7c01017e0 msgr2=0x7fe7c0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7bf400640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7c01017e0 0x7fe7c0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7bf400640 1 -- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c01029d0 msgr2=0x7fe7c019a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7bf400640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c01029d0 0x7fe7c019a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.239+0000 7fe7bf400640 1 -- 172.21.15.7:0/509475889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c01a18d0 con 0x7fe7c0103390 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.240+0000 7fe7bea00640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7c01017e0 0x7fe7c0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.240+0000 7fe7bf400640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c0103390 0x7fe7c01a1230 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800e940 tx=0x7fe7b800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:50.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.240+0000 7fe7b7400640 1 -- 172.21.15.7:0/509475889 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8004d20 con 0x7fe7c0103390 2024-03-18T01:39:50.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.240+0000 7fe7b7400640 1 -- 172.21.15.7:0/509475889 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b8005020 con 0x7fe7c0103390 2024-03-18T01:39:50.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.240+0000 7fe7b7400640 1 -- 172.21.15.7:0/509475889 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80116d0 con 0x7fe7c0103390 2024-03-18T01:39:50.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.240+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c01a1bc0 con 0x7fe7c0103390 2024-03-18T01:39:50.242 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.241+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c01a2090 con 0x7fe7c0103390 2024-03-18T01:39:50.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.243+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c0067dd0 con 0x7fe7c0103390 2024-03-18T01:39:50.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.246+0000 7fe7b7400640 1 -- 172.21.15.7:0/509475889 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fe7b80054d0 con 0x7fe7c0103390 2024-03-18T01:39:50.249 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.249+0000 7fe7b7400640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7a40a2880 0x7fe7a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:50.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.249+0000 7fe7b7400640 1 -- 172.21.15.7:0/509475889 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe7b80cf1f0 con 0x7fe7c0103390 2024-03-18T01:39:50.250 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.250+0000 7fe7bea00640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7a40a2880 0x7fe7a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:50.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.251+0000 7fe7bea00640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7a40a2880 0x7fe7a40a4d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe7c019ab60 tx=0x7fe7a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:50.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.254+0000 7fe7b7400640 1 -- 172.21.15.7:0/509475889 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe7b8016030 con 0x7fe7c0103390 2024-03-18T01:39:50.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.740+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7c01a2320 con 0x7fe7c0103390 2024-03-18T01:39:50.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.743+0000 7fe7b7400640 1 -- 172.21.15.7:0/509475889 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11921 (secure 0 0 0) 0x7fe7b809bfb0 con 0x7fe7c0103390 2024-03-18T01:39:50.744 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:39:50.745 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:39:20.242716+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24596,24562,24542,24523,24544]},"id":1}]} 2024-03-18T01:39:50.745 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 30 2024-03-18T01:39:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.747+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7a40a2880 msgr2=0x7fe7a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.747+0000 7fe7c5800640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7a40a2880 0x7fe7a40a4d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe7c019ab60 tx=0x7fe7a8005f90 comp rx=0 tx=0).stop 2024-03-18T01:39:50.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.747+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c0103390 msgr2=0x7fe7c01a1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.747+0000 7fe7c5800640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c0103390 0x7fe7c01a1230 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800e940 tx=0x7fe7b800ee00 comp rx=0 tx=0).stop 2024-03-18T01:39:50.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.748+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 shutdown_connections 2024-03-18T01:39:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.748+0000 7fe7c5800640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7c0103390 0x7fe7c01a1230 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.749+0000 7fe7c5800640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fe7a40a2880 0x7fe7a40a4d40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.749+0000 7fe7c5800640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fe7c01029d0 0x7fe7c019a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.749+0000 7fe7c5800640 1 --2- 172.21.15.7:0/509475889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7c01017e0 0x7fe7c0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.749+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 >> 172.21.15.7:0/509475889 conn(0x7fe7c00fcd50 msgr2=0x7fe7c0073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.749+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 shutdown_connections 2024-03-18T01:39:50.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:50.749+0000 7fe7c5800640 1 -- 172.21.15.7:0/509475889 wait complete. 2024-03-18T01:39:50.768 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:50 smithi046 ceph-mon[36108]: pgmap v467: 129 pgs: 129 active+clean; 3.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 242 MiB/s wr, 102 op/s 2024-03-18T01:39:50.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:50 smithi007 ceph-mon[31996]: pgmap v467: 129 pgs: 129 active+clean; 3.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 242 MiB/s wr, 102 op/s 2024-03-18T01:39:50.785 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T01:39:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:50 smithi203 ceph-mon[36345]: pgmap v467: 129 pgs: 129 active+clean; 3.0 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 242 MiB/s wr, 102 op/s 2024-03-18T01:39:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.139+0000 7f5f4f200640 1 -- 172.21.15.7:0/1247103867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f481029d0 msgr2=0x7f5f48102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.139+0000 7f5f4f200640 1 --2- 172.21.15.7:0/1247103867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f481029d0 0x7f5f48102e50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5f38013080 tx=0x7f5f38033f30 comp rx=0 tx=0).stop 2024-03-18T01:39:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.140+0000 7f5f4f200640 1 -- 172.21.15.7:0/1247103867 shutdown_connections 2024-03-18T01:39:51.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.140+0000 7f5f4f200640 1 --2- 172.21.15.7:0/1247103867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f48103390 0x7f5f48109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.140+0000 7f5f4f200640 1 --2- 172.21.15.7:0/1247103867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f481029d0 0x7f5f48102e50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.140+0000 7f5f4f200640 1 --2- 172.21.15.7:0/1247103867 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f481017e0 0x7f5f48101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.140+0000 7f5f4f200640 1 -- 172.21.15.7:0/1247103867 >> 172.21.15.7:0/1247103867 conn(0x7f5f480fcd30 msgr2=0x7f5f480ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.140+0000 7f5f4f200640 1 -- 172.21.15.7:0/1247103867 shutdown_connections 2024-03-18T01:39:51.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.140+0000 7f5f4f200640 1 -- 172.21.15.7:0/1247103867 wait complete. 2024-03-18T01:39:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.142+0000 7f5f4f200640 1 Processor -- start 2024-03-18T01:39:51.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.142+0000 7f5f4f200640 1 -- start start 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.142+0000 7f5f4f200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f481017e0 0x7f5f480701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.142+0000 7f5f4f200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f481029d0 0x7f5f48070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f47e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f481017e0 0x7f5f480701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f47e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f481017e0 0x7f5f480701c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45024/0 (socket says 172.21.15.7:45024) 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f47400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f481029d0 0x7f5f48070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f47400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f481029d0 0x7f5f48070700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49102/0 (socket says 172.21.15.7:49102) 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f4f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f48103390 0x7f5f481a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f4f200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4810b1b0 con 0x7f5f481029d0 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f4f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4810b030 con 0x7f5f48103390 2024-03-18T01:39:51.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f4f200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4810b330 con 0x7f5f481017e0 2024-03-18T01:39:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.143+0000 7f5f47e00640 1 -- 172.21.15.7:0/3605955006 learned_addr learned my addr 172.21.15.7:0/3605955006 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.144+0000 7f5f4d200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f48103390 0x7f5f481a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.144+0000 7f5f47e00640 1 -- 172.21.15.7:0/3605955006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f48103390 msgr2=0x7f5f481a5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.144+0000 7f5f47e00640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f48103390 0x7f5f481a5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.144+0000 7f5f47e00640 1 -- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f481029d0 msgr2=0x7f5f48070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.144+0000 7f5f47e00640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f481029d0 0x7f5f48070700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.144+0000 7f5f47e00640 1 -- 172.21.15.7:0/3605955006 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f481a5dc0 con 0x7f5f481017e0 2024-03-18T01:39:51.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.145+0000 7f5f47400640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f481029d0 0x7f5f48070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.145+0000 7f5f47e00640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f481017e0 0x7f5f480701c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000ad10 tx=0x7f5f40007590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.145+0000 7f5f44c00640 1 -- 172.21.15.7:0/3605955006 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4001a070 con 0x7f5f481017e0 2024-03-18T01:39:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.145+0000 7f5f44c00640 1 -- 172.21.15.7:0/3605955006 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f40003b70 con 0x7f5f481017e0 2024-03-18T01:39:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.145+0000 7f5f4f200640 1 -- 172.21.15.7:0/3605955006 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f481a5f90 con 0x7f5f481017e0 2024-03-18T01:39:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.145+0000 7f5f44c00640 1 -- 172.21.15.7:0/3605955006 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40015660 con 0x7f5f481017e0 2024-03-18T01:39:51.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.145+0000 7f5f4f200640 1 -- 172.21.15.7:0/3605955006 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f481a6340 con 0x7f5f481017e0 2024-03-18T01:39:51.152 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.150+0000 7f5f44c00640 1 -- 172.21.15.7:0/3605955006 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f5f4001d070 con 0x7f5f481017e0 2024-03-18T01:39:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.152+0000 7f5f51f63740 1 Processor -- start 2024-03-18T01:39:51.153 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.152+0000 7f5f51f63740 1 -- start start 2024-03-18T01:39:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.153+0000 7f5f44c00640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5f2c0a2880 0x7f5f2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.153+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 0x557c8c729ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.153+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72a2b0 0x557c8c72edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.153+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7318b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.153+0000 7f5f51f63740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557c8c6f13b0 con 0x557c8c72a2b0 2024-03-18T01:39:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.153+0000 7f5f51f63740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c8c68e450 con 0x557c8c72f420 2024-03-18T01:39:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.153+0000 7f5f51f63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557c8c636cb0 con 0x557c8c7298c0 2024-03-18T01:39:51.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.153+0000 7f5f44c00640 1 -- 172.21.15.7:0/3605955006 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5f400ce830 con 0x7f5f481017e0 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.154+0000 7f5f4d200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.154+0000 7f5f4d200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7318b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54942/0 (socket says 172.21.15.7:54942) 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.154+0000 7f5f4d200640 1 -- 172.21.15.7:0/729103794 learned_addr learned my addr 172.21.15.7:0/729103794 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.154+0000 7f5f47e00640 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 0x557c8c729ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.154+0000 7f5f47400640 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72a2b0 0x557c8c72edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.154+0000 7f5f4d200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5f2c0a2880 0x7f5f2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.155+0000 7f5f4d200640 1 -- 172.21.15.7:0/729103794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 msgr2=0x557c8c729ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.155+0000 7f5f4d200640 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 0x557c8c729ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.155+0000 7f5f4d200640 1 -- 172.21.15.7:0/729103794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72a2b0 msgr2=0x557c8c72edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.155+0000 7f5f4d200640 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72a2b0 0x557c8c72edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.155+0000 7f5f4d200640 1 -- 172.21.15.7:0/729103794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c8c6581f0 con 0x557c8c72f420 2024-03-18T01:39:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.155+0000 7f5f47400640 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72a2b0 0x557c8c72edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.155+0000 7f5f4d200640 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7318b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5f34012710 tx=0x7f5f34012bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1888aec808f338a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.156+0000 7f5f3d600640 1 -- 172.21.15.7:0/729103794 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f34021040 con 0x557c8c72f420 2024-03-18T01:39:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.156+0000 7f5f3d600640 1 -- 172.21.15.7:0/729103794 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f34022020 con 0x557c8c72f420 2024-03-18T01:39:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.156+0000 7f5f3d600640 1 -- 172.21.15.7:0/729103794 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3400e890 con 0x557c8c72f420 2024-03-18T01:39:51.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.156+0000 7f5f4d200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5f2c0a2880 0x7f5f2c0a4d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5f34023000 tx=0x7f5f34020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.157+0000 7f5f51f63740 1 -- 172.21.15.7:0/729103794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 msgr2=0x557c8c7318b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.157+0000 7f5f51f63740 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7318b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5f34012710 tx=0x7f5f34012bd0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.158+0000 7f5f51f63740 1 -- 172.21.15.7:0/729103794 shutdown_connections 2024-03-18T01:39:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.158+0000 7f5f51f63740 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7318b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.158+0000 7f5f51f63740 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72a2b0 0x557c8c72edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.158+0000 7f5f51f63740 1 --2- 172.21.15.7:0/729103794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 0x557c8c729ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.158+0000 7f5f51f63740 1 -- 172.21.15.7:0/729103794 >> 172.21.15.7:0/729103794 conn(0x557c8c724e30 msgr2=0x557c8c725230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.158+0000 7f5f51f63740 1 -- 172.21.15.7:0/729103794 shutdown_connections 2024-03-18T01:39:51.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.158+0000 7f5f51f63740 1 -- 172.21.15.7:0/729103794 wait complete. 2024-03-18T01:39:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.159+0000 7f5f51f63740 1 -- start start 2024-03-18T01:39:51.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.159+0000 7f5f51f63740 1 Processor -- start 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.160+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 0x557c8c733b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.160+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c7340a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.160+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72f420 0x557c8c73b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.160+0000 7f5f47400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c7340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.160+0000 7f5f47400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c7340a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54944/0 (socket says 172.21.15.7:54944) 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.160+0000 7f5f47e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 0x557c8c733b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.160+0000 7f5f47400640 1 -- 172.21.15.7:0/2290809446 learned_addr learned my addr 172.21.15.7:0/2290809446 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557c8c64a110 con 0x557c8c72f420 2024-03-18T01:39:51.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c8c68a4f0 con 0x557c8c72a2b0 2024-03-18T01:39:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557c8c7323d0 con 0x557c8c7298c0 2024-03-18T01:39:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f51f63740 10 client.0 _ll_register_callbacks cb 0x557c8c637b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f47400640 1 -- 172.21.15.7:0/2290809446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 msgr2=0x557c8c733b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f47400640 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 0x557c8c733b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f47400640 1 -- 172.21.15.7:0/2290809446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72f420 msgr2=0x557c8c73b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f47400640 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72f420 0x557c8c73b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.161+0000 7f5f47400640 1 -- 172.21.15.7:0/2290809446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c8c3a23f0 con 0x557c8c72a2b0 2024-03-18T01:39:51.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.162+0000 7f5f47400640 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c7340a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5f3803e040 tx=0x7f5f38011300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9105d0b7e57edd7d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.164+0000 7f5f32000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.164+0000 7f5f32000640 1 -- 172.21.15.7:0/2290809446 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f38005240 con 0x557c8c72a2b0 2024-03-18T01:39:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.164+0000 7f5f32000640 1 -- 172.21.15.7:0/2290809446 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5f380053e0 con 0x557c8c72a2b0 2024-03-18T01:39:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.164+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f34022020 con 0x557c8c72a2b0 2024-03-18T01:39:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.164+0000 7f5f32000640 1 -- 172.21.15.7:0/2290809446 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5f38059050 con 0x557c8c72a2b0 2024-03-18T01:39:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.164+0000 7f5f51f63740 10 client.25946 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.164+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c8c46c7d0 con 0x557c8c72a2b0 2024-03-18T01:39:51.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.165+0000 7f5f32000640 1 -- 172.21.15.7:0/2290809446 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f34022020 con 0x557c8c72a2b0 2024-03-18T01:39:51.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.166+0000 7f5f32000640 1 -- 172.21.15.7:0/2290809446 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5f3803c700 con 0x557c8c72a2b0 2024-03-18T01:39:51.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.166+0000 7f5f51f63740 10 client.25946 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:51.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.166+0000 7f5f51f63740 10 client.25946 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:39:51.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.166+0000 7f5f51f63740 20 client.25946 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:51.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.166+0000 7f5f51f63740 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x557c8c73f970 0x557c8c741e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.166+0000 7f5f51f63740 4 client.25946 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.166+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c8c69b990 con 0x557c8c73f970 2024-03-18T01:39:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.167+0000 7f5f47e00640 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x557c8c73f970 0x557c8c741e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.167+0000 7f5f32000640 10 client.25946 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:39:51.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.167+0000 7f5f47e00640 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x557c8c73f970 0x557c8c741e10 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6bee835a403cf3 server_cookie=a7ef8a9d1f2f21d5 in_seq=0 out_seq=0 2024-03-18T01:39:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.168+0000 7f5f32000640 1 -- 172.21.15.7:0/2290809446 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c8c69b990 con 0x557c8c73f970 2024-03-18T01:39:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.168+0000 7f5f32000640 10 client.25946 handle_command_reply: tid=0 2024-03-18T01:39:51.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.168+0000 7f5f51f63740 1 client.25946 shutdown 2024-03-18T01:39:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 msgr2=0x557c8c7340a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c7340a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5f3803e040 tx=0x7f5f38011300 comp rx=0 tx=0).stop 2024-03-18T01:39:51.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 shutdown_connections 2024-03-18T01:39:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x557c8c73f970 0x557c8c741e10 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c72f420 0x557c8c73b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c7340a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 --2- 172.21.15.7:0/2290809446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c7298c0 0x557c8c733b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 >> 172.21.15.7:0/2290809446 conn(0x557c8c724e30 msgr2=0x557c8c7241f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f32000640 0 client.25946 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:39:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 shutdown_connections 2024-03-18T01:39:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 1 -- 172.21.15.7:0/2290809446 wait complete. 2024-03-18T01:39:51.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.169+0000 7f5f51f63740 20 client.25946 trim_cache size 0 max 16384 2024-03-18T01:39:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.190+0000 7f5f51f63740 1 Processor -- start 2024-03-18T01:39:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.190+0000 7f5f51f63740 1 -- start start 2024-03-18T01:39:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.190+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 0x557c8c724070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.190+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72a2b0 0x557c8c73b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.190+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7466d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.190+0000 7f5f51f63740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5f481b0990 con 0x557c8c7298c0 2024-03-18T01:39:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.190+0000 7f5f51f63740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5f481b2530 con 0x557c8c72f420 2024-03-18T01:39:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.190+0000 7f5f51f63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f481b1060 con 0x557c8c72a2b0 2024-03-18T01:39:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.191+0000 7f5f4d200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7466d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.191+0000 7f5f4d200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7466d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54954/0 (socket says 172.21.15.7:54954) 2024-03-18T01:39:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.191+0000 7f5f4d200640 1 -- 172.21.15.7:0/45193650 learned_addr learned my addr 172.21.15.7:0/45193650 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.191+0000 7f5f47400640 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72a2b0 0x557c8c73b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.191+0000 7f5f47e00640 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 0x557c8c724070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.192+0000 7f5f47400640 1 -- 172.21.15.7:0/45193650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 msgr2=0x557c8c7466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.192+0000 7f5f47400640 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7466d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.192+0000 7f5f47400640 1 -- 172.21.15.7:0/45193650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 msgr2=0x557c8c724070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.192+0000 7f5f47400640 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 0x557c8c724070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.192+0000 7f5f47400640 1 -- 172.21.15.7:0/45193650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c8c720c10 con 0x557c8c72a2b0 2024-03-18T01:39:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.192+0000 7f5f47400640 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72a2b0 0x557c8c73b0a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5f38037020 tx=0x7f5f38005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8670c747129544f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.193+0000 7f5f3cc00640 1 -- 172.21.15.7:0/45193650 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3803ccc0 con 0x557c8c72a2b0 2024-03-18T01:39:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.193+0000 7f5f3cc00640 1 -- 172.21.15.7:0/45193650 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f38004040 con 0x557c8c72a2b0 2024-03-18T01:39:51.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.193+0000 7f5f3cc00640 1 -- 172.21.15.7:0/45193650 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f38005360 con 0x557c8c72a2b0 2024-03-18T01:39:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 -- 172.21.15.7:0/45193650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72a2b0 msgr2=0x557c8c73b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72a2b0 0x557c8c73b0a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5f38037020 tx=0x7f5f38005f90 comp rx=0 tx=0).stop 2024-03-18T01:39:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 -- 172.21.15.7:0/45193650 shutdown_connections 2024-03-18T01:39:51.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72f420 0x557c8c7466d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72a2b0 0x557c8c73b0a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 --2- 172.21.15.7:0/45193650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 0x557c8c724070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 -- 172.21.15.7:0/45193650 >> 172.21.15.7:0/45193650 conn(0x557c8c724e30 msgr2=0x557c8c723960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 -- 172.21.15.7:0/45193650 shutdown_connections 2024-03-18T01:39:51.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.194+0000 7f5f51f63740 1 -- 172.21.15.7:0/45193650 wait complete. 2024-03-18T01:39:51.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.195+0000 7f5f51f63740 1 -- start start 2024-03-18T01:39:51.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.195+0000 7f5f51f63740 1 Processor -- start 2024-03-18T01:39:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 0x557c8c738970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c738eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f51f63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72f420 0x557c8c739e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f51f63740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5f481b1e00 con 0x557c8c7298c0 2024-03-18T01:39:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f51f63740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5f481b1730 con 0x557c8c72a2b0 2024-03-18T01:39:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f51f63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f481ae970 con 0x557c8c72f420 2024-03-18T01:39:51.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f51f63740 10 client.0 _ll_register_callbacks cb 0x7f5f38011ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f47e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 0x557c8c738970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f47400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c738eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f47400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c738eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54964/0 (socket says 172.21.15.7:54964) 2024-03-18T01:39:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.196+0000 7f5f47400640 1 -- 172.21.15.7:0/3436684994 learned_addr learned my addr 172.21.15.7:0/3436684994 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.197+0000 7f5f4d200640 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72f420 0x557c8c739e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.197+0000 7f5f47400640 1 -- 172.21.15.7:0/3436684994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72f420 msgr2=0x557c8c739e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.198+0000 7f5f47400640 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72f420 0x557c8c739e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.198+0000 7f5f47400640 1 -- 172.21.15.7:0/3436684994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 msgr2=0x557c8c738970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.198+0000 7f5f47400640 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 0x557c8c738970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.198+0000 7f5f47400640 1 -- 172.21.15.7:0/3436684994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c8c732510 con 0x557c8c72a2b0 2024-03-18T01:39:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.199+0000 7f5f47400640 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c738eb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5f38011f90 tx=0x7f5f38011710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=810001dbcaf62bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.199+0000 7f5f32000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:51.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.199+0000 7f5f32000640 1 -- 172.21.15.7:0/3436684994 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3803c770 con 0x557c8c72a2b0 2024-03-18T01:39:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.200+0000 7f5f32000640 1 -- 172.21.15.7:0/3436684994 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5f380050e0 con 0x557c8c72a2b0 2024-03-18T01:39:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.200+0000 7f5f51f63740 1 -- 172.21.15.7:0/3436684994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f38004040 con 0x557c8c72a2b0 2024-03-18T01:39:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.200+0000 7f5f32000640 1 -- 172.21.15.7:0/3436684994 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5f380052e0 con 0x557c8c72a2b0 2024-03-18T01:39:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.201+0000 7f5f51f63740 10 client.25952 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.201+0000 7f5f51f63740 1 -- 172.21.15.7:0/3436684994 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c8c7a5d10 con 0x557c8c72a2b0 2024-03-18T01:39:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.201+0000 7f5f32000640 1 -- 172.21.15.7:0/3436684994 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f38004040 con 0x557c8c72a2b0 2024-03-18T01:39:51.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.202+0000 7f5f32000640 1 -- 172.21.15.7:0/3436684994 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5f38004d30 con 0x557c8c72a2b0 2024-03-18T01:39:51.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.202+0000 7f5f51f63740 10 client.25952 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:51.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.202+0000 7f5f51f63740 10 client.25952 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:39:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.202+0000 7f5f51f63740 20 client.25952 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.202+0000 7f5f51f63740 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x557c8c7a65a0 0x557c8c7a89c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.202+0000 7f5f51f63740 4 client.25952 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.202+0000 7f5f51f63740 1 -- 172.21.15.7:0/3436684994 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c8c695ce0 con 0x557c8c7a65a0 2024-03-18T01:39:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.202+0000 7f5f47e00640 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x557c8c7a65a0 0x557c8c7a89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.203+0000 7f5f47e00640 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x557c8c7a65a0 0x557c8c7a89c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3d53ad970c057680 server_cookie=c32c51ab189d751e in_seq=0 out_seq=0 2024-03-18T01:39:51.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.203+0000 7f5f32000640 10 client.25952 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:39:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.204+0000 7f5f32000640 1 -- 172.21.15.7:0/3436684994 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c8c695ce0 con 0x557c8c7a65a0 2024-03-18T01:39:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.204+0000 7f5f32000640 10 client.25952 handle_command_reply: tid=0 2024-03-18T01:39:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.204+0000 7f5f51f63740 1 client.25952 shutdown 2024-03-18T01:39:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 -- 172.21.15.7:0/3436684994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 msgr2=0x557c8c738eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c738eb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5f38011f90 tx=0x7f5f38011710 comp rx=0 tx=0).stop 2024-03-18T01:39:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 -- 172.21.15.7:0/3436684994 shutdown_connections 2024-03-18T01:39:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x557c8c7a65a0 0x557c8c7a89c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c8c72f420 0x557c8c739e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c8c72a2b0 0x557c8c738eb0 secure :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5f38011f90 tx=0x7f5f38011710 comp rx=0 tx=0).stop 2024-03-18T01:39:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 --2- 172.21.15.7:0/3436684994 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c8c7298c0 0x557c8c738970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 -- 172.21.15.7:0/3436684994 >> 172.21.15.7:0/3436684994 conn(0x557c8c724e30 msgr2=0x557c8c7241f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f32000640 0 client.25952 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:39:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.205+0000 7f5f51f63740 1 -- 172.21.15.7:0/3436684994 shutdown_connections 2024-03-18T01:39:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.206+0000 7f5f51f63740 1 -- 172.21.15.7:0/3436684994 wait complete. 2024-03-18T01:39:51.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.206+0000 7f5f51f63740 20 client.25952 trim_cache size 0 max 16384 2024-03-18T01:39:51.208 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.207+0000 7f5f4f200640 1 -- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5f2c0a2880 msgr2=0x7f5f2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.207+0000 7f5f4f200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5f2c0a2880 0x7f5f2c0a4d40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5f34023000 tx=0x7f5f34020000 comp rx=0 tx=0).stop 2024-03-18T01:39:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.207+0000 7f5f4f200640 1 -- 172.21.15.7:0/3605955006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f481017e0 msgr2=0x7f5f480701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.207+0000 7f5f4f200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f481017e0 0x7f5f480701c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000ad10 tx=0x7f5f40007590 comp rx=0 tx=0).stop 2024-03-18T01:39:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.208+0000 7f5f4f200640 1 -- 172.21.15.7:0/3605955006 shutdown_connections 2024-03-18T01:39:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.208+0000 7f5f4f200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f48103390 0x7f5f481a5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.208+0000 7f5f4f200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5f2c0a2880 0x7f5f2c0a4d40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.208+0000 7f5f4f200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5f481029d0 0x7f5f48070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.208+0000 7f5f4f200640 1 --2- 172.21.15.7:0/3605955006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f481017e0 0x7f5f480701c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.208+0000 7f5f4f200640 1 -- 172.21.15.7:0/3605955006 >> 172.21.15.7:0/3605955006 conn(0x7f5f480fcd30 msgr2=0x7f5f48105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.208+0000 7f5f4f200640 1 -- 172.21.15.7:0/3605955006 shutdown_connections 2024-03-18T01:39:51.210 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.209+0000 7f5f4f200640 1 -- 172.21.15.7:0/3605955006 wait complete. 2024-03-18T01:39:51.234 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T01:39:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.428+0000 7f7ef5200640 1 -- 172.21.15.7:0/3076712144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 msgr2=0x7f7ef01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.428+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3076712144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 0x7f7ef01093f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009900 tx=0x7f7ed802bc40 comp rx=0 tx=0).stop 2024-03-18T01:39:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.428+0000 7f7ef5200640 1 -- 172.21.15.7:0/3076712144 shutdown_connections 2024-03-18T01:39:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.428+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3076712144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef0109930 0x7f7ef0110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.428+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3076712144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 0x7f7ef01093f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.428+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3076712144 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef0107e20 0x7f7ef0108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.428+0000 7f7ef5200640 1 -- 172.21.15.7:0/3076712144 >> 172.21.15.7:0/3076712144 conn(0x7f7ef0076de0 msgr2=0x7f7ef0079200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.428+0000 7f7ef5200640 1 -- 172.21.15.7:0/3076712144 shutdown_connections 2024-03-18T01:39:51.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.429+0000 7f7ef5200640 1 -- 172.21.15.7:0/3076712144 wait complete. 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.429+0000 7f7ef5200640 1 Processor -- start 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ef5200640 1 -- start start 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ef5200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef0107e20 0x7f7ef019e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ef5200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 0x7f7ef019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ef5200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef0109930 0x7f7ef01a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ef5200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef0113b20 con 0x7f7ef0107e20 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ef5200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01139a0 con 0x7f7ef0108f70 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ef5200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef0113ca0 con 0x7f7ef0109930 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ee5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 0x7f7ef019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ee5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 0x7f7ef019e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54976/0 (socket says 172.21.15.7:54976) 2024-03-18T01:39:51.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7ee5400640 1 -- 172.21.15.7:0/3343273258 learned_addr learned my addr 172.21.15.7:0/3343273258 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7eede00640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef0107e20 0x7f7ef019e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.430+0000 7f7eee800640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef0109930 0x7f7ef01a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee5400640 1 -- 172.21.15.7:0/3343273258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef0109930 msgr2=0x7f7ef01a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee5400640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef0109930 0x7f7ef01a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee5400640 1 -- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef0107e20 msgr2=0x7f7ef019e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee5400640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef0107e20 0x7f7ef019e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee5400640 1 -- 172.21.15.7:0/3343273258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef01a5d20 con 0x7f7ef0108f70 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee5400640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 0x7f7ef019e640 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7ed802efd0 tx=0x7f7ed802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee7400640 1 -- 172.21.15.7:0/3343273258 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8041070 con 0x7f7ef0108f70 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee7400640 1 -- 172.21.15.7:0/3343273258 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed8004450 con 0x7f7ef0108f70 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ee7400640 1 -- 172.21.15.7:0/3343273258 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8003e50 con 0x7f7ef0108f70 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.431+0000 7f7ef5200640 1 -- 172.21.15.7:0/3343273258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef01a5fb0 con 0x7f7ef0108f70 2024-03-18T01:39:51.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.432+0000 7f7ef5200640 1 -- 172.21.15.7:0/3343273258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef01a6360 con 0x7f7ef0108f70 2024-03-18T01:39:51.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.434+0000 7f7ef7fc9740 1 Processor -- start 2024-03-18T01:39:51.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ee7400640 1 -- 172.21.15.7:0/3343273258 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f7ed8044070 con 0x7f7ef0108f70 2024-03-18T01:39:51.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ef7fc9740 1 -- start start 2024-03-18T01:39:51.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fc8c0 0x55bc360fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fd2b0 0x55bc36101db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc36102420 0x55bc361048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bc360c43b0 con 0x55bc360fc8c0 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bc36061450 con 0x55bc360fd2b0 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bc36009cb0 con 0x55bc36102420 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7eee800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fc8c0 0x55bc360fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ee5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc36102420 0x55bc361048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ee5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc36102420 0x55bc361048b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45086/0 (socket says 172.21.15.7:45086) 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7ee5400640 1 -- 172.21.15.7:0/3086746557 learned_addr learned my addr 172.21.15.7:0/3086746557 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7eede00640 1 --2- 172.21.15.7:0/3086746557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fd2b0 0x55bc36101db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.435+0000 7f7eee800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fc8c0 0x55bc360fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49168/0 (socket says 172.21.15.7:49168) 2024-03-18T01:39:51.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7eede00640 1 -- 172.21.15.7:0/3086746557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc36102420 msgr2=0x55bc361048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7eede00640 1 --2- 172.21.15.7:0/3086746557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc36102420 0x55bc361048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7eede00640 1 -- 172.21.15.7:0/3086746557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fc8c0 msgr2=0x55bc360fcce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7eede00640 1 --2- 172.21.15.7:0/3086746557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fc8c0 0x55bc360fcce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7eede00640 1 -- 172.21.15.7:0/3086746557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc3602b1f0 con 0x55bc360fd2b0 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7eede00640 1 --2- 172.21.15.7:0/3086746557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fd2b0 0x55bc36101db0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009fa0 tx=0x7f7ee00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a21fe19f2dc2491 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7ee7400640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ed40a29a0 0x7f7ed40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7edf400640 1 -- 172.21.15.7:0/3086746557 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0003c30 con 0x55bc360fd2b0 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7ee7400640 1 -- 172.21.15.7:0/3343273258 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7ed80f0960 con 0x7f7ef0108f70 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7edf400640 1 -- 172.21.15.7:0/3086746557 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee0003dd0 con 0x55bc360fd2b0 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7edf400640 1 -- 172.21.15.7:0/3086746557 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee000fb00 con 0x55bc360fd2b0 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.436+0000 7f7eee800640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ed40a29a0 0x7f7ed40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3086746557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fd2b0 msgr2=0x55bc36101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3086746557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fd2b0 0x55bc36101db0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009fa0 tx=0x7f7ee00118d0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7eee800640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ed40a29a0 0x7f7ed40a4e60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55bc360fdb10 tx=0x7f7ee8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3086746557 shutdown_connections 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3086746557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc36102420 0x55bc361048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3086746557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fd2b0 0x55bc36101db0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3086746557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fc8c0 0x55bc360fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3086746557 >> 172.21.15.7:0/3086746557 conn(0x55bc360f7e30 msgr2=0x55bc360f8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3086746557 shutdown_connections 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.437+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3086746557 wait complete. 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.438+0000 7f7ef7fc9740 1 -- start start 2024-03-18T01:39:51.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.438+0000 7f7ef7fc9740 1 Processor -- start 2024-03-18T01:39:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc36106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc361070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 0x55bc3610e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55bc3601d110 con 0x55bc360fd2b0 2024-03-18T01:39:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55bc3605d4f0 con 0x55bc36102420 2024-03-18T01:39:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55bc361053d0 con 0x55bc360fc8c0 2024-03-18T01:39:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7ef7fc9740 10 client.0 _ll_register_callbacks cb 0x55bc3600ab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:51.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7eede00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc36106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7eede00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc36106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45090/0 (socket says 172.21.15.7:45090) 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7eede00640 1 -- 172.21.15.7:0/3536281398 learned_addr learned my addr 172.21.15.7:0/3536281398 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7eee800640 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 0x55bc3610e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.439+0000 7f7ee5400640 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc361070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7eede00640 1 -- 172.21.15.7:0/3536281398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 msgr2=0x55bc3610e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7eede00640 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 0x55bc3610e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7eede00640 1 -- 172.21.15.7:0/3536281398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 msgr2=0x55bc361070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7eede00640 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc361070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7eede00640 1 -- 172.21.15.7:0/3536281398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc35d753f0 con 0x55bc360fc8c0 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7ee5400640 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc361070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7eede00640 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc36106ba0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7ee001a000 tx=0x7f7ee0004c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7556a1193f7c5e7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7ed3e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7ed3e00640 1 -- 172.21.15.7:0/3536281398 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0029020 con 0x55bc360fc8c0 2024-03-18T01:39:51.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7ed3e00640 1 -- 172.21.15.7:0/3536281398 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7ee000fc50 con 0x55bc360fc8c0 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.440+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3536281398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee0003dd0 con 0x55bc360fc8c0 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ed3e00640 1 -- 172.21.15.7:0/3536281398 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7ee0034050 con 0x55bc360fc8c0 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ef7fc9740 10 client.25936 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3536281398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bc35e3f7d0 con 0x55bc360fc8c0 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ed3e00640 1 -- 172.21.15.7:0/3536281398 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee0003dd0 con 0x55bc360fc8c0 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ed3e00640 1 -- 172.21.15.7:0/3536281398 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7ee0005040 con 0x55bc360fc8c0 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ef7fc9740 10 client.25936 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ef7fc9740 10 client.25936 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ef7fc9740 20 client.25936 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55bc36112810 0x55bc36114c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ef7fc9740 4 client.25936 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.441+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3536281398 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc3606e990 con 0x55bc36112810 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.442+0000 7f7ee5400640 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55bc36112810 0x55bc36114c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.442+0000 7f7ed3e00640 10 client.25936 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.442+0000 7f7ee5400640 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55bc36112810 0x55bc36114c80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6821fc2a7dde56d8 server_cookie=681c02446d436856 in_seq=0 out_seq=0 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ed3e00640 1 -- 172.21.15.7:0/3536281398 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc3606e990 con 0x55bc36112810 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ed3e00640 10 client.25936 handle_command_reply: tid=0 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 client.25936 shutdown 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3536281398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 msgr2=0x55bc36106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc36106ba0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7ee001a000 tx=0x7f7ee0004c70 comp rx=0 tx=0).stop 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3536281398 shutdown_connections 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55bc36112810 0x55bc36114c80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 0x55bc3610e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc361070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/3536281398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc36106ba0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ed3e00640 0 client.25936 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.443+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3536281398 >> 172.21.15.7:0/3536281398 conn(0x55bc360f7e30 msgr2=0x55bc360f71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.444+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3536281398 shutdown_connections 2024-03-18T01:39:51.448 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.444+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/3536281398 wait complete. 2024-03-18T01:39:51.449 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.444+0000 7f7ef7fc9740 20 client.25936 trim_cache size 0 max 16384 2024-03-18T01:39:51.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7ef7fc9740 1 Processor -- start 2024-03-18T01:39:51.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7ef7fc9740 1 -- start start 2024-03-18T01:39:51.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc360f8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc3610e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 0x55bc36105e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01b08c0 con 0x55bc360fd2b0 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01b2460 con 0x55bc36102420 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01b0f90 con 0x55bc360fc8c0 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7eede00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc360f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7eede00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc360f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45096/0 (socket says 172.21.15.7:45096) 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7eede00640 1 -- 172.21.15.7:0/2187013188 learned_addr learned my addr 172.21.15.7:0/2187013188 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.458+0000 7f7eee800640 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 0x55bc36105e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7ee5400640 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc3610e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7eede00640 1 -- 172.21.15.7:0/2187013188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 msgr2=0x55bc36105e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7eede00640 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 0x55bc36105e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7eede00640 1 -- 172.21.15.7:0/2187013188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 msgr2=0x55bc3610e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7eede00640 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc3610e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7eede00640 1 -- 172.21.15.7:0/2187013188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc360f3c10 con 0x55bc360fc8c0 2024-03-18T01:39:51.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7ee5400640 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc3610e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:51.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7eede00640 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc360f8210 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0005b40 tx=0x7f7ee0009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2138f1bbbe297813 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7edea00640 1 -- 172.21.15.7:0/2187013188 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0029040 con 0x55bc360fc8c0 2024-03-18T01:39:51.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7edea00640 1 -- 172.21.15.7:0/2187013188 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee0005880 con 0x55bc360fc8c0 2024-03-18T01:39:51.462 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.459+0000 7f7edea00640 1 -- 172.21.15.7:0/2187013188 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee00051f0 con 0x55bc360fc8c0 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.460+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2187013188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 msgr2=0x55bc360f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.460+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc360f8210 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0005b40 tx=0x7f7ee0009900 comp rx=0 tx=0).stop 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.460+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2187013188 shutdown_connections 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.460+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc36102420 0x55bc36105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.460+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc360fd2b0 0x55bc3610e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.460+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2187013188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fc8c0 0x55bc360f8210 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.460+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2187013188 >> 172.21.15.7:0/2187013188 conn(0x55bc360f7e30 msgr2=0x55bc360f71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.460+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2187013188 shutdown_connections 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.461+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2187013188 wait complete. 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.461+0000 7f7ef7fc9740 1 -- start start 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.461+0000 7f7ef7fc9740 1 Processor -- start 2024-03-18T01:39:51.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.461+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fc8c0 0x55bc3610b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.461+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fd2b0 0x55bc3610b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.461+0000 7f7ef7fc9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc36102420 0x55bc36176b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01b1d30 con 0x55bc36102420 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01b1660 con 0x55bc360fc8c0 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ef7fc9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01ae8a0 con 0x55bc360fd2b0 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ef7fc9740 10 client.0 _ll_register_callbacks cb 0x7f7ee8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fd2b0 0x55bc3610b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7eede00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fc8c0 0x55bc3610b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fd2b0 0x55bc3610b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45106/0 (socket says 172.21.15.7:45106) 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7eede00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fc8c0 0x55bc3610b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55018/0 (socket says 172.21.15.7:55018) 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 -- 172.21.15.7:0/2211682344 learned_addr learned my addr 172.21.15.7:0/2211682344 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7eee800640 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc36102420 0x55bc36176b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 -- 172.21.15.7:0/2211682344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fc8c0 msgr2=0x55bc3610b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fc8c0 0x55bc3610b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 -- 172.21.15.7:0/2211682344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc36102420 msgr2=0x55bc36176b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc36102420 0x55bc36176b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 -- 172.21.15.7:0/2211682344 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc361770e0 con 0x55bc360fd2b0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7eede00640 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fc8c0 0x55bc3610b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7eee800640 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc36102420 0x55bc36176b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ee5400640 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fd2b0 0x55bc3610b5f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8006010 tx=0x7f7ed802c200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b72657e110105318 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.462+0000 7f7ed3e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ed3e00640 1 -- 172.21.15.7:0/2211682344 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed809be60 con 0x55bc360fd2b0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2211682344 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ee0005880 con 0x55bc360fd2b0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ed3e00640 1 -- 172.21.15.7:0/2211682344 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7ed80b9660 con 0x55bc360fd2b0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ed3e00640 1 -- 172.21.15.7:0/2211682344 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7ed804b040 con 0x55bc360fd2b0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 10 client.25942 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2211682344 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bc36178490 con 0x55bc360fd2b0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ed3e00640 1 -- 172.21.15.7:0/2211682344 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee0005880 con 0x55bc360fd2b0 2024-03-18T01:39:51.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ed3e00640 1 -- 172.21.15.7:0/2211682344 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7ed80035f0 con 0x55bc360fd2b0 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 10 client.25942 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 10 client.25942 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 20 client.25942 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55bc36178da0 0x55bc3617b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 4 client.25942 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.463+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2211682344 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc36068ce0 con 0x55bc36178da0 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.464+0000 7f7eede00640 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55bc36178da0 0x55bc3617b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.464+0000 7f7eede00640 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55bc36178da0 0x55bc3617b1c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1e2bae6912e0c799 server_cookie=26e83a3d94eef97c in_seq=0 out_seq=0 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.464+0000 7f7ed3e00640 10 client.25942 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.464+0000 7f7ed3e00640 1 -- 172.21.15.7:0/2211682344 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc36068ce0 con 0x55bc36178da0 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.464+0000 7f7ed3e00640 10 client.25942 handle_command_reply: tid=0 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.464+0000 7f7ef7fc9740 1 client.25942 shutdown 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.464+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2211682344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fd2b0 msgr2=0x55bc3610b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.464+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fd2b0 0x55bc3610b5f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8006010 tx=0x7f7ed802c200 comp rx=0 tx=0).stop 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2211682344 shutdown_connections 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55bc36178da0 0x55bc3617b1c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55bc36102420 0x55bc36176b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55bc360fd2b0 0x55bc3610b5f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 1 --2- 172.21.15.7:0/2211682344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bc360fc8c0 0x55bc3610b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ed3e00640 0 client.25942 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2211682344 >> 172.21.15.7:0/2211682344 conn(0x55bc360f7e30 msgr2=0x55bc360f71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2211682344 shutdown_connections 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 1 -- 172.21.15.7:0/2211682344 wait complete. 2024-03-18T01:39:51.470 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.465+0000 7f7ef7fc9740 20 client.25942 trim_cache size 0 max 16384 2024-03-18T01:39:51.471 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 -- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ed40a29a0 msgr2=0x7f7ed40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ed40a29a0 0x7f7ed40a4e60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55bc360fdb10 tx=0x7f7ee8005f90 comp rx=0 tx=0).stop 2024-03-18T01:39:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 -- 172.21.15.7:0/3343273258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 msgr2=0x7f7ef019e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 0x7f7ef019e640 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7ed802efd0 tx=0x7f7ed802c660 comp rx=0 tx=0).stop 2024-03-18T01:39:51.471 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 -- 172.21.15.7:0/3343273258 shutdown_connections 2024-03-18T01:39:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef0109930 0x7f7ef01a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef0108f70 0x7f7ef019e640 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7ed40a29a0 0x7f7ed40a4e60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 --2- 172.21.15.7:0/3343273258 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef0107e20 0x7f7ef019e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 -- 172.21.15.7:0/3343273258 >> 172.21.15.7:0/3343273258 conn(0x7f7ef0076de0 msgr2=0x7f7ef010c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 -- 172.21.15.7:0/3343273258 shutdown_connections 2024-03-18T01:39:51.472 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.466+0000 7f7ef5200640 1 -- 172.21.15.7:0/3343273258 wait complete. 2024-03-18T01:39:51.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:51 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/509475889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:39:51.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T01:39:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.670+0000 7f3e34a00640 1 -- 172.21.15.7:0/1943632330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e30074240 msgr2=0x7f3e300746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.670+0000 7f3e34a00640 1 --2- 172.21.15.7:0/1943632330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e30074240 0x7f3e300746c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009930 tx=0x7f3e1802bc30 comp rx=0 tx=0).stop 2024-03-18T01:39:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.670+0000 7f3e34a00640 1 -- 172.21.15.7:0/1943632330 shutdown_connections 2024-03-18T01:39:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.670+0000 7f3e34a00640 1 --2- 172.21.15.7:0/1943632330 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e3010dbf0 0x7f3e3010ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.670+0000 7f3e34a00640 1 --2- 172.21.15.7:0/1943632330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e30074240 0x7f3e300746c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.670+0000 7f3e34a00640 1 --2- 172.21.15.7:0/1943632330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e30075820 0x7f3e30073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.670+0000 7f3e34a00640 1 -- 172.21.15.7:0/1943632330 >> 172.21.15.7:0/1943632330 conn(0x7f3e300fcb90 msgr2=0x7f3e300fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.671+0000 7f3e34a00640 1 -- 172.21.15.7:0/1943632330 shutdown_connections 2024-03-18T01:39:51.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.671+0000 7f3e34a00640 1 -- 172.21.15.7:0/1943632330 wait complete. 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.671+0000 7f3e34a00640 1 Processor -- start 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e34a00640 1 -- start start 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e34a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e30074240 0x7f3e3019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e34a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e30075820 0x7f3e3019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e34a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e3010dbf0 0x7f3e301a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e34a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301138e0 con 0x7f3e3010dbf0 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e34a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30113760 con 0x7f3e30074240 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e34a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30113a60 con 0x7f3e30075820 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e26a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e30075820 0x7f3e3019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e2f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e30074240 0x7f3e3019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e26a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e30075820 0x7f3e3019e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45126/0 (socket says 172.21.15.7:45126) 2024-03-18T01:39:51.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e26a00640 1 -- 172.21.15.7:0/2445206546 learned_addr learned my addr 172.21.15.7:0/2445206546 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e2fe00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e3010dbf0 0x7f3e301a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e26a00640 1 -- 172.21.15.7:0/2445206546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e30074240 msgr2=0x7f3e3019df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e26a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e30074240 0x7f3e3019df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e26a00640 1 -- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e3010dbf0 msgr2=0x7f3e301a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.672+0000 7f3e26a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e3010dbf0 0x7f3e301a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.673+0000 7f3e26a00640 1 -- 172.21.15.7:0/2445206546 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e301a5b80 con 0x7f3e30075820 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.673+0000 7f3e26a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e30075820 0x7f3e3019e4a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e18000c00 tx=0x7f3e1802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.673+0000 7f3e2cc00640 1 -- 172.21.15.7:0/2445206546 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18043020 con 0x7f3e30075820 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.673+0000 7f3e2cc00640 1 -- 172.21.15.7:0/2445206546 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e180054e0 con 0x7f3e30075820 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.673+0000 7f3e34a00640 1 -- 172.21.15.7:0/2445206546 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e301a5e10 con 0x7f3e30075820 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.673+0000 7f3e2cc00640 1 -- 172.21.15.7:0/2445206546 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18003c40 con 0x7f3e30075820 2024-03-18T01:39:51.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.673+0000 7f3e34a00640 1 -- 172.21.15.7:0/2445206546 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e301a6320 con 0x7f3e30075820 2024-03-18T01:39:51.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.676+0000 7f3e389fd740 1 Processor -- start 2024-03-18T01:39:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.676+0000 7f3e2cc00640 1 -- 172.21.15.7:0/2445206546 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f3e18046020 con 0x7f3e30075820 2024-03-18T01:39:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.676+0000 7f3e389fd740 1 -- start start 2024-03-18T01:39:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b624aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b624ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b6252a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e389fd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5570b62123b0 con 0x5570b624aa90 2024-03-18T01:39:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e389fd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5570b61af450 con 0x5570b62505f0 2024-03-18T01:39:51.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e389fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5570b6157cb0 con 0x5570b624b480 2024-03-18T01:39:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b624aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b624ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b624aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49212/0 (socket says 172.21.15.7:49212) 2024-03-18T01:39:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2fe00640 1 -- 172.21.15.7:0/2477143543 learned_addr learned my addr 172.21.15.7:0/2477143543 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e26a00640 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b6252a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2fe00640 1 -- 172.21.15.7:0/2477143543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 msgr2=0x5570b624ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2fe00640 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b624ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2fe00640 1 -- 172.21.15.7:0/2477143543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 msgr2=0x5570b6252a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2fe00640 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b6252a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2fe00640 1 -- 172.21.15.7:0/2477143543 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570b61791f0 con 0x5570b624aa90 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e2f400640 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b624ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.677+0000 7f3e26a00640 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b6252a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.678+0000 7f3e2fe00640 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b624aeb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009c10 tx=0x7f3e280117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55175daf8ddd912 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.678+0000 7f3e2cc00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e140a2870 0x7f3e140a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.678+0000 7f3e1fe00640 1 -- 172.21.15.7:0/2477143543 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e28012450 con 0x5570b624aa90 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.678+0000 7f3e1fe00640 1 -- 172.21.15.7:0/2477143543 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e28012ad0 con 0x5570b624aa90 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.678+0000 7f3e2cc00640 1 -- 172.21.15.7:0/2445206546 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e180f0dd0 con 0x7f3e30075820 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.678+0000 7f3e1fe00640 1 -- 172.21.15.7:0/2477143543 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800fb70 con 0x5570b624aa90 2024-03-18T01:39:51.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.678+0000 7f3e2f400640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e140a2870 0x7f3e140a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 -- 172.21.15.7:0/2477143543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 msgr2=0x5570b624aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b624aeb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009c10 tx=0x7f3e280117b0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 -- 172.21.15.7:0/2477143543 shutdown_connections 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b6252a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b624ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2477143543 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b624aeb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 -- 172.21.15.7:0/2477143543 >> 172.21.15.7:0/2477143543 conn(0x5570b6246000 msgr2=0x5570b6246400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 -- 172.21.15.7:0/2477143543 shutdown_connections 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e2f400640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e140a2870 0x7f3e140a4d30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x5570b624b880 tx=0x7f3e20009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.679+0000 7f3e389fd740 1 -- 172.21.15.7:0/2477143543 wait complete. 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 1 -- start start 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 1 Processor -- start 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b6254df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.683 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b6255330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b625c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5570b616b110 con 0x5570b624aa90 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5570b61ab4f0 con 0x5570b62505f0 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5570b62535a0 con 0x5570b624b480 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e389fd740 10 client.0 _ll_register_callbacks cb 0x5570b6158b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e2f400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b6254df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e2fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b625c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e2fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b625c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55060/0 (socket says 172.21.15.7:55060) 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e26a00640 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b6255330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e26a00640 1 -- 172.21.15.7:0/25702252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 msgr2=0x5570b625c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e26a00640 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b625c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e26a00640 1 -- 172.21.15.7:0/25702252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 msgr2=0x5570b6254df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e26a00640 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b6254df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e26a00640 1 -- 172.21.15.7:0/25702252 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570b5ec33f0 con 0x5570b624b480 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e2f400640 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b6254df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e26a00640 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b6255330 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3e180ee5c0 tx=0x7f3e1802c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48ecea2b505102d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.680+0000 7f3e2fe00640 1 -- 172.21.15.7:0/25702252 learned_addr learned my addr 172.21.15.7:0/25702252 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e2fe00640 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b625c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e1cc00640 1 -- 172.21.15.7:0/25702252 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e180bcf50 con 0x5570b624b480 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e389fd740 1 -- 172.21.15.7:0/25702252 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e28012ad0 con 0x5570b624b480 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.681+0000 7f3e1cc00640 1 -- 172.21.15.7:0/25702252 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e180bb710 con 0x5570b624b480 2024-03-18T01:39:51.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e1cc00640 1 -- 172.21.15.7:0/25702252 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3e180bcd70 con 0x5570b624b480 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e389fd740 10 client.25954 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e389fd740 1 -- 172.21.15.7:0/25702252 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5570b5f8d7d0 con 0x5570b624b480 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e1cc00640 1 -- 172.21.15.7:0/25702252 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e28012ad0 con 0x5570b624b480 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e1cc00640 1 -- 172.21.15.7:0/25702252 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3e180c11a0 con 0x5570b624b480 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e389fd740 10 client.25954 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e389fd740 10 client.25954 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e389fd740 20 client.25954 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e389fd740 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5570b6260b10 0x5570b6262f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e389fd740 4 client.25954 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e389fd740 1 -- 172.21.15.7:0/25702252 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570b61bc990 con 0x5570b6260b10 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.682+0000 7f3e2f400640 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5570b6260b10 0x5570b6262f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.683+0000 7f3e2f400640 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5570b6260b10 0x5570b6262f90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99c30de3c280192c server_cookie=e5a01939cd3374cb in_seq=0 out_seq=0 2024-03-18T01:39:51.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.683+0000 7f3e1cc00640 10 client.25954 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e1cc00640 1 -- 172.21.15.7:0/25702252 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570b61bc990 con 0x5570b6260b10 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e1cc00640 10 client.25954 handle_command_reply: tid=0 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 client.25954 shutdown 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 -- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 msgr2=0x5570b6255330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b6255330 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3e180ee5c0 tx=0x7f3e1802c5d0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 -- 172.21.15.7:0/25702252 shutdown_connections 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b62505f0 0x5570b625c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5570b6260b10 0x5570b6262f90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624b480 0x5570b6255330 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 --2- 172.21.15.7:0/25702252 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624aa90 0x5570b6254df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 -- 172.21.15.7:0/25702252 >> 172.21.15.7:0/25702252 conn(0x5570b6246000 msgr2=0x5570b6245220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e1cc00640 0 client.25954 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 -- 172.21.15.7:0/25702252 shutdown_connections 2024-03-18T01:39:51.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 1 -- 172.21.15.7:0/25702252 wait complete. 2024-03-18T01:39:51.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.684+0000 7f3e389fd740 20 client.25954 trim_cache size 0 max 16384 2024-03-18T01:39:51.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.698+0000 7f3e389fd740 1 Processor -- start 2024-03-18T01:39:51.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e389fd740 1 -- start start 2024-03-18T01:39:51.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624aa90 0x5570b624c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624b480 0x5570b624ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b62505f0 0x5570b625c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e389fd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301c9f30 con 0x5570b62505f0 2024-03-18T01:39:51.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e389fd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301cbad0 con 0x5570b624b480 2024-03-18T01:39:51.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e389fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301ca600 con 0x5570b624aa90 2024-03-18T01:39:51.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624aa90 0x5570b624c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e26a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624b480 0x5570b624ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e26a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624b480 0x5570b624ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55074/0 (socket says 172.21.15.7:55074) 2024-03-18T01:39:51.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e26a00640 1 -- 172.21.15.7:0/3442738496 learned_addr learned my addr 172.21.15.7:0/3442738496 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2f400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624aa90 0x5570b624c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45168/0 (socket says 172.21.15.7:45168) 2024-03-18T01:39:51.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b62505f0 0x5570b625c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2f400640 1 -- 172.21.15.7:0/3442738496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624b480 msgr2=0x5570b624ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2f400640 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624b480 0x5570b624ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2f400640 1 -- 172.21.15.7:0/3442738496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b62505f0 msgr2=0x5570b625c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2f400640 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b62505f0 0x5570b625c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2f400640 1 -- 172.21.15.7:0/3442738496 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570b6245e10 con 0x5570b624aa90 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e26a00640 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624b480 0x5570b624ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.699+0000 7f3e2fe00640 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b62505f0 0x5570b625c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.700+0000 7f3e2f400640 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624aa90 0x5570b624c920 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5570b62637d0 tx=0x7f3e2001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c87c9f7ab9a5270c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.700+0000 7f3e1f400640 1 -- 172.21.15.7:0/3442738496 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20003e00 con 0x5570b624aa90 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.700+0000 7f3e1f400640 1 -- 172.21.15.7:0/3442738496 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e20011d20 con 0x5570b624aa90 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.700+0000 7f3e1f400640 1 -- 172.21.15.7:0/3442738496 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20005020 con 0x5570b624aa90 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 -- 172.21.15.7:0/3442738496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624aa90 msgr2=0x5570b624c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624aa90 0x5570b624c920 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x5570b62637d0 tx=0x7f3e2001a000 comp rx=0 tx=0).stop 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 -- 172.21.15.7:0/3442738496 shutdown_connections 2024-03-18T01:39:51.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b62505f0 0x5570b625c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624b480 0x5570b624ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 --2- 172.21.15.7:0/3442738496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b624aa90 0x5570b624c920 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 -- 172.21.15.7:0/3442738496 >> 172.21.15.7:0/3442738496 conn(0x5570b6246000 msgr2=0x5570b62463e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 -- 172.21.15.7:0/3442738496 shutdown_connections 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 -- 172.21.15.7:0/3442738496 wait complete. 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 -- start start 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.701+0000 7f3e389fd740 1 Processor -- start 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624aa90 0x5570b62598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624b480 0x5570b6259e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e389fd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b62505f0 0x5570b625ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e389fd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301cb3a0 con 0x5570b624b480 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e389fd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e301cacd0 con 0x5570b624aa90 2024-03-18T01:39:51.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e389fd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30073420 con 0x5570b62505f0 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e389fd740 10 client.0 _ll_register_callbacks cb 0x7f3e18005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e2f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624aa90 0x5570b62598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e2f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624aa90 0x5570b62598e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55090/0 (socket says 172.21.15.7:55090) 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e2f400640 1 -- 172.21.15.7:0/2022596448 learned_addr learned my addr 172.21.15.7:0/2022596448 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e2fe00640 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b62505f0 0x5570b625ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e26a00640 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624b480 0x5570b6259e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e2f400640 1 -- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b62505f0 msgr2=0x5570b625ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.702+0000 7f3e2f400640 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b62505f0 0x5570b625ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e2f400640 1 -- 172.21.15.7:0/2022596448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624b480 msgr2=0x5570b6259e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e2f400640 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624b480 0x5570b6259e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e2f400640 1 -- 172.21.15.7:0/2022596448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570b625b420 con 0x5570b624aa90 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e2fe00640 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b62505f0 0x5570b625ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e2f400640 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624aa90 0x5570b62598e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3e20011ff0 tx=0x7f3e2000fad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d769b5c0dc93ba3d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:51.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:51.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e1cc00640 1 -- 172.21.15.7:0/2022596448 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2002a5f0 con 0x5570b624aa90 2024-03-18T01:39:51.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e389fd740 1 -- 172.21.15.7:0/2022596448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e20011d20 con 0x5570b624aa90 2024-03-18T01:39:51.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e1cc00640 1 -- 172.21.15.7:0/2022596448 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e2002f040 con 0x5570b624aa90 2024-03-18T01:39:51.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e1cc00640 1 -- 172.21.15.7:0/2022596448 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3e2002a790 con 0x5570b624aa90 2024-03-18T01:39:51.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e389fd740 10 client.25985 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:51.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.703+0000 7f3e389fd740 1 -- 172.21.15.7:0/2022596448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5570b62c6e90 con 0x5570b624aa90 2024-03-18T01:39:51.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.704+0000 7f3e1cc00640 1 -- 172.21.15.7:0/2022596448 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e20011d20 con 0x5570b624aa90 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.704+0000 7f3e1cc00640 1 -- 172.21.15.7:0/2022596448 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3e20004490 con 0x5570b624aa90 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.704+0000 7f3e389fd740 10 client.25985 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.704+0000 7f3e389fd740 10 client.25985 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.704+0000 7f3e389fd740 20 client.25985 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.704+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5570b62c7620 0x5570b62c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.704+0000 7f3e389fd740 4 client.25985 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.704+0000 7f3e389fd740 1 -- 172.21.15.7:0/2022596448 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5570b61b6ce0 con 0x5570b62c7620 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.705+0000 7f3e26a00640 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5570b62c7620 0x5570b62c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.705+0000 7f3e26a00640 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5570b62c7620 0x5570b62c9a40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=508b540314452b15 server_cookie=8f364e4bafc5a2c4 in_seq=0 out_seq=0 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.705+0000 7f3e1cc00640 10 client.25985 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:39:51.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.705+0000 7f3e1cc00640 1 -- 172.21.15.7:0/2022596448 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5570b61b6ce0 con 0x5570b62c7620 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.705+0000 7f3e1cc00640 10 client.25985 handle_command_reply: tid=0 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.705+0000 7f3e389fd740 1 client.25985 shutdown 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 -- 172.21.15.7:0/2022596448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624aa90 msgr2=0x5570b62598e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624aa90 0x5570b62598e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3e20011ff0 tx=0x7f3e2000fad0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 -- 172.21.15.7:0/2022596448 shutdown_connections 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x5570b62c7620 0x5570b62c9a40 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5570b62505f0 0x5570b625ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5570b624b480 0x5570b6259e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e1cc00640 0 client.25985 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 --2- 172.21.15.7:0/2022596448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b624aa90 0x5570b62598e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 -- 172.21.15.7:0/2022596448 >> 172.21.15.7:0/2022596448 conn(0x5570b6246000 msgr2=0x5570b6254d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 -- 172.21.15.7:0/2022596448 shutdown_connections 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 1 -- 172.21.15.7:0/2022596448 wait complete. 2024-03-18T01:39:51.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.706+0000 7f3e389fd740 20 client.25985 trim_cache size 0 max 16384 2024-03-18T01:39:51.712 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 -- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e140a2870 msgr2=0x7f3e140a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e140a2870 0x7f3e140a4d30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x5570b624b880 tx=0x7f3e20009500 comp rx=0 tx=0).stop 2024-03-18T01:39:51.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 -- 172.21.15.7:0/2445206546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e30075820 msgr2=0x7f3e3019e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e30075820 0x7f3e3019e4a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e18000c00 tx=0x7f3e1802c650 comp rx=0 tx=0).stop 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 -- 172.21.15.7:0/2445206546 shutdown_connections 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e140a2870 0x7f3e140a4d30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e3010dbf0 0x7f3e301a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e30075820 0x7f3e3019e4a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 --2- 172.21.15.7:0/2445206546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e30074240 0x7f3e3019df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 -- 172.21.15.7:0/2445206546 >> 172.21.15.7:0/2445206546 conn(0x7f3e300fcb90 msgr2=0x7f3e30100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 -- 172.21.15.7:0/2445206546 shutdown_connections 2024-03-18T01:39:51.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:51.707+0000 7f3e34a00640 1 -- 172.21.15.7:0/2445206546 wait complete. 2024-03-18T01:39:51.727 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T01:39:51.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:51 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/509475889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:39:51.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:51 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/509475889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:39:52.020 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.009+0000 7fb2e2a00640 1 -- 172.21.15.7:0/1611950463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2dc06b7a0 msgr2=0x7fb2dc06bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.009+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/1611950463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2dc06b7a0 0x7fb2dc06bbc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009860 tx=0x7fb2c802bb90 comp rx=0 tx=0).stop 2024-03-18T01:39:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.020+0000 7fb2e2a00640 1 -- 172.21.15.7:0/1611950463 shutdown_connections 2024-03-18T01:39:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.020+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/1611950463 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2dc06d380 0x7fb2dc071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.020+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/1611950463 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2dc06c9e0 0x7fb2dc06ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.020+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/1611950463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2dc06b7a0 0x7fb2dc06bbc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.020+0000 7fb2e2a00640 1 -- 172.21.15.7:0/1611950463 >> 172.21.15.7:0/1611950463 conn(0x7fb2dc0fbc10 msgr2=0x7fb2dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.024+0000 7fb2e2a00640 1 -- 172.21.15.7:0/1611950463 shutdown_connections 2024-03-18T01:39:52.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.024+0000 7fb2e2a00640 1 -- 172.21.15.7:0/1611950463 wait complete. 2024-03-18T01:39:52.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.026+0000 7fb2e2a00640 1 Processor -- start 2024-03-18T01:39:52.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.027+0000 7fb2e2a00640 1 -- start start 2024-03-18T01:39:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.030+0000 7fb2e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2dc06b7a0 0x7fb2dc119410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.030+0000 7fb2e2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2dc06c9e0 0x7fb2dc119950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.030+0000 7fb2e2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2dc06d380 0x7fb2dc120920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.035+0000 7fb2e0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2dc06d380 0x7fb2dc120920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.035+0000 7fb2e0a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2dc06d380 0x7fb2dc120920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45180/0 (socket says 172.21.15.7:45180) 2024-03-18T01:39:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.036+0000 7fb2e0a00640 1 -- 172.21.15.7:0/433668579 learned_addr learned my addr 172.21.15.7:0/433668579 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.036+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2dc06b7a0 0x7fb2dc119410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.037+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc113a80 con 0x7fb2dc06c9e0 2024-03-18T01:39:52.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.038+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc113900 con 0x7fb2dc06b7a0 2024-03-18T01:39:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.038+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc113c00 con 0x7fb2dc06d380 2024-03-18T01:39:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.039+0000 7fb2e0a00640 1 -- 172.21.15.7:0/433668579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2dc06b7a0 msgr2=0x7fb2dc119410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.039+0000 7fb2e0a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2dc06b7a0 0x7fb2dc119410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.039+0000 7fb2db400640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2dc06c9e0 0x7fb2dc119950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.039+0000 7fb2e0a00640 1 -- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2dc06c9e0 msgr2=0x7fb2dc119950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.039+0000 7fb2e0a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2dc06c9e0 0x7fb2dc119950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.039+0000 7fb2e0a00640 1 -- 172.21.15.7:0/433668579 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2dc120f90 con 0x7fb2dc06d380 2024-03-18T01:39:52.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.039+0000 7fb2db400640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2dc06c9e0 0x7fb2dc119950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:52.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.040+0000 7fb2e0a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2dc06d380 0x7fb2dc120920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009fd0 tx=0x7fb2d400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.041+0000 7fb2d8c00640 1 -- 172.21.15.7:0/433668579 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4004b20 con 0x7fb2dc06d380 2024-03-18T01:39:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.041+0000 7fb2d8c00640 1 -- 172.21.15.7:0/433668579 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d4003e50 con 0x7fb2dc06d380 2024-03-18T01:39:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.041+0000 7fb2d8c00640 1 -- 172.21.15.7:0/433668579 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d400b630 con 0x7fb2dc06d380 2024-03-18T01:39:52.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.041+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2dc121280 con 0x7fb2dc06d380 2024-03-18T01:39:52.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.042+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2dc121710 con 0x7fb2dc06d380 2024-03-18T01:39:52.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.059+0000 7fb2e579f740 1 Processor -- start 2024-03-18T01:39:52.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.060+0000 7fb2d8c00640 1 -- 172.21.15.7:0/433668579 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fb2d401d020 con 0x7fb2dc06d380 2024-03-18T01:39:52.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.061+0000 7fb2e579f740 1 -- start start 2024-03-18T01:39:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.061+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd198c0 0x564c3bd19ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.061+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd1edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.061+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 0x564c3bd218b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.061+0000 7fb2e579f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564c3bce13b0 con 0x564c3bd198c0 2024-03-18T01:39:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.061+0000 7fb2e579f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564c3bc7e450 con 0x564c3bd1f420 2024-03-18T01:39:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.061+0000 7fb2e579f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564c3bc26cb0 con 0x564c3bd1a2b0 2024-03-18T01:39:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.062+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd1edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.062+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd1edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45182/0 (socket says 172.21.15.7:45182) 2024-03-18T01:39:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.062+0000 7fb2dbe00640 1 -- 172.21.15.7:0/1873675358 learned_addr learned my addr 172.21.15.7:0/1873675358 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.062+0000 7fb2d8c00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2c00a2950 0x7fb2c00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.063+0000 7fb2d8c00640 1 -- 172.21.15.7:0/433668579 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb2d40cec60 con 0x7fb2dc06d380 2024-03-18T01:39:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.063+0000 7fb2dbe00640 1 -- 172.21.15.7:0/1873675358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 msgr2=0x564c3bd218b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:39:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.063+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/1873675358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 0x564c3bd218b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.063+0000 7fb2dbe00640 1 -- 172.21.15.7:0/1873675358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd198c0 msgr2=0x564c3bd19ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:39:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.063+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/1873675358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd198c0 0x564c3bd19ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.063+0000 7fb2dbe00640 1 -- 172.21.15.7:0/1873675358 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c3bc481f0 con 0x564c3bd1a2b0 2024-03-18T01:39:52.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.065+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/1873675358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd1edb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8034a70 tx=0x7fb2c8009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58cb2b6e21f57b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.066+0000 7fb2d1200640 1 -- 172.21.15.7:0/1873675358 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8039040 con 0x564c3bd1a2b0 2024-03-18T01:39:52.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.066+0000 7fb2d1200640 1 -- 172.21.15.7:0/1873675358 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c8034640 con 0x564c3bd1a2b0 2024-03-18T01:39:52.068 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.066+0000 7fb2d1200640 1 -- 172.21.15.7:0/1873675358 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8004920 con 0x564c3bd1a2b0 2024-03-18T01:39:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.068+0000 7fb2e579f740 1 -- 172.21.15.7:0/1873675358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 msgr2=0x564c3bd1edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.068+0000 7fb2e579f740 1 --2- 172.21.15.7:0/1873675358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd1edb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8034a70 tx=0x7fb2c8009520 comp rx=0 tx=0).stop 2024-03-18T01:39:52.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.070+0000 7fb2e579f740 1 -- 172.21.15.7:0/1873675358 shutdown_connections 2024-03-18T01:39:52.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.070+0000 7fb2e579f740 1 --2- 172.21.15.7:0/1873675358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 0x564c3bd218b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.070+0000 7fb2e579f740 1 --2- 172.21.15.7:0/1873675358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd1edb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.070+0000 7fb2e579f740 1 --2- 172.21.15.7:0/1873675358 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd198c0 0x564c3bd19ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.070+0000 7fb2e579f740 1 -- 172.21.15.7:0/1873675358 >> 172.21.15.7:0/1873675358 conn(0x564c3bd14e30 msgr2=0x564c3bd15230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.071+0000 7fb2e0a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2c00a2950 0x7fb2c00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.072+0000 7fb2e579f740 1 -- 172.21.15.7:0/1873675358 shutdown_connections 2024-03-18T01:39:52.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.072+0000 7fb2e579f740 1 -- 172.21.15.7:0/1873675358 wait complete. 2024-03-18T01:39:52.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.073+0000 7fb2e579f740 1 -- start start 2024-03-18T01:39:52.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.073+0000 7fb2e579f740 1 Processor -- start 2024-03-18T01:39:52.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd198c0 0x564c3bd23b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd24090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd2b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2e579f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564c3bc3a110 con 0x564c3bd1f420 2024-03-18T01:39:52.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2e579f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564c3bc7a4f0 con 0x564c3bd198c0 2024-03-18T01:39:52.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2e579f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564c3bd22370 con 0x564c3bd1a2b0 2024-03-18T01:39:52.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2e579f740 10 client.0 _ll_register_callbacks cb 0x564c3bc27b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:52.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd24090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd24090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45188/0 (socket says 172.21.15.7:45188) 2024-03-18T01:39:52.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.077+0000 7fb2dbe00640 1 -- 172.21.15.7:0/2799606279 learned_addr learned my addr 172.21.15.7:0/2799606279 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.078+0000 7fb2dbe00640 1 -- 172.21.15.7:0/2799606279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd198c0 msgr2=0x564c3bd23b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:39:52.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.078+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd198c0 0x564c3bd23b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.078+0000 7fb2dbe00640 1 -- 172.21.15.7:0/2799606279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 msgr2=0x564c3bd2b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.079+0000 7fb2db400640 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd2b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.079+0000 7fb2e0a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2c00a2950 0x7fb2c00a4e10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4004990 tx=0x7fb2d4004310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.079+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd2b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.079+0000 7fb2dbe00640 1 -- 172.21.15.7:0/2799606279 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c3b9923f0 con 0x564c3bd1a2b0 2024-03-18T01:39:52.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.082+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd24090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb2c802c1a0 tx=0x7fb2c8034440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a105f1b41669ac64 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.083+0000 7fb2c5600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:52.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.083+0000 7fb2c5600640 1 -- 172.21.15.7:0/2799606279 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8039050 con 0x564c3bd1a2b0 2024-03-18T01:39:52.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.083+0000 7fb2c5600640 1 -- 172.21.15.7:0/2799606279 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb2c80046c0 con 0x564c3bd1a2b0 2024-03-18T01:39:52.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.085+0000 7fb2e579f740 1 -- 172.21.15.7:0/2799606279 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c8034640 con 0x564c3bd1a2b0 2024-03-18T01:39:52.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.086+0000 7fb2c5600640 1 -- 172.21.15.7:0/2799606279 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb2c8004900 con 0x564c3bd1a2b0 2024-03-18T01:39:52.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.086+0000 7fb2e579f740 10 client.25972 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.086+0000 7fb2e579f740 1 -- 172.21.15.7:0/2799606279 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564c3ba5c7d0 con 0x564c3bd1a2b0 2024-03-18T01:39:52.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.087+0000 7fb2c5600640 1 -- 172.21.15.7:0/2799606279 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c8034640 con 0x564c3bd1a2b0 2024-03-18T01:39:52.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.089+0000 7fb2c5600640 1 -- 172.21.15.7:0/2799606279 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb2c8003b20 con 0x564c3bd1a2b0 2024-03-18T01:39:52.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.090+0000 7fb2e579f740 10 client.25972 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:52.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.090+0000 7fb2e579f740 10 client.25972 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:39:52.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.090+0000 7fb2e579f740 20 client.25972 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:52.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.090+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x564c3bd2f5c0 0x564c3bd319b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.090+0000 7fb2e579f740 4 client.25972 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:52.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.090+0000 7fb2e579f740 1 -- 172.21.15.7:0/2799606279 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c3bc8b990 con 0x564c3bd2f5c0 2024-03-18T01:39:52.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.091+0000 7fb2db400640 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x564c3bd2f5c0 0x564c3bd319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.092+0000 7fb2c5600640 10 client.25972 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:39:52.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.092+0000 7fb2db400640 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x564c3bd2f5c0 0x564c3bd319b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f175d14f1640f35 server_cookie=40f5ff86a8def99c in_seq=0 out_seq=0 2024-03-18T01:39:52.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.094+0000 7fb2c5600640 1 -- 172.21.15.7:0/2799606279 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564c3bc8b990 con 0x564c3bd2f5c0 2024-03-18T01:39:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.094+0000 7fb2c5600640 10 client.25972 handle_command_reply: tid=0 2024-03-18T01:39:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.094+0000 7fb2e579f740 1 client.25972 shutdown 2024-03-18T01:39:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.094+0000 7fb2e579f740 1 -- 172.21.15.7:0/2799606279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 msgr2=0x564c3bd24090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.094+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd24090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb2c802c1a0 tx=0x7fb2c8034440 comp rx=0 tx=0).stop 2024-03-18T01:39:52.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.094+0000 7fb2e579f740 1 -- 172.21.15.7:0/2799606279 shutdown_connections 2024-03-18T01:39:52.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.095+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x564c3bd2f5c0 0x564c3bd319b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.095+0000 7fb2c5600640 0 client.25972 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:39:52.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.095+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd2b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.095+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd1a2b0 0x564c3bd24090 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.095+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2799606279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd198c0 0x564c3bd23b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.096+0000 7fb2e579f740 1 -- 172.21.15.7:0/2799606279 >> 172.21.15.7:0/2799606279 conn(0x564c3bd14e30 msgr2=0x564c3bd141f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.096+0000 7fb2e579f740 1 -- 172.21.15.7:0/2799606279 shutdown_connections 2024-03-18T01:39:52.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.096+0000 7fb2e579f740 1 -- 172.21.15.7:0/2799606279 wait complete. 2024-03-18T01:39:52.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.098+0000 7fb2e579f740 20 client.25972 trim_cache size 0 max 16384 2024-03-18T01:39:52.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2e579f740 1 Processor -- start 2024-03-18T01:39:52.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2e579f740 1 -- start start 2024-03-18T01:39:52.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd14070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1a2b0 0x564c3bd2b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd361d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2e579f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1c18f0 con 0x564c3bd1f420 2024-03-18T01:39:52.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2e579f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1c3490 con 0x564c3bd1a2b0 2024-03-18T01:39:52.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2e579f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1c1fc0 con 0x564c3bd198c0 2024-03-18T01:39:52.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.130+0000 7fb2db400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd14070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1a2b0 0x564c3bd2b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1a2b0 0x564c3bd2b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55136/0 (socket says 172.21.15.7:55136) 2024-03-18T01:39:52.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2dbe00640 1 -- 172.21.15.7:0/373223705 learned_addr learned my addr 172.21.15.7:0/373223705 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2db400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd14070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45196/0 (socket says 172.21.15.7:45196) 2024-03-18T01:39:52.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2db400640 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2dbe00640 1 -- 172.21.15.7:0/373223705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 msgr2=0x564c3bd14070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd14070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2dbe00640 1 -- 172.21.15.7:0/373223705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 msgr2=0x564c3bd361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd361d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.131+0000 7fb2dbe00640 1 -- 172.21.15.7:0/373223705 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c3bd10c10 con 0x564c3bd1a2b0 2024-03-18T01:39:52.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.132+0000 7fb2db400640 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:52.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.132+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1a2b0 0x564c3bd2b090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8001eb0 tx=0x7fb2c8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef2701141713135 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.132+0000 7fb2c7e00640 1 -- 172.21.15.7:0/373223705 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8004900 con 0x564c3bd1a2b0 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.132+0000 7fb2c7e00640 1 -- 172.21.15.7:0/373223705 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c804b070 con 0x564c3bd1a2b0 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.132+0000 7fb2c7e00640 1 -- 172.21.15.7:0/373223705 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8034710 con 0x564c3bd1a2b0 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.133+0000 7fb2e579f740 1 -- 172.21.15.7:0/373223705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1a2b0 msgr2=0x564c3bd2b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.133+0000 7fb2e579f740 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1a2b0 0x564c3bd2b090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8001eb0 tx=0x7fb2c8005850 comp rx=0 tx=0).stop 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.133+0000 7fb2e579f740 1 -- 172.21.15.7:0/373223705 shutdown_connections 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.133+0000 7fb2e579f740 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1f420 0x564c3bd361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.133+0000 7fb2e579f740 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1a2b0 0x564c3bd2b090 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.133+0000 7fb2e579f740 1 --2- 172.21.15.7:0/373223705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd14070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.133+0000 7fb2e579f740 1 -- 172.21.15.7:0/373223705 >> 172.21.15.7:0/373223705 conn(0x564c3bd14e30 msgr2=0x564c3bd13960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.134+0000 7fb2e579f740 1 -- 172.21.15.7:0/373223705 shutdown_connections 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.134+0000 7fb2e579f740 1 -- 172.21.15.7:0/373223705 wait complete. 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.134+0000 7fb2e579f740 1 -- start start 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.135+0000 7fb2e579f740 1 Processor -- start 2024-03-18T01:39:52.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.135+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd287b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.135+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1a2b0 0x564c3bd28cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.135+0000 7fb2e579f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 0x564c3bd29d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.135+0000 7fb2e579f740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1c2d60 con 0x564c3bd1a2b0 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.135+0000 7fb2e579f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1c2690 con 0x564c3bd1f420 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.135+0000 7fb2e579f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1bf890 con 0x564c3bd198c0 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.135+0000 7fb2e579f740 10 client.0 _ll_register_callbacks cb 0x7fb2c80096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.136+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1a2b0 0x564c3bd28cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.136+0000 7fb2db400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd287b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.136+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1a2b0 0x564c3bd28cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49272/0 (socket says 172.21.15.7:49272) 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.136+0000 7fb2dbe00640 1 -- 172.21.15.7:0/2542727885 learned_addr learned my addr 172.21.15.7:0/2542727885 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.136+0000 7fb2db400640 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 0x564c3bd29d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.137+0000 7fb2dbe00640 1 -- 172.21.15.7:0/2542727885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 msgr2=0x564c3bd287b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.137+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd287b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.137+0000 7fb2dbe00640 1 -- 172.21.15.7:0/2542727885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 msgr2=0x564c3bd29d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.137+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 0x564c3bd29d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.137+0000 7fb2dbe00640 1 -- 172.21.15.7:0/2542727885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c3bd224f0 con 0x564c3bd1a2b0 2024-03-18T01:39:52.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.137+0000 7fb2dbe00640 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1a2b0 0x564c3bd28cf0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2c802c0d0 tx=0x7fb2c8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6221ac0f98f3664 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.138+0000 7fb2c5600640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:39:52.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.138+0000 7fb2c5600640 1 -- 172.21.15.7:0/2542727885 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8060070 con 0x564c3bd1a2b0 2024-03-18T01:39:52.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.138+0000 7fb2c5600640 1 -- 172.21.15.7:0/2542727885 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb2c802cc60 con 0x564c3bd1a2b0 2024-03-18T01:39:52.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.138+0000 7fb2e579f740 1 -- 172.21.15.7:0/2542727885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c804b070 con 0x564c3bd1a2b0 2024-03-18T01:39:52.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.139+0000 7fb2c5600640 1 -- 172.21.15.7:0/2542727885 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb2c8004900 con 0x564c3bd1a2b0 2024-03-18T01:39:52.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.139+0000 7fb2e579f740 10 client.15795 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:52.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.139+0000 7fb2e579f740 1 -- 172.21.15.7:0/2542727885 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564c3bd2b390 con 0x564c3bd1a2b0 2024-03-18T01:39:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.139+0000 7fb2c5600640 1 -- 172.21.15.7:0/2542727885 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c804b070 con 0x564c3bd1a2b0 2024-03-18T01:39:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.140+0000 7fb2c5600640 1 -- 172.21.15.7:0/2542727885 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb2c8063020 con 0x564c3bd1a2b0 2024-03-18T01:39:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.140+0000 7fb2e579f740 10 client.15795 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.140+0000 7fb2e579f740 10 client.15795 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:39:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.140+0000 7fb2e579f740 20 client.15795 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.141+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x564c3bd2f5c0 0x564c3bd97d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.141+0000 7fb2e579f740 4 client.15795 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:52.142 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.141+0000 7fb2e579f740 1 -- 172.21.15.7:0/2542727885 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564c3bc85ce0 con 0x564c3bd2f5c0 2024-03-18T01:39:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.141+0000 7fb2db400640 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x564c3bd2f5c0 0x564c3bd97d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.141+0000 7fb2c5600640 10 client.15795 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:39:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.141+0000 7fb2db400640 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x564c3bd2f5c0 0x564c3bd97d10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c4a1f201b5aa131e server_cookie=63de87c43dce3957 in_seq=0 out_seq=0 2024-03-18T01:39:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2c5600640 1 -- 172.21.15.7:0/2542727885 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564c3bc85ce0 con 0x564c3bd2f5c0 2024-03-18T01:39:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2c5600640 10 client.15795 handle_command_reply: tid=0 2024-03-18T01:39:52.143 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 client.15795 shutdown 2024-03-18T01:39:52.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 -- 172.21.15.7:0/2542727885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1a2b0 msgr2=0x564c3bd28cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1a2b0 0x564c3bd28cf0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2c802c0d0 tx=0x7fb2c8004610 comp rx=0 tx=0).stop 2024-03-18T01:39:52.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 -- 172.21.15.7:0/2542727885 shutdown_connections 2024-03-18T01:39:52.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x564c3bd2f5c0 0x564c3bd97d10 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.144 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c3bd1f420 0x564c3bd29d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564c3bd1a2b0 0x564c3bd28cf0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 --2- 172.21.15.7:0/2542727885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564c3bd198c0 0x564c3bd287b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.142+0000 7fb2e579f740 1 -- 172.21.15.7:0/2542727885 >> 172.21.15.7:0/2542727885 conn(0x564c3bd14e30 msgr2=0x564c3bd141f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.143+0000 7fb2c5600640 0 client.15795 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:39:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.143+0000 7fb2e579f740 1 -- 172.21.15.7:0/2542727885 shutdown_connections 2024-03-18T01:39:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.143+0000 7fb2e579f740 1 -- 172.21.15.7:0/2542727885 wait complete. 2024-03-18T01:39:52.145 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.143+0000 7fb2e579f740 20 client.15795 trim_cache size 0 max 16384 2024-03-18T01:39:52.145 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.144+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2c00a2950 msgr2=0x7fb2c00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.144+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2c00a2950 0x7fb2c00a4e10 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4004990 tx=0x7fb2d4004310 comp rx=0 tx=0).stop 2024-03-18T01:39:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.144+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2dc06d380 msgr2=0x7fb2dc120920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.144+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2dc06d380 0x7fb2dc120920 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009fd0 tx=0x7fb2d400c800 comp rx=0 tx=0).stop 2024-03-18T01:39:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.145+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 shutdown_connections 2024-03-18T01:39:52.146 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.145+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2dc06d380 0x7fb2dc120920 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.145+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fb2c00a2950 0x7fb2c00a4e10 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.145+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fb2dc06c9e0 0x7fb2dc119950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.145+0000 7fb2e2a00640 1 --2- 172.21.15.7:0/433668579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2dc06b7a0 0x7fb2dc119410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.145+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 >> 172.21.15.7:0/433668579 conn(0x7fb2dc0fbc10 msgr2=0x7fb2dc0fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.145+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 shutdown_connections 2024-03-18T01:39:52.147 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.145+0000 7fb2e2a00640 1 -- 172.21.15.7:0/433668579 wait complete. 2024-03-18T01:39:52.172 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:39:52.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.496+0000 7ff5adc00640 1 -- 172.21.15.7:0/2682944964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a810fb20 msgr2=0x7ff5a8111f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.496+0000 7ff5adc00640 1 --2- 172.21.15.7:0/2682944964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a810fb20 0x7ff5a8111f10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000ac70 tx=0x7ff5a002c100 comp rx=0 tx=0).stop 2024-03-18T01:39:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.496+0000 7ff5adc00640 1 -- 172.21.15.7:0/2682944964 shutdown_connections 2024-03-18T01:39:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.496+0000 7ff5adc00640 1 --2- 172.21.15.7:0/2682944964 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a810fb20 0x7ff5a8111f10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.496+0000 7ff5adc00640 1 --2- 172.21.15.7:0/2682944964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8072cf0 0x7ff5a8073170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.496+0000 7ff5adc00640 1 --2- 172.21.15.7:0/2682944964 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a81093d0 0x7ff5a80727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.497+0000 7ff5adc00640 1 -- 172.21.15.7:0/2682944964 >> 172.21.15.7:0/2682944964 conn(0x7ff5a806e620 msgr2=0x7ff5a8070a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.497+0000 7ff5adc00640 1 -- 172.21.15.7:0/2682944964 shutdown_connections 2024-03-18T01:39:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.497+0000 7ff5adc00640 1 -- 172.21.15.7:0/2682944964 wait complete. 2024-03-18T01:39:52.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.498+0000 7ff5adc00640 1 Processor -- start 2024-03-18T01:39:52.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.499+0000 7ff5adc00640 1 -- start start 2024-03-18T01:39:52.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.503+0000 7ff5adc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8072cf0 0x7ff5a8119170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.503+0000 7ff5adc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a81093d0 0x7ff5a81196b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.503+0000 7ff5adc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a810fb20 0x7ff5a811e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.503+0000 7ff5adc00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81138c0 con 0x7ff5a810fb20 2024-03-18T01:39:52.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.503+0000 7ff5adc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a8113740 con 0x7ff5a8072cf0 2024-03-18T01:39:52.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.503+0000 7ff5adc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a8113a40 con 0x7ff5a81093d0 2024-03-18T01:39:52.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.504+0000 7ff5a6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a81093d0 0x7ff5a81196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.504+0000 7ff5a6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a81093d0 0x7ff5a81196b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45214/0 (socket says 172.21.15.7:45214) 2024-03-18T01:39:52.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.504+0000 7ff5a6000640 1 -- 172.21.15.7:0/3922713857 learned_addr learned my addr 172.21.15.7:0/3922713857 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.504+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a810fb20 0x7ff5a811e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.505+0000 7ff5a6000640 1 -- 172.21.15.7:0/3922713857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8072cf0 msgr2=0x7ff5a8119170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:39:52.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.505+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8072cf0 0x7ff5a8119170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.505+0000 7ff5a6000640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8072cf0 0x7ff5a8119170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.505+0000 7ff5a6000640 1 -- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a810fb20 msgr2=0x7ff5a811e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.505+0000 7ff5a6000640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a810fb20 0x7ff5a811e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.505+0000 7ff5a6000640 1 -- 172.21.15.7:0/3922713857 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a811ed80 con 0x7ff5a81093d0 2024-03-18T01:39:52.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.505+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a810fb20 0x7ff5a811e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:52.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.506+0000 7ff5a6000640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a81093d0 0x7ff5a81196b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff59400d770 tx=0x7ff59400dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.510+0000 7ff59f400640 1 -- 172.21.15.7:0/3922713857 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594017020 con 0x7ff5a81093d0 2024-03-18T01:39:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.510+0000 7ff59f400640 1 -- 172.21.15.7:0/3922713857 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff594004440 con 0x7ff5a81093d0 2024-03-18T01:39:52.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.510+0000 7ff59f400640 1 -- 172.21.15.7:0/3922713857 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594003d40 con 0x7ff5a81093d0 2024-03-18T01:39:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.511+0000 7ff5adc00640 1 -- 172.21.15.7:0/3922713857 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a811f070 con 0x7ff5a81093d0 2024-03-18T01:39:52.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.511+0000 7ff5adc00640 1 -- 172.21.15.7:0/3922713857 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a811f540 con 0x7ff5a81093d0 2024-03-18T01:39:52.526 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.525+0000 7ff5b09a5740 1 Processor -- start 2024-03-18T01:39:52.528 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.527+0000 7ff5b09a5740 1 -- start start 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.527+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dca60 0x55b2b59dce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.528+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.528+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59e25c0 0x55b2b59e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.528+0000 7ff5b09a5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b2b59a43b0 con 0x55b2b59e25c0 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.528+0000 7ff5b09a5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b2b5941450 con 0x55b2b59dd450 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.528+0000 7ff5b09a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b2b58e9cb0 con 0x55b2b59dca60 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.528+0000 7ff5a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dca60 0x55b2b59dce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.528+0000 7ff5a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dca60 0x55b2b59dce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45218/0 (socket says 172.21.15.7:45218) 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.528+0000 7ff5a7400640 1 -- 172.21.15.7:0/3369566584 learned_addr learned my addr 172.21.15.7:0/3369566584 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.529+0000 7ff59f400640 1 -- 172.21.15.7:0/3922713857 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7ff59401a020 con 0x7ff5a81093d0 2024-03-18T01:39:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.529+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.529+0000 7ff5a6a00640 1 -- 172.21.15.7:0/3369566584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dca60 msgr2=0x55b2b59dce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.529+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dca60 0x55b2b59dce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.530+0000 7ff5a6a00640 1 -- 172.21.15.7:0/3369566584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59e25c0 msgr2=0x55b2b59e4a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:39:52.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.530+0000 7ff5a6000640 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59e25c0 0x55b2b59e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.532+0000 7ff59f400640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a28f0 0x7ff58c0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.532+0000 7ff59f400640 1 -- 172.21.15.7:0/3922713857 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff5940ce0f0 con 0x7ff5a81093d0 2024-03-18T01:39:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.530+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59e25c0 0x55b2b59e4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.533+0000 7ff5a6a00640 1 -- 172.21.15.7:0/3369566584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2b590b1f0 con 0x55b2b59dd450 2024-03-18T01:39:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.533+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dca60 0x55b2b59dce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.534+0000 7ff5a6000640 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59e25c0 0x55b2b59e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:52.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.534+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e1f50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff598009ab0 tx=0x7ff598009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccbb2adcf50cdd60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.536 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.534+0000 7ff593e00640 1 -- 172.21.15.7:0/3369566584 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598016070 con 0x55b2b59dd450 2024-03-18T01:39:52.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.535+0000 7ff593e00640 1 -- 172.21.15.7:0/3369566584 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff598005500 con 0x55b2b59dd450 2024-03-18T01:39:52.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.535+0000 7ff593e00640 1 -- 172.21.15.7:0/3369566584 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598011470 con 0x55b2b59dd450 2024-03-18T01:39:52.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.540+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3369566584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 msgr2=0x55b2b59e1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.540+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e1f50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff598009ab0 tx=0x7ff598009f70 comp rx=0 tx=0).stop 2024-03-18T01:39:52.542 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.542+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a28f0 0x7ff58c0a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.543+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a28f0 0x7ff58c0a4db0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000aa00 tx=0x7ff5a0031330 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.543+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3369566584 shutdown_connections 2024-03-18T01:39:52.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.543+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59e25c0 0x55b2b59e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.543+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e1f50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.543+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3369566584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dca60 0x55b2b59dce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.543+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3369566584 >> 172.21.15.7:0/3369566584 conn(0x55b2b59d7fd0 msgr2=0x55b2b59d83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.543+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3369566584 shutdown_connections 2024-03-18T01:39:52.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.545+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3369566584 wait complete. 2024-03-18T01:39:52.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.545+0000 7ff5b09a5740 1 -- start start 2024-03-18T01:39:52.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.546+0000 7ff5b09a5740 1 Processor -- start 2024-03-18T01:39:52.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.551+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59e6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.553+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.553+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59e25c0 0x55b2b59ee320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.554+0000 7ff5a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59e25c0 0x55b2b59ee320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.554+0000 7ff5a7400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59e25c0 0x55b2b59ee320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45232/0 (socket says 172.21.15.7:45232) 2024-03-18T01:39:52.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.555+0000 7ff5a7400640 1 -- 172.21.15.7:0/726958024 learned_addr learned my addr 172.21.15.7:0/726958024 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.556+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b2b58fd110 con 0x55b2b59dca60 2024-03-18T01:39:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.556+0000 7ff5a6000640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.556+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b2b593d4f0 con 0x55b2b59dd450 2024-03-18T01:39:52.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.556+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b2b59e5570 con 0x55b2b59e25c0 2024-03-18T01:39:52.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.557+0000 7ff5b09a5740 10 client.0 _ll_register_callbacks cb 0x55b2b58eab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:52.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.557+0000 7ff5a7400640 1 -- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 msgr2=0x55b2b59e7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.559+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59e6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.560+0000 7ff5a7400640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.560+0000 7ff5a7400640 1 -- 172.21.15.7:0/726958024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 msgr2=0x55b2b59e6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.560+0000 7ff5a7400640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59e6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.560+0000 7ff5a7400640 1 -- 172.21.15.7:0/726958024 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2b56553f0 con 0x55b2b59e25c0 2024-03-18T01:39:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.560+0000 7ff5a6000640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.561+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59e6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:52.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.564+0000 7ff5a7400640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59e25c0 0x55b2b59ee320 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000ac40 tx=0x7ff5a002c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=329928af8018486e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.565+0000 7ff590c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:52.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.565+0000 7ff590c00640 1 -- 172.21.15.7:0/726958024 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a004e070 con 0x55b2b59e25c0 2024-03-18T01:39:52.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.565+0000 7ff590c00640 1 -- 172.21.15.7:0/726958024 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff5a0031aa0 con 0x55b2b59e25c0 2024-03-18T01:39:52.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.568+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff598005500 con 0x55b2b59e25c0 2024-03-18T01:39:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.568+0000 7ff590c00640 1 -- 172.21.15.7:0/726958024 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff5a0031ca0 con 0x55b2b59e25c0 2024-03-18T01:39:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.568+0000 7ff5b09a5740 10 client.25990 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:52.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.568+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b2b571f7d0 con 0x55b2b59e25c0 2024-03-18T01:39:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.570+0000 7ff590c00640 1 -- 172.21.15.7:0/726958024 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff598005500 con 0x55b2b59e25c0 2024-03-18T01:39:52.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.570+0000 7ff590c00640 1 -- 172.21.15.7:0/726958024 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ff5a0049070 con 0x55b2b59e25c0 2024-03-18T01:39:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.572+0000 7ff5b09a5740 10 client.25990 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.573+0000 7ff5b09a5740 10 client.25990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.573+0000 7ff5b09a5740 20 client.25990 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.573+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b2b59f2b30 0x55b2b59f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.573+0000 7ff5b09a5740 4 client.25990 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:52.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.573+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2b594e990 con 0x55b2b59f2b30 2024-03-18T01:39:52.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.573+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b2b59f2b30 0x55b2b59f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.575+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b2b59f2b30 0x55b2b59f4fc0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d0848b5c19a9674 server_cookie=607fc372c5f3c1d3 in_seq=0 out_seq=0 2024-03-18T01:39:52.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.579+0000 7ff590c00640 10 client.25990 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:52.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.579+0000 7ff590c00640 1 -- 172.21.15.7:0/726958024 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b2b594e990 con 0x55b2b59f2b30 2024-03-18T01:39:52.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.579+0000 7ff590c00640 10 client.25990 handle_command_reply: tid=0 2024-03-18T01:39:52.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.583+0000 7ff5b09a5740 1 client.25990 shutdown 2024-03-18T01:39:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59e25c0 msgr2=0x55b2b59ee320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59e25c0 0x55b2b59ee320 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000ac40 tx=0x7ff5a002c100 comp rx=0 tx=0).stop 2024-03-18T01:39:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 shutdown_connections 2024-03-18T01:39:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b2b59f2b30 0x55b2b59f4fc0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff590c00640 0 client.25990 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59e25c0 0x55b2b59ee320 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59dd450 0x55b2b59e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/726958024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59e6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.585+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 >> 172.21.15.7:0/726958024 conn(0x55b2b59d7fd0 msgr2=0x55b2b59d71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.586+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 shutdown_connections 2024-03-18T01:39:52.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.587+0000 7ff5b09a5740 1 -- 172.21.15.7:0/726958024 wait complete. 2024-03-18T01:39:52.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.587+0000 7ff5b09a5740 20 client.25990 trim_cache size 0 max 16384 2024-03-18T01:39:52.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.632+0000 7ff5b09a5740 1 Processor -- start 2024-03-18T01:39:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.633+0000 7ff5b09a5740 1 -- start start 2024-03-18T01:39:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.633+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59f9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.633+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 0x55b2b59ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.633+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 0x55b2b59f9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.633+0000 7ff5b09a5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81c1840 con 0x55b2b59dca60 2024-03-18T01:39:52.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.633+0000 7ff5b09a5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81c33e0 con 0x55b2b59e25c0 2024-03-18T01:39:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.633+0000 7ff5b09a5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81c1f10 con 0x55b2b59dd450 2024-03-18T01:39:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.634+0000 7ff5a6a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59f9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.634+0000 7ff5a6a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59f9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49328/0 (socket says 172.21.15.7:49328) 2024-03-18T01:39:52.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.634+0000 7ff5a6a00640 1 -- 172.21.15.7:0/645640531 learned_addr learned my addr 172.21.15.7:0/645640531 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.636+0000 7ff5a6a00640 1 -- 172.21.15.7:0/645640531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 msgr2=0x55b2b59ee2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:39:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.636+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/645640531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 0x55b2b59ee2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.636+0000 7ff5a6a00640 1 -- 172.21.15.7:0/645640531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 msgr2=0x55b2b59f9dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:39:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.636+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/645640531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 0x55b2b59f9dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.636+0000 7ff5a6a00640 1 -- 172.21.15.7:0/645640531 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2b59d7920 con 0x55b2b59dca60 2024-03-18T01:39:52.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.636+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/645640531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59f9880 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff598009ab0 tx=0x7ff598005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319934c23c497de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.637+0000 7ff593400640 1 -- 172.21.15.7:0/645640531 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff598016040 con 0x55b2b59dca60 2024-03-18T01:39:52.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.637+0000 7ff593400640 1 -- 172.21.15.7:0/645640531 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff598026e10 con 0x55b2b59dca60 2024-03-18T01:39:52.639 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.637+0000 7ff593400640 1 -- 172.21.15.7:0/645640531 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5980116d0 con 0x55b2b59dca60 2024-03-18T01:39:52.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.639+0000 7ff5b09a5740 1 -- 172.21.15.7:0/645640531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 msgr2=0x55b2b59f9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.639+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/645640531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59f9880 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff598009ab0 tx=0x7ff598005850 comp rx=0 tx=0).stop 2024-03-18T01:39:52.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.639+0000 7ff5b09a5740 1 -- 172.21.15.7:0/645640531 shutdown_connections 2024-03-18T01:39:52.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.639+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/645640531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 0x55b2b59f9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.639+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/645640531 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 0x55b2b59ee2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.639+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/645640531 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59f9880 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.639+0000 7ff5b09a5740 1 -- 172.21.15.7:0/645640531 >> 172.21.15.7:0/645640531 conn(0x55b2b59d7fd0 msgr2=0x55b2b59d71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.639+0000 7ff5b09a5740 1 -- 172.21.15.7:0/645640531 shutdown_connections 2024-03-18T01:39:52.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.640+0000 7ff5b09a5740 1 -- 172.21.15.7:0/645640531 wait complete. 2024-03-18T01:39:52.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.640+0000 7ff5b09a5740 1 -- start start 2024-03-18T01:39:52.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.640+0000 7ff5b09a5740 1 Processor -- start 2024-03-18T01:39:52.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.642+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59ec1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.642+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 0x55b2b59ec730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.643+0000 7ff5b09a5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 0x55b2b59ed680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.643+0000 7ff5a6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 0x55b2b59ec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.643+0000 7ff5a6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 0x55b2b59ec730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45244/0 (socket says 172.21.15.7:45244) 2024-03-18T01:39:52.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.643+0000 7ff5a6000640 1 -- 172.21.15.7:0/3587158260 learned_addr learned my addr 172.21.15.7:0/3587158260 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:52.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.643+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 0x55b2b59ed680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.644+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59ec1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.644+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81c2cb0 con 0x55b2b59dca60 2024-03-18T01:39:52.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.644+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81c25e0 con 0x55b2b59e25c0 2024-03-18T01:39:52.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.644+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a81bf820 con 0x55b2b59dd450 2024-03-18T01:39:52.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.645+0000 7ff5a7400640 1 -- 172.21.15.7:0/3587158260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 msgr2=0x55b2b59ec730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.645+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 0x55b2b59ec730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.645+0000 7ff5a7400640 1 -- 172.21.15.7:0/3587158260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 msgr2=0x55b2b59ec1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:52.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.645+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59ec1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.645+0000 7ff5a7400640 1 -- 172.21.15.7:0/3587158260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2b59e7690 con 0x55b2b59e25c0 2024-03-18T01:39:52.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.646+0000 7ff5a7400640 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 0x55b2b59ed680 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0034560 tx=0x7ff5a0031ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69a084e8ac6b194c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:52.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.646+0000 7ff590c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:52.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.646+0000 7ff590c00640 1 -- 172.21.15.7:0/3587158260 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0060020 con 0x55b2b59e25c0 2024-03-18T01:39:52.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.646+0000 7ff590c00640 1 -- 172.21.15.7:0/3587158260 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff5a005c450 con 0x55b2b59e25c0 2024-03-18T01:39:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.647+0000 7ff590c00640 10 client.0 inactive, discarding osd_map(91..91 src has 1..91) 2024-03-18T01:39:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.647+0000 7ff5b09a5740 10 client.0 _ll_register_callbacks cb 0x7ff5a00014f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.647+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff598026e10 con 0x55b2b59e25c0 2024-03-18T01:39:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.647+0000 7ff590c00640 1 -- 172.21.15.7:0/3587158260 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff5a004e040 con 0x55b2b59e25c0 2024-03-18T01:39:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.648+0000 7ff5b09a5740 10 client.26006 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:52.652 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.648+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55b2b5a591d0 con 0x55b2b59e25c0 2024-03-18T01:39:52.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.648+0000 7ff590c00640 1 -- 172.21.15.7:0/3587158260 <== mon.2 v2:172.21.15.46:3300/0 4 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7ff5a0049040 con 0x55b2b59e25c0 2024-03-18T01:39:52.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.648+0000 7ff5b09a5740 10 client.26006 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:52.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.648+0000 7ff5b09a5740 10 client.26006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:52.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.648+0000 7ff5b09a5740 20 client.26006 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:52.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.648+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b2b5a59ae0 0x55b2b5a5bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.649+0000 7ff5b09a5740 4 client.26006 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:39:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.649+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b2b5a59ae0 0x55b2b5a5bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:52.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.649+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff598026e10 con 0x55b2b5a59ae0 2024-03-18T01:39:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.650+0000 7ff5a6a00640 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b2b5a59ae0 0x55b2b5a5bfa0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=278ba74c64fb2d7a server_cookie=52c95bdcf83883c in_seq=0 out_seq=0 2024-03-18T01:39:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.650+0000 7ff590c00640 10 client.26006 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.650+0000 7ff590c00640 1 -- 172.21.15.7:0/3587158260 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff598026e10 con 0x55b2b5a59ae0 2024-03-18T01:39:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.650+0000 7ff590c00640 10 client.26006 handle_command_reply: tid=0 2024-03-18T01:39:52.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.650+0000 7ff5b09a5740 1 client.26006 shutdown 2024-03-18T01:39:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.650+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 msgr2=0x55b2b59ed680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.650+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 0x55b2b59ed680 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0034560 tx=0x7ff5a0031ad0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 shutdown_connections 2024-03-18T01:39:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b2b5a59ae0 0x55b2b5a5bfa0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b2b59e25c0 0x55b2b59ed680 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b2b59dd450 0x55b2b59ec730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 1 --2- 172.21.15.7:0/3587158260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b2b59dca60 0x55b2b59ec1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff590c00640 0 client.26006 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 >> 172.21.15.7:0/3587158260 conn(0x55b2b59d7fd0 msgr2=0x55b2b59db230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 shutdown_connections 2024-03-18T01:39:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 1 -- 172.21.15.7:0/3587158260 wait complete. 2024-03-18T01:39:52.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.651+0000 7ff5b09a5740 20 client.26006 trim_cache size 0 max 16384 2024-03-18T01:39:52.660 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:39:52.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 -- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a28f0 msgr2=0x7ff58c0a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a28f0 0x7ff58c0a4db0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000aa00 tx=0x7ff5a0031330 comp rx=0 tx=0).stop 2024-03-18T01:39:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 -- 172.21.15.7:0/3922713857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a81093d0 msgr2=0x7ff5a81196b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a81093d0 0x7ff5a81196b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff59400d770 tx=0x7ff59400dc30 comp rx=0 tx=0).stop 2024-03-18T01:39:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 -- 172.21.15.7:0/3922713857 shutdown_connections 2024-03-18T01:39:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7ff58c0a28f0 0x7ff58c0a4db0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7ff5a810fb20 0x7ff5a811e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a81093d0 0x7ff5a81196b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 --2- 172.21.15.7:0/3922713857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5a8072cf0 0x7ff5a8119170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:52.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 -- 172.21.15.7:0/3922713857 >> 172.21.15.7:0/3922713857 conn(0x7ff5a806e620 msgr2=0x7ff5a810bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 -- 172.21.15.7:0/3922713857 shutdown_connections 2024-03-18T01:39:52.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:52.654+0000 7ff5adc00640 1 -- 172.21.15.7:0/3922713857 wait complete. 2024-03-18T01:39:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:52 smithi046 ceph-mon[36108]: pgmap v468: 129 pgs: 129 active+clean; 3.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 269 MiB/s wr, 99 op/s 2024-03-18T01:39:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:52 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:39:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:52 smithi007 ceph-mon[31996]: pgmap v468: 129 pgs: 129 active+clean; 3.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 269 MiB/s wr, 99 op/s 2024-03-18T01:39:52.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:52 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:39:52.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:52 smithi203 ceph-mon[36345]: pgmap v468: 129 pgs: 129 active+clean; 3.7 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 269 MiB/s wr, 99 op/s 2024-03-18T01:39:52.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:52 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:39:53.682 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T01:39:53.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:39:53.682 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T01:39:53.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.938+0000 7f64e3800640 1 -- 172.21.15.7:0/2330940193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc102300 msgr2=0x7f64dc1046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.938+0000 7f64e3800640 1 --2- 172.21.15.7:0/2330940193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc102300 0x7f64dc1046f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f64d0009900 tx=0x7f64d002bc40 comp rx=0 tx=0).stop 2024-03-18T01:39:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.938+0000 7f64e3800640 1 -- 172.21.15.7:0/2330940193 shutdown_connections 2024-03-18T01:39:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.938+0000 7f64e3800640 1 --2- 172.21.15.7:0/2330940193 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64dc107560 0x7f64dc109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.938+0000 7f64e3800640 1 --2- 172.21.15.7:0/2330940193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64dc104c30 0x7f64dc107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.938+0000 7f64e3800640 1 --2- 172.21.15.7:0/2330940193 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc102300 0x7f64dc1046f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.938+0000 7f64e3800640 1 -- 172.21.15.7:0/2330940193 >> 172.21.15.7:0/2330940193 conn(0x7f64dc0fbc70 msgr2=0x7f64dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:53.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.939+0000 7f64e3800640 1 -- 172.21.15.7:0/2330940193 shutdown_connections 2024-03-18T01:39:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.939+0000 7f64e3800640 1 -- 172.21.15.7:0/2330940193 wait complete. 2024-03-18T01:39:53.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e3800640 1 Processor -- start 2024-03-18T01:39:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e3800640 1 -- start start 2024-03-18T01:39:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e3800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64dc102300 0x7f64dc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e3800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64dc104c30 0x7f64dc198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e3800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc107560 0x7f64dc19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e3800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10d4a0 con 0x7f64dc107560 2024-03-18T01:39:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e3800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10d320 con 0x7f64dc102300 2024-03-18T01:39:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e3800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc10d620 con 0x7f64dc104c30 2024-03-18T01:39:53.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.941+0000 7f64e1800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc107560 0x7f64dc19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.942+0000 7f64e1800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc107560 0x7f64dc19f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49360/0 (socket says 172.21.15.7:49360) 2024-03-18T01:39:53.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.942+0000 7f64e1800640 1 -- 172.21.15.7:0/2180739584 learned_addr learned my addr 172.21.15.7:0/2180739584 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:53.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.942+0000 7f64e1800640 1 -- 172.21.15.7:0/2180739584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64dc104c30 msgr2=0x7f64dc198090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:39:53.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.942+0000 7f64e1800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64dc104c30 0x7f64dc198090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.942+0000 7f64e1800640 1 -- 172.21.15.7:0/2180739584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64dc102300 msgr2=0x7f64dc197b50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:39:53.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.942+0000 7f64e1800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64dc102300 0x7f64dc197b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.942+0000 7f64e1800640 1 -- 172.21.15.7:0/2180739584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc06b330 con 0x7f64dc107560 2024-03-18T01:39:53.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.942+0000 7f64e1800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc107560 0x7f64dc19f100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009f60 tx=0x7f64d4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:53.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.943+0000 7f64d9600640 1 -- 172.21.15.7:0/2180739584 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d400cd80 con 0x7f64dc107560 2024-03-18T01:39:53.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.943+0000 7f64d9600640 1 -- 172.21.15.7:0/2180739584 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64d4013770 con 0x7f64dc107560 2024-03-18T01:39:53.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.943+0000 7f64d9600640 1 -- 172.21.15.7:0/2180739584 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d40054e0 con 0x7f64dc107560 2024-03-18T01:39:53.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.943+0000 7f64e3800640 1 -- 172.21.15.7:0/2180739584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc100150 con 0x7f64dc107560 2024-03-18T01:39:53.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.943+0000 7f64e3800640 1 -- 172.21.15.7:0/2180739584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64dc100650 con 0x7f64dc107560 2024-03-18T01:39:53.948 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.947+0000 7f64d9600640 1 -- 172.21.15.7:0/2180739584 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f64d4013920 con 0x7f64dc107560 2024-03-18T01:39:53.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64e6583740 1 Processor -- start 2024-03-18T01:39:53.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64e6583740 1 -- start start 2024-03-18T01:39:53.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64e6583740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d77e0 0x559b119d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64e6583740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119dccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.950 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64e6583740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119df7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64e6583740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559b1193af80 con 0x559b119d77e0 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64e6583740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559b118e3ad0 con 0x559b119dd340 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64e6583740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b11937020 con 0x559b119d81d0 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.948+0000 7f64d9600640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f64c40a2880 0x7f64c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.949+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d77e0 0x559b119d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.949+0000 7f64d9600640 1 -- 172.21.15.7:0/2180739584 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f64d40d3240 con 0x7f64dc107560 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.949+0000 7f64e1800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f64c40a2880 0x7f64c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.949+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119df7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.949+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119df7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55206/0 (socket says 172.21.15.7:55206) 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.949+0000 7f64dbe00640 1 -- 172.21.15.7:0/246030456 learned_addr learned my addr 172.21.15.7:0/246030456 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.949+0000 7f64e0e00640 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119dccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.950+0000 7f64e0e00640 1 -- 172.21.15.7:0/246030456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 msgr2=0x559b119df7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:53.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.950+0000 7f64e0e00640 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119df7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.950+0000 7f64e0e00640 1 -- 172.21.15.7:0/246030456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d77e0 msgr2=0x559b119d7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:53.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.950+0000 7f64e1800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f64c40a2880 0x7f64c40a4d40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009270 tx=0x7f64d4015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:53.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.950+0000 7f64e0e00640 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d77e0 0x559b119d7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.950+0000 7f64e0e00640 1 -- 172.21.15.7:0/246030456 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b118fb6f0 con 0x559b119d81d0 2024-03-18T01:39:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.950+0000 7f64dbe00640 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d77e0 0x559b119d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:39:53.955 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.950+0000 7f64e0e00640 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119dccd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f64d002c180 tx=0x7f64d0005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a6e900d77ad4ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64ce000640 1 -- 172.21.15.7:0/246030456 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0042020 con 0x559b119d81d0 2024-03-18T01:39:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64ce000640 1 -- 172.21.15.7:0/246030456 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64d0009520 con 0x559b119d81d0 2024-03-18T01:39:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64ce000640 1 -- 172.21.15.7:0/246030456 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d0003bb0 con 0x559b119d81d0 2024-03-18T01:39:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64e6583740 1 -- 172.21.15.7:0/246030456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 msgr2=0x559b119dccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64e6583740 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119dccd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f64d002c180 tx=0x7f64d0005780 comp rx=0 tx=0).stop 2024-03-18T01:39:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64e6583740 1 -- 172.21.15.7:0/246030456 shutdown_connections 2024-03-18T01:39:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64e6583740 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119df7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64e6583740 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119dccd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64e6583740 1 --2- 172.21.15.7:0/246030456 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d77e0 0x559b119d7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.951+0000 7f64e6583740 1 -- 172.21.15.7:0/246030456 >> 172.21.15.7:0/246030456 conn(0x559b1199f930 msgr2=0x559b119d3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.952+0000 7f64e6583740 1 -- 172.21.15.7:0/246030456 shutdown_connections 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.952+0000 7f64e6583740 1 -- 172.21.15.7:0/246030456 wait complete. 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.952+0000 7f64e6583740 1 -- start start 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.952+0000 7f64e6583740 1 Processor -- start 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64e6583740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119ea270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64e6583740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64e6583740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64e6583740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559b119e0290 con 0x559b119d81d0 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64e6583740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559b118f6f30 con 0x559b119dd340 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64e6583740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b119e0410 con 0x559b119d77e0 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119ea270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119ea270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45278/0 (socket says 172.21.15.7:45278) 2024-03-18T01:39:53.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64e6583740 10 client.0 _ll_register_callbacks cb 0x559b118e4bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:53.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64dbe00640 1 -- 172.21.15.7:0/2475622929 learned_addr learned my addr 172.21.15.7:0/2475622929 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:53.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.953+0000 7f64e0e00640 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.961 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64dbe00640 1 -- 172.21.15.7:0/2475622929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 msgr2=0x559b119f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64dbe00640 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64dbe00640 1 -- 172.21.15.7:0/2475622929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 msgr2=0x559b119ea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64dbe00640 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119ea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64dbe00640 1 -- 172.21.15.7:0/2475622929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b117177e0 con 0x559b119d77e0 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64dbe00640 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119ea270 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f64c8009f20 tx=0x7f64c8013b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da5df4a5335fab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64c2a00640 1 -- 172.21.15.7:0/2475622929 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8004230 con 0x559b119d77e0 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64c2a00640 1 -- 172.21.15.7:0/2475622929 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f64c80043d0 con 0x559b119d77e0 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.954+0000 7f64e6583740 1 -- 172.21.15.7:0/2475622929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64d0009520 con 0x559b119d77e0 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64c2a00640 1 -- 172.21.15.7:0/2475622929 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f64c800e7d0 con 0x559b119d77e0 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64e6583740 10 client.26002 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:53.962 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64e6583740 1 -- 172.21.15.7:0/2475622929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559b11777f70 con 0x559b119d77e0 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64c2a00640 1 -- 172.21.15.7:0/2475622929 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64d0009520 con 0x559b119d77e0 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64c2a00640 1 -- 172.21.15.7:0/2475622929 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f64c8003bc0 con 0x559b119d77e0 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64e6583740 10 client.26002 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64e6583740 10 client.26002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64e6583740 20 client.26002 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64e6583740 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559b119f6140 0x559b119f85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64e6583740 4 client.26002 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.955+0000 7f64e6583740 1 -- 172.21.15.7:0/2475622929 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b119484c0 con 0x559b119f6140 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.956+0000 7f64e0e00640 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559b119f6140 0x559b119f85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.963 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.956+0000 7f64e0e00640 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559b119f6140 0x559b119f85c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=102f34e4d5b6cb73 server_cookie=26217316239030c5 in_seq=0 out_seq=0 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.956+0000 7f64c2a00640 10 client.26002 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.957+0000 7f64c2a00640 1 -- 172.21.15.7:0/2475622929 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559b119484c0 con 0x559b119f6140 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.957+0000 7f64c2a00640 10 client.26002 handle_command_reply: tid=0 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.957+0000 7f64e6583740 1 client.26002 shutdown 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.957+0000 7f64e6583740 1 -- 172.21.15.7:0/2475622929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 msgr2=0x559b119ea270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.957+0000 7f64e6583740 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119ea270 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f64c8009f20 tx=0x7f64c8013b10 comp rx=0 tx=0).stop 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.957+0000 7f64e6583740 1 -- 172.21.15.7:0/2475622929 shutdown_connections 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.957+0000 7f64e6583740 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559b119f6140 0x559b119f85c0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.957+0000 7f64e6583740 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.958+0000 7f64e6583740 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.958+0000 7f64e6583740 1 --2- 172.21.15.7:0/2475622929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119ea270 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.958+0000 7f64e6583740 1 -- 172.21.15.7:0/2475622929 >> 172.21.15.7:0/2475622929 conn(0x559b1199f930 msgr2=0x559b119d2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.958+0000 7f64c2a00640 0 client.26002 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:53.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.958+0000 7f64e6583740 1 -- 172.21.15.7:0/2475622929 shutdown_connections 2024-03-18T01:39:53.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.958+0000 7f64e6583740 1 -- 172.21.15.7:0/2475622929 wait complete. 2024-03-18T01:39:53.965 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.958+0000 7f64e6583740 20 client.26002 trim_cache size 0 max 16384 2024-03-18T01:39:53.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.986+0000 7f64e6583740 1 Processor -- start 2024-03-18T01:39:53.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.986+0000 7f64e6583740 1 -- start start 2024-03-18T01:39:53.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64e6583740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119e9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64e6583740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.988 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64e6583740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119ea480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64e6583740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1b08e0 con 0x559b119d81d0 2024-03-18T01:39:53.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64e6583740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1b2480 con 0x559b119dd340 2024-03-18T01:39:53.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64e6583740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1b0fb0 con 0x559b119d77e0 2024-03-18T01:39:53.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119e9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119e9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45294/0 (socket says 172.21.15.7:45294) 2024-03-18T01:39:53.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64dbe00640 1 -- 172.21.15.7:0/2208004214 learned_addr learned my addr 172.21.15.7:0/2208004214 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:53.989 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.987+0000 7f64dbe00640 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119ea480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.988+0000 7f64e0e00640 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.988+0000 7f64dbe00640 1 -- 172.21.15.7:0/2208004214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 msgr2=0x559b119ea480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:53.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.988+0000 7f64dbe00640 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119ea480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.988+0000 7f64dbe00640 1 -- 172.21.15.7:0/2208004214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 msgr2=0x559b119f17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:53.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.988+0000 7f64dbe00640 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119f17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.990 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.988+0000 7f64dbe00640 1 -- 172.21.15.7:0/2208004214 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b1160f170 con 0x559b119d77e0 2024-03-18T01:39:53.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.988+0000 7f64e0e00640 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119f17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:53.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.988+0000 7f64dbe00640 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119e9f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f64c8005d80 tx=0x7f64c8005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75de9593f8e3b9ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:53.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.989+0000 7f64cd600640 1 -- 172.21.15.7:0/2208004214 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c802b030 con 0x559b119d77e0 2024-03-18T01:39:53.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.989+0000 7f64cd600640 1 -- 172.21.15.7:0/2208004214 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64c8017070 con 0x559b119d77e0 2024-03-18T01:39:53.991 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.989+0000 7f64cd600640 1 -- 172.21.15.7:0/2208004214 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8025b50 con 0x559b119d77e0 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 -- 172.21.15.7:0/2208004214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 msgr2=0x559b119e9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119e9f40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f64c8005d80 tx=0x7f64c8005e30 comp rx=0 tx=0).stop 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 -- 172.21.15.7:0/2208004214 shutdown_connections 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119dd340 0x559b119ea480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119d81d0 0x559b119f17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 --2- 172.21.15.7:0/2208004214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d77e0 0x559b119e9f40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 -- 172.21.15.7:0/2208004214 >> 172.21.15.7:0/2208004214 conn(0x559b1199f930 msgr2=0x559b119d94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 -- 172.21.15.7:0/2208004214 shutdown_connections 2024-03-18T01:39:53.992 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.990+0000 7f64e6583740 1 -- 172.21.15.7:0/2208004214 wait complete. 2024-03-18T01:39:53.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.991+0000 7f64e6583740 1 -- start start 2024-03-18T01:39:53.993 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.991+0000 7f64e6583740 1 Processor -- start 2024-03-18T01:39:53.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.991+0000 7f64e6583740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119d77e0 0x559b119eea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.991+0000 7f64e6583740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119eef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.991+0000 7f64e6583740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119dd340 0x559b119effa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.991+0000 7f64e6583740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1b1d50 con 0x559b119dd340 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.991+0000 7f64e6583740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc1b1680 con 0x559b119d77e0 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64e6583740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc071c80 con 0x559b119d81d0 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64e6583740 10 client.0 _ll_register_callbacks cb 0x7f64c8005c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119d77e0 0x559b119eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119d77e0 0x559b119eea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55222/0 (socket says 172.21.15.7:55222) 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64dbe00640 1 -- 172.21.15.7:0/34897584 learned_addr learned my addr 172.21.15.7:0/34897584 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64e0e00640 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64dbe00640 1 -- 172.21.15.7:0/34897584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 msgr2=0x559b119eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64dbe00640 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64dbe00640 1 -- 172.21.15.7:0/34897584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119dd340 msgr2=0x559b119effa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64dbe00640 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119dd340 0x559b119effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64dbe00640 1 -- 172.21.15.7:0/34897584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b119f0590 con 0x559b119d77e0 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.992+0000 7f64e0e00640 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.993+0000 7f64dbe00640 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119d77e0 0x559b119eea50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f64c8025560 tx=0x7f64c8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=729d072ffc1361af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.993+0000 7f64c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.993+0000 7f64c2a00640 1 -- 172.21.15.7:0/34897584 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c802b040 con 0x559b119d77e0 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.993+0000 7f64e6583740 1 -- 172.21.15.7:0/34897584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64c8017070 con 0x559b119d77e0 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.993+0000 7f64c2a00640 1 -- 172.21.15.7:0/34897584 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f64c80258f0 con 0x559b119d77e0 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.993+0000 7f64c2a00640 1 -- 172.21.15.7:0/34897584 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f64c8025af0 con 0x559b119d77e0 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.993+0000 7f64e6583740 10 client.26018 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.993+0000 7f64e6583740 1 -- 172.21.15.7:0/34897584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559b119f1840 con 0x559b119d77e0 2024-03-18T01:39:53.996 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64c2a00640 1 -- 172.21.15.7:0/34897584 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64c8017070 con 0x559b119d77e0 2024-03-18T01:39:53.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64c2a00640 1 -- 172.21.15.7:0/34897584 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f64c80035f0 con 0x559b119d77e0 2024-03-18T01:39:53.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64e6583740 10 client.26018 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:53.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64e6583740 10 client.26018 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:53.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64e6583740 20 client.26018 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:53.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64e6583740 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559b11a5c990 0x559b11a5edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:53.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64e6583740 4 client.26018 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T01:39:53.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64e6583740 1 -- 172.21.15.7:0/34897584 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559b11942810 con 0x559b11a5c990 2024-03-18T01:39:53.998 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.994+0000 7f64e0e00640 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559b11a5c990 0x559b11a5edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.995+0000 7f64c2a00640 10 client.26018 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.995+0000 7f64e0e00640 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559b11a5c990 0x559b11a5edb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd7298d2675c99d server_cookie=8deed01c3b88d939 in_seq=0 out_seq=0 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.996+0000 7f64c2a00640 1 -- 172.21.15.7:0/34897584 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559b11942810 con 0x559b11a5c990 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.996+0000 7f64c2a00640 10 client.26018 handle_command_reply: tid=0 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.996+0000 7f64e6583740 1 client.26018 shutdown 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.996+0000 7f64e6583740 1 -- 172.21.15.7:0/34897584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119d77e0 msgr2=0x559b119eea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119d77e0 0x559b119eea50 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f64c8025560 tx=0x7f64c8000bc0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 -- 172.21.15.7:0/34897584 shutdown_connections 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559b11a5c990 0x559b11a5edb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559b119dd340 0x559b119effa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b119d81d0 0x559b119eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64c2a00640 0 client.26018 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:53.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 --2- 172.21.15.7:0/34897584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559b119d77e0 0x559b119eea50 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 -- 172.21.15.7:0/34897584 >> 172.21.15.7:0/34897584 conn(0x559b1199f930 msgr2=0x559b119e9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:54.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 -- 172.21.15.7:0/34897584 shutdown_connections 2024-03-18T01:39:54.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 1 -- 172.21.15.7:0/34897584 wait complete. 2024-03-18T01:39:54.000 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.997+0000 7f64e6583740 20 client.26018 trim_cache size 0 max 16384 2024-03-18T01:39:54.001 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:39:54.001 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-03-18T01:39:54.001 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "5aa9bc1d-b838-4892-9c16-b54eabe8a7d1", 2024-03-18T01:39:54.001 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-03-18T01:39:54.001 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:39:54.001 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.998+0000 7f64e3800640 1 -- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f64c40a2880 msgr2=0x7f64c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.998+0000 7f64e3800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f64c40a2880 0x7f64c40a4d40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009270 tx=0x7f64d4015040 comp rx=0 tx=0).stop 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.998+0000 7f64e3800640 1 -- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc107560 msgr2=0x7f64dc19f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.998+0000 7f64e3800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc107560 0x7f64dc19f100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f64d4009f60 tx=0x7f64d4009f90 comp rx=0 tx=0).stop 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.999+0000 7f64e3800640 1 -- 172.21.15.7:0/2180739584 shutdown_connections 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.999+0000 7f64e3800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f64c40a2880 0x7f64c40a4d40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.999+0000 7f64e3800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f64dc107560 0x7f64dc19f100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.999+0000 7f64e3800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64dc104c30 0x7f64dc198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.999+0000 7f64e3800640 1 --2- 172.21.15.7:0/2180739584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64dc102300 0x7f64dc197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.999+0000 7f64e3800640 1 -- 172.21.15.7:0/2180739584 >> 172.21.15.7:0/2180739584 conn(0x7f64dc0fbc70 msgr2=0x7f64dc102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:54.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:53.999+0000 7f64e3800640 1 -- 172.21.15.7:0/2180739584 shutdown_connections 2024-03-18T01:39:54.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.001+0000 7f64e3800640 1 -- 172.21.15.7:0/2180739584 wait complete. 2024-03-18T01:39:54.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:39:54.031 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:39:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.290+0000 7f9cdf200640 1 -- 172.21.15.7:0/3073217417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 msgr2=0x7f9ce0103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.290+0000 7f9cdf200640 1 --2- 172.21.15.7:0/3073217417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 0x7f9ce0103c60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0009900 tx=0x7f9cd002bc50 comp rx=0 tx=0).stop 2024-03-18T01:39:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.291+0000 7f9cdf200640 1 -- 172.21.15.7:0/3073217417 shutdown_connections 2024-03-18T01:39:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.291+0000 7f9cdf200640 1 --2- 172.21.15.7:0/3073217417 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce01053f0 0x7f9ce010bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.291+0000 7f9cdf200640 1 --2- 172.21.15.7:0/3073217417 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce0104a30 0x7f9ce0104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.291+0000 7f9cdf200640 1 --2- 172.21.15.7:0/3073217417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 0x7f9ce0103c60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.291+0000 7f9cdf200640 1 -- 172.21.15.7:0/3073217417 >> 172.21.15.7:0/3073217417 conn(0x7f9ce00fedb0 msgr2=0x7f9ce01011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.291+0000 7f9cdf200640 1 -- 172.21.15.7:0/3073217417 shutdown_connections 2024-03-18T01:39:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.291+0000 7f9cdf200640 1 -- 172.21.15.7:0/3073217417 wait complete. 2024-03-18T01:39:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.292+0000 7f9cdf200640 1 Processor -- start 2024-03-18T01:39:54.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.292+0000 7f9cdf200640 1 -- start start 2024-03-18T01:39:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdf200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 0x7f9ce0199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdf200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce0104a30 0x7f9ce019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdf200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce01053f0 0x7f9ce01a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdf200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce010f580 con 0x7f9ce01053f0 2024-03-18T01:39:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdf200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce010f400 con 0x7f9ce0103840 2024-03-18T01:39:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdf200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce010f700 con 0x7f9ce0104a30 2024-03-18T01:39:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 0x7f9ce0199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 0x7f9ce0199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55248/0 (socket says 172.21.15.7:55248) 2024-03-18T01:39:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdde00640 1 -- 172.21.15.7:0/603135425 learned_addr learned my addr 172.21.15.7:0/603135425 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cdd400640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce0104a30 0x7f9ce019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.293+0000 7f9cde800640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce01053f0 0x7f9ce01a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.294+0000 7f9cdde00640 1 -- 172.21.15.7:0/603135425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce0104a30 msgr2=0x7f9ce019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.294+0000 7f9cdde00640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce0104a30 0x7f9ce019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.294+0000 7f9cdde00640 1 -- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce01053f0 msgr2=0x7f9ce01a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.294+0000 7f9cdde00640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce01053f0 0x7f9ce01a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.294+0000 7f9cdde00640 1 -- 172.21.15.7:0/603135425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ce01a17f0 con 0x7f9ce0103840 2024-03-18T01:39:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.294+0000 7f9cdde00640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 0x7f9ce0199bd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9cd002c260 tx=0x7f9cd002c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.295+0000 7f9cd6a00640 1 -- 172.21.15.7:0/603135425 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0041070 con 0x7f9ce0103840 2024-03-18T01:39:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.295+0000 7f9cd6a00640 1 -- 172.21.15.7:0/603135425 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cd00054e0 con 0x7f9ce0103840 2024-03-18T01:39:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.295+0000 7f9cd6a00640 1 -- 172.21.15.7:0/603135425 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0034440 con 0x7f9ce0103840 2024-03-18T01:39:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.295+0000 7f9cdf200640 1 -- 172.21.15.7:0/603135425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ce01a1a80 con 0x7f9ce0103840 2024-03-18T01:39:54.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.295+0000 7f9cdf200640 1 -- 172.21.15.7:0/603135425 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ce01a1f80 con 0x7f9ce0103840 2024-03-18T01:39:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.298+0000 7f9ce7e29740 1 Processor -- start 2024-03-18T01:39:54.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.298+0000 7f9cd6a00640 1 -- 172.21.15.7:0/603135425 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9cd0044070 con 0x7f9ce0103840 2024-03-18T01:39:54.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.299+0000 7f9cd6a00640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc40a2900 0x7f9cc40a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.300+0000 7f9cd6a00640 1 -- 172.21.15.7:0/603135425 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9cd00f0990 con 0x7f9ce0103840 2024-03-18T01:39:54.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.300+0000 7f9cde800640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc40a2900 0x7f9cc40a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.300+0000 7f9ce7e29740 1 -- start start 2024-03-18T01:39:54.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.300+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97991a10 0x557c97991e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.300+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97992400 0x557c97996f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.300+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c97999a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.301+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557c978f61f0 con 0x557c97991a10 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.301+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c9789ea80 con 0x557c97997570 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.301+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557c978f2290 con 0x557c97992400 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.301+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97991a10 0x557c97991e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.301+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97991a10 0x557c97991e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:49416/0 (socket says 172.21.15.7:49416) 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.301+0000 7f9cdde00640 1 -- 172.21.15.7:0/2171884693 learned_addr learned my addr 172.21.15.7:0/2171884693 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.301+0000 7f9cde800640 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c97999a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.302+0000 7f9cdde00640 1 -- 172.21.15.7:0/2171884693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97992400 msgr2=0x557c97996f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.302+0000 7f9cdde00640 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97992400 0x557c97996f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.302+0000 7f9cdde00640 1 -- 172.21.15.7:0/2171884693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 msgr2=0x557c97999a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.302+0000 7f9cdde00640 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c97999a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.302+0000 7f9cdde00640 1 -- 172.21.15.7:0/2171884693 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c976d47d0 con 0x557c97991a10 2024-03-18T01:39:54.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.302+0000 7f9cde800640 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c97999a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.302+0000 7f9cde800640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc40a2900 0x7f9cc40a4dc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009f30 tx=0x7f9cd8009450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:54.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.302+0000 7f9cdde00640 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97991a10 0x557c97991e30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00b9940 tx=0x7f9cd0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19de92ffdf8fda30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:54.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.303+0000 7f9ccb400640 1 -- 172.21.15.7:0/2171884693 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd004b040 con 0x557c97991a10 2024-03-18T01:39:54.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.303+0000 7f9ccb400640 1 -- 172.21.15.7:0/2171884693 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cd00bd670 con 0x557c97991a10 2024-03-18T01:39:54.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.303+0000 7f9ccb400640 1 -- 172.21.15.7:0/2171884693 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00bb300 con 0x557c97991a10 2024-03-18T01:39:54.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.303+0000 7f9ce7e29740 1 -- 172.21.15.7:0/2171884693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97991a10 msgr2=0x557c97991e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.303+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97991a10 0x557c97991e30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00b9940 tx=0x7f9cd0009260 comp rx=0 tx=0).stop 2024-03-18T01:39:54.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.304+0000 7f9ce7e29740 1 -- 172.21.15.7:0/2171884693 shutdown_connections 2024-03-18T01:39:54.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.304+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c97999a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.304+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97992400 0x557c97996f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.304+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/2171884693 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97991a10 0x557c97991e30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.304+0000 7f9ce7e29740 1 -- 172.21.15.7:0/2171884693 >> 172.21.15.7:0/2171884693 conn(0x557c9798cf80 msgr2=0x557c9798d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:54.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.304+0000 7f9ce7e29740 1 -- 172.21.15.7:0/2171884693 shutdown_connections 2024-03-18T01:39:54.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.304+0000 7f9ce7e29740 1 -- 172.21.15.7:0/2171884693 wait complete. 2024-03-18T01:39:54.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.304+0000 7f9ce7e29740 1 -- start start 2024-03-18T01:39:54.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.305+0000 7f9ce7e29740 1 Processor -- start 2024-03-18T01:39:54.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.305+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c979b4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557c979ab2f0 con 0x557c97992400 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c978b1ed0 con 0x557c97997570 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557c979ab470 con 0x557c97991a10 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9ce7e29740 10 client.0 _ll_register_callbacks cb 0x557c978ed220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45340/0 (socket says 172.21.15.7:45340) 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9cdde00640 1 -- 172.21.15.7:0/4060131368 learned_addr learned my addr 172.21.15.7:0/4060131368 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9cdd400640 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.306+0000 7f9cde800640 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c979b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cdde00640 1 -- 172.21.15.7:0/4060131368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 msgr2=0x557c979b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cdde00640 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c979b4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cdde00640 1 -- 172.21.15.7:0/4060131368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 msgr2=0x557c979ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cdde00640 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cdde00640 1 -- 172.21.15.7:0/4060131368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c9760a3f0 con 0x557c97991a10 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cdde00640 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979accb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00b93d0 tx=0x7f9cd00bd0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa0333f3e886b31e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cbfe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cbfe00640 1 -- 172.21.15.7:0/4060131368 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00d9030 con 0x557c97991a10 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9cbfe00640 1 -- 172.21.15.7:0/4060131368 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9cd00c5280 con 0x557c97991a10 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.307+0000 7f9ce7e29740 1 -- 172.21.15.7:0/4060131368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cd00bd670 con 0x557c97991a10 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9cbfe00640 1 -- 172.21.15.7:0/4060131368 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9cd004b050 con 0x557c97991a10 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9ce7e29740 10 client.26014 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9ce7e29740 1 -- 172.21.15.7:0/4060131368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c975ca0f0 con 0x557c97991a10 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9cbfe00640 1 -- 172.21.15.7:0/4060131368 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cd00bd670 con 0x557c97991a10 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9cbfe00640 1 -- 172.21.15.7:0/4060131368 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9cd00c0030 con 0x557c97991a10 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9ce7e29740 10 client.26014 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9ce7e29740 10 client.26014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9ce7e29740 20 client.26014 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:54.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c979c0ea0 0x557c979c3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9ce7e29740 4 client.26014 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:39:54.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.308+0000 7f9ce7e29740 1 -- 172.21.15.7:0/4060131368 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c97909be0 con 0x557c979c0ea0 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.309+0000 7f9cdd400640 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c979c0ea0 0x557c979c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.309+0000 7f9cbfe00640 10 client.26014 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.309+0000 7f9cdd400640 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c979c0ea0 0x557c979c3290 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d8805a2e434cc88 server_cookie=9b1b3f73c75b5fe in_seq=0 out_seq=0 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.310+0000 7f9cbfe00640 1 -- 172.21.15.7:0/4060131368 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c97909be0 con 0x557c979c0ea0 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.310+0000 7f9cbfe00640 10 client.26014 handle_command_reply: tid=0 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.310+0000 7f9ce7e29740 1 client.26014 shutdown 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.310+0000 7f9ce7e29740 1 -- 172.21.15.7:0/4060131368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 msgr2=0x557c979accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.310+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979accb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00b93d0 tx=0x7f9cd00bd0b0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 1 -- 172.21.15.7:0/4060131368 shutdown_connections 2024-03-18T01:39:54.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c979c0ea0 0x557c979c3290 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97997570 0x557c979b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/4060131368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979accb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 1 -- 172.21.15.7:0/4060131368 >> 172.21.15.7:0/4060131368 conn(0x557c9798cf80 msgr2=0x557c9798c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9cbfe00640 0 client.26014 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 1 -- 172.21.15.7:0/4060131368 shutdown_connections 2024-03-18T01:39:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 1 -- 172.21.15.7:0/4060131368 wait complete. 2024-03-18T01:39:54.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.311+0000 7f9ce7e29740 20 client.26014 trim_cache size 0 max 16384 2024-03-18T01:39:54.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9ce7e29740 1 Processor -- start 2024-03-18T01:39:54.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9ce7e29740 1 -- start start 2024-03-18T01:39:54.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97991a10 0x557c979acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97997570 0x557c9798b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce0071020 con 0x557c97992400 2024-03-18T01:39:54.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce0072bc0 con 0x557c97991a10 2024-03-18T01:39:54.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce00716f0 con 0x557c97997570 2024-03-18T01:39:54.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.336+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97991a10 0x557c979acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97991a10 0x557c979acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:55286/0 (socket says 172.21.15.7:55286) 2024-03-18T01:39:54.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cde800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97997570 0x557c9798b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdde00640 1 -- 172.21.15.7:0/1918404072 learned_addr learned my addr 172.21.15.7:0/1918404072 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdd400640 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdde00640 1 -- 172.21.15.7:0/1918404072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97997570 msgr2=0x557c9798b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdde00640 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97997570 0x557c9798b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdde00640 1 -- 172.21.15.7:0/1918404072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 msgr2=0x557c979b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdde00640 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdde00640 1 -- 172.21.15.7:0/1918404072 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c97992040 con 0x557c97991a10 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdd400640 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.337+0000 7f9cdde00640 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97991a10 0x557c979acfd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0005ab0 tx=0x7f9cd0049f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4ea5b34186caf62 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.338+0000 7f9ccaa00640 1 -- 172.21.15.7:0/1918404072 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00d9070 con 0x557c97991a10 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.338+0000 7f9ccaa00640 1 -- 172.21.15.7:0/1918404072 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cd00bd670 con 0x557c97991a10 2024-03-18T01:39:54.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.338+0000 7f9ccaa00640 1 -- 172.21.15.7:0/1918404072 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00c5280 con 0x557c97991a10 2024-03-18T01:39:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.339+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1918404072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97991a10 msgr2=0x557c979acfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.339+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97991a10 0x557c979acfd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0005ab0 tx=0x7f9cd0049f90 comp rx=0 tx=0).stop 2024-03-18T01:39:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.339+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1918404072 shutdown_connections 2024-03-18T01:39:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.339+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97997570 0x557c9798b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.339+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97992400 0x557c979b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.339+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1918404072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97991a10 0x557c979acfd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.339+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1918404072 >> 172.21.15.7:0/1918404072 conn(0x557c9798cf80 msgr2=0x557c979ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:54.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.340+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1918404072 shutdown_connections 2024-03-18T01:39:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.340+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1918404072 wait complete. 2024-03-18T01:39:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.340+0000 7f9ce7e29740 1 -- start start 2024-03-18T01:39:54.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.341+0000 7f9ce7e29740 1 Processor -- start 2024-03-18T01:39:54.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.341+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.341+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97992400 0x557c979b1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.341+0000 7f9ce7e29740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97997570 0x557c979b2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.341+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce0072490 con 0x557c97997570 2024-03-18T01:39:54.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.341+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce0071dc0 con 0x557c97992400 2024-03-18T01:39:54.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.341+0000 7f9ce7e29740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ce006f000 con 0x557c97991a10 2024-03-18T01:39:54.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.341+0000 7f9ce7e29740 10 client.0 _ll_register_callbacks cb 0x7f9cd00b92a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:39:54.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.342+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.342+0000 7f9cdde00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979b13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:45358/0 (socket says 172.21.15.7:45358) 2024-03-18T01:39:54.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.342+0000 7f9cdde00640 1 -- 172.21.15.7:0/1410913420 learned_addr learned my addr 172.21.15.7:0/1410913420 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:39:54.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.342+0000 7f9cdd400640 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97992400 0x557c979b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.342+0000 7f9cdde00640 1 -- 172.21.15.7:0/1410913420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97992400 msgr2=0x557c979b1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:54.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.342+0000 7f9cdde00640 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97992400 0x557c979b1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.342+0000 7f9cdde00640 1 -- 172.21.15.7:0/1410913420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97997570 msgr2=0x557c979b2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:39:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.343+0000 7f9cde800640 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97997570 0x557c979b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.343+0000 7f9cdde00640 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97997570 0x557c979b2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.343+0000 7f9cdde00640 1 -- 172.21.15.7:0/1410913420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c979b2f30 con 0x557c97991a10 2024-03-18T01:39:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.343+0000 7f9cde800640 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97997570 0x557c979b2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:39:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.343+0000 7f9cdde00640 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979b13f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0036040 tx=0x7f9cd00bb270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36c806ffb167c17e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:39:54.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.343+0000 7f9cbfe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:39:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.344+0000 7f9cbfe00640 1 -- 172.21.15.7:0/1410913420 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00d9070 con 0x557c97991a10 2024-03-18T01:39:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.344+0000 7f9cbfe00640 1 -- 172.21.15.7:0/1410913420 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9cd00b7b80 con 0x557c97991a10 2024-03-18T01:39:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.344+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1410913420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cd00bd670 con 0x557c97991a10 2024-03-18T01:39:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.344+0000 7f9cbfe00640 1 -- 172.21.15.7:0/1410913420 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9cd00bc6d0 con 0x557c97991a10 2024-03-18T01:39:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.344+0000 7f9ce7e29740 10 client.26020 fetch_fsmap learned FSMap version 30 2024-03-18T01:39:54.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.344+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1410913420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557c979b2f30 con 0x557c97991a10 2024-03-18T01:39:54.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.346+0000 7f9cbfe00640 1 -- 172.21.15.7:0/1410913420 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cd00bd670 con 0x557c97991a10 2024-03-18T01:39:54.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.346+0000 7f9cbfe00640 1 -- 172.21.15.7:0/1410913420 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9cd00c0070 con 0x557c97991a10 2024-03-18T01:39:54.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.347+0000 7f9ce7e29740 10 client.26020 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:39:54.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.347+0000 7f9ce7e29740 10 client.26020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:39:54.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.347+0000 7f9ce7e29740 20 client.26020 populate_metadata read hostname 'smithi007' 2024-03-18T01:39:54.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.347+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c97a272b0 0x557c97a296a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:39:54.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.347+0000 7f9ce7e29740 4 client.26020 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:39:54.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.347+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1410913420 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557c97903730 con 0x557c97a272b0 2024-03-18T01:39:54.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.349+0000 7f9cdd400640 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c97a272b0 0x557c97a296a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:39:54.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9cdd400640 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c97a272b0 0x557c97a296a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4938fe3b54d10eb server_cookie=59cb86e89adf9d0f in_seq=0 out_seq=0 2024-03-18T01:39:54.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9cbfe00640 10 client.26020 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:54.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9cbfe00640 1 -- 172.21.15.7:0/1410913420 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x557c97903730 con 0x557c97a272b0 2024-03-18T01:39:54.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9cbfe00640 10 client.26020 handle_command_reply: tid=0 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 client.26020 shutdown 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1410913420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 msgr2=0x557c979b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979b13f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0036040 tx=0x7f9cd00bb270 comp rx=0 tx=0).stop 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1410913420 shutdown_connections 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c97a272b0 0x557c97a296a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c97997570 0x557c979b2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c97992400 0x557c979b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 --2- 172.21.15.7:0/1410913420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c97991a10 0x557c979b13f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1410913420 >> 172.21.15.7:0/1410913420 conn(0x557c9798cf80 msgr2=0x557c9798b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:54.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.351+0000 7f9cbfe00640 0 client.26020 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:39:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.352+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1410913420 shutdown_connections 2024-03-18T01:39:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.352+0000 7f9ce7e29740 1 -- 172.21.15.7:0/1410913420 wait complete. 2024-03-18T01:39:54.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.352+0000 7f9ce7e29740 20 client.26020 trim_cache size 0 max 16384 2024-03-18T01:39:54.354 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:39:54.354 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T01:39:54.355 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:39:54.355 INFO:teuthology.orchestra.run.smithi007.stdout: "5aa9bc1d-b838-4892-9c16-b54eabe8a7d1": { 2024-03-18T01:39:54.355 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:39:54.355 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "5aa9bc1d-b838-4892-9c16-b54eabe8a7d1", 2024-03-18T01:39:54.355 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:39:54.355 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:39:54.355 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:39:54.355 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.353+0000 7f9cdf200640 1 -- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc40a2900 msgr2=0x7f9cc40a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.353+0000 7f9cdf200640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc40a2900 0x7f9cc40a4dc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009f30 tx=0x7f9cd8009450 comp rx=0 tx=0).stop 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.353+0000 7f9cdf200640 1 -- 172.21.15.7:0/603135425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 msgr2=0x7f9ce0199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.353+0000 7f9cdf200640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 0x7f9ce0199bd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9cd002c260 tx=0x7f9cd002c6e0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.354+0000 7f9cdf200640 1 -- 172.21.15.7:0/603135425 shutdown_connections 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.354+0000 7f9cdf200640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9cc40a2900 0x7f9cc40a4dc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.354+0000 7f9cdf200640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ce01053f0 0x7f9ce01a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.354+0000 7f9cdf200640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ce0104a30 0x7f9ce019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.354+0000 7f9cdf200640 1 --2- 172.21.15.7:0/603135425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ce0103840 0x7f9ce0199bd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:39:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.354+0000 7f9cdf200640 1 -- 172.21.15.7:0/603135425 >> 172.21.15.7:0/603135425 conn(0x7f9ce00fedb0 msgr2=0x7f9ce0107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:39:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.354+0000 7f9cdf200640 1 -- 172.21.15.7:0/603135425 shutdown_connections 2024-03-18T01:39:54.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:39:54.355+0000 7f9cdf200640 1 -- 172.21.15.7:0/603135425 wait complete. 2024-03-18T01:39:54.376 INFO:tasks.cephfs.filesystem:scrub status for tag:5aa9bc1d-b838-4892-9c16-b54eabe8a7d1 - {'path': '/', 'tag': '5aa9bc1d-b838-4892-9c16-b54eabe8a7d1', 'options': 'recursive,force'} 2024-03-18T01:39:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: pgmap v469: 129 pgs: 129 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 228 MiB/s wr, 88 op/s 2024-03-18T01:39:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:39:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:39:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:39:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: scrub queued for path: / 2024-03-18T01:39:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:54.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:54.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:54.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:54 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:54.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: pgmap v469: 129 pgs: 129 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 228 MiB/s wr, 88 op/s 2024-03-18T01:39:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:39:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:39:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:39:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: scrub queued for path: / 2024-03-18T01:39:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:54 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: pgmap v469: 129 pgs: 129 active+clean; 4.0 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 228 MiB/s wr, 88 op/s 2024-03-18T01:39:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:39:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:39:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:39:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: scrub queued for path: / 2024-03-18T01:39:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:54 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.772 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:39:55.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.773 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:55 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:55 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.895 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.895 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.895 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:39:55.895 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:55.895 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:55 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:39:56.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:56 smithi046 ceph-mon[36108]: pgmap v470: 129 pgs: 129 active+clean; 4.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 243 MiB/s wr, 100 op/s 2024-03-18T01:39:56.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:56 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:56.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:56 smithi007 ceph-mon[31996]: pgmap v470: 129 pgs: 129 active+clean; 4.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 243 MiB/s wr, 100 op/s 2024-03-18T01:39:56.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:56 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:56.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:56 smithi203 ceph-mon[36345]: pgmap v470: 129 pgs: 129 active+clean; 4.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 243 MiB/s wr, 100 op/s 2024-03-18T01:39:56.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:56 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:39:58.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:39:58 smithi046 ceph-mon[36108]: pgmap v471: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 231 MiB/s wr, 90 op/s 2024-03-18T01:39:58.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:39:58 smithi007 ceph-mon[31996]: pgmap v471: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 231 MiB/s wr, 90 op/s 2024-03-18T01:39:58.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:39:58 smithi203 ceph-mon[36345]: pgmap v471: 129 pgs: 129 active+clean; 4.9 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 231 MiB/s wr, 90 op/s 2024-03-18T01:40:00.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:00 smithi046 ceph-mon[36108]: pgmap v472: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 232 MiB/s wr, 91 op/s 2024-03-18T01:40:00.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:00 smithi046 ceph-mon[36108]: overall HEALTH_OK 2024-03-18T01:40:00.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:00 smithi007 ceph-mon[31996]: pgmap v472: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 232 MiB/s wr, 91 op/s 2024-03-18T01:40:00.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:00 smithi007 ceph-mon[31996]: overall HEALTH_OK 2024-03-18T01:40:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:00 smithi203 ceph-mon[36345]: pgmap v472: 129 pgs: 129 active+clean; 5.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 232 MiB/s wr, 91 op/s 2024-03-18T01:40:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:00 smithi203 ceph-mon[36345]: overall HEALTH_OK 2024-03-18T01:40:02.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:02 smithi046 ceph-mon[36108]: pgmap v473: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 255 MiB/s wr, 98 op/s 2024-03-18T01:40:02.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:02 smithi007 ceph-mon[31996]: pgmap v473: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 255 MiB/s wr, 98 op/s 2024-03-18T01:40:02.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:02 smithi203 ceph-mon[36345]: pgmap v473: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 255 MiB/s wr, 98 op/s 2024-03-18T01:40:04.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:04 smithi046 ceph-mon[36108]: pgmap v474: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 222 MiB/s wr, 88 op/s 2024-03-18T01:40:04.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:04 smithi007 ceph-mon[31996]: pgmap v474: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 222 MiB/s wr, 88 op/s 2024-03-18T01:40:04.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:04 smithi203 ceph-mon[36345]: pgmap v474: 129 pgs: 129 active+clean; 6.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 222 MiB/s wr, 88 op/s 2024-03-18T01:40:05.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:05 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:40:05.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:05 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:40:05.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:05 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:40:06.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:06 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:40:06.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:06 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:40:06.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:06 smithi046 ceph-mon[36108]: pgmap v475: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 259 MiB/s wr, 97 op/s 2024-03-18T01:40:06.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:06 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:40:06.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:06 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:40:06.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:06 smithi007 ceph-mon[31996]: pgmap v475: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 259 MiB/s wr, 97 op/s 2024-03-18T01:40:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:06 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:40:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:06 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:40:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:06 smithi203 ceph-mon[36345]: pgmap v475: 129 pgs: 129 active+clean; 7.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 259 MiB/s wr, 97 op/s 2024-03-18T01:40:08.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:08 smithi046 ceph-mon[36108]: pgmap v476: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 256 MiB/s wr, 98 op/s 2024-03-18T01:40:08.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:08 smithi007 ceph-mon[31996]: pgmap v476: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 256 MiB/s wr, 98 op/s 2024-03-18T01:40:08.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:08 smithi203 ceph-mon[36345]: pgmap v476: 129 pgs: 129 active+clean; 7.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 256 MiB/s wr, 98 op/s 2024-03-18T01:40:10.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:10 smithi046 ceph-mon[36108]: pgmap v477: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 255 MiB/s wr, 97 op/s 2024-03-18T01:40:10.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:10 smithi007 ceph-mon[31996]: pgmap v477: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 255 MiB/s wr, 97 op/s 2024-03-18T01:40:10.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:10 smithi203 ceph-mon[36345]: pgmap v477: 129 pgs: 129 active+clean; 7.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 255 MiB/s wr, 97 op/s 2024-03-18T01:40:12.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:12 smithi046 ceph-mon[36108]: pgmap v478: 129 pgs: 129 active+clean; 8.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 291 MiB/s wr, 114 op/s 2024-03-18T01:40:12.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:12 smithi007 ceph-mon[31996]: pgmap v478: 129 pgs: 129 active+clean; 8.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 291 MiB/s wr, 114 op/s 2024-03-18T01:40:12.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:12 smithi203 ceph-mon[36345]: pgmap v478: 129 pgs: 129 active+clean; 8.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 291 MiB/s wr, 114 op/s 2024-03-18T01:40:13.626 INFO:tasks.workunit.client.0.smithi007.stdout:fs setup took 46 secs 2024-03-18T01:40:14.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:14 smithi046 ceph-mon[36108]: pgmap v479: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 252 MiB/s wr, 95 op/s 2024-03-18T01:40:14.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:14 smithi007 ceph-mon[31996]: pgmap v479: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 252 MiB/s wr, 95 op/s 2024-03-18T01:40:14.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:14 smithi203 ceph-mon[36345]: pgmap v479: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 252 MiB/s wr, 95 op/s 2024-03-18T01:40:16.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:16 smithi007 ceph-mon[31996]: pgmap v480: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 270 MiB/s wr, 99 op/s 2024-03-18T01:40:16.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:16 smithi203 ceph-mon[36345]: pgmap v480: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 270 MiB/s wr, 99 op/s 2024-03-18T01:40:17.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:16 smithi046 ceph-mon[36108]: pgmap v480: 129 pgs: 129 active+clean; 9.5 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 270 MiB/s wr, 99 op/s 2024-03-18T01:40:18.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:18 smithi007 ceph-mon[31996]: pgmap v481: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 248 MiB/s wr, 104 op/s 2024-03-18T01:40:18.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:18 smithi203 ceph-mon[36345]: pgmap v481: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 248 MiB/s wr, 104 op/s 2024-03-18T01:40:19.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:18 smithi046 ceph-mon[36108]: pgmap v481: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 248 MiB/s wr, 104 op/s 2024-03-18T01:40:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:20 smithi007 ceph-mon[31996]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 250 MiB/s wr, 97 op/s 2024-03-18T01:40:20.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:20 smithi203 ceph-mon[36345]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 250 MiB/s wr, 97 op/s 2024-03-18T01:40:21.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:20 smithi046 ceph-mon[36108]: pgmap v482: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 250 MiB/s wr, 97 op/s 2024-03-18T01:40:22.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:22 smithi203 ceph-mon[36345]: pgmap v483: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 281 MiB/s wr, 108 op/s 2024-03-18T01:40:23.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:22 smithi046 ceph-mon[36108]: pgmap v483: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 281 MiB/s wr, 108 op/s 2024-03-18T01:40:23.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:22 smithi007 ceph-mon[31996]: pgmap v483: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 281 MiB/s wr, 108 op/s 2024-03-18T01:40:24.012 INFO:tasks.workunit.client.0.smithi007.stdout:Syncing()...10 sec 2024-03-18T01:40:24.014 INFO:tasks.workunit.client.0.smithi007.stdout:Starting Actual Benchmark At: Mon Mar 18 01:40:24 2024 2024-03-18T01:40:24.014 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:40:24.377 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:40:24.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:40:24.821 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:24 smithi046 ceph-mon[36108]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 243 MiB/s wr, 88 op/s 2024-03-18T01:40:24.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:24 smithi007 ceph-mon[31996]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 243 MiB/s wr, 88 op/s 2024-03-18T01:40:24.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:24 smithi203 ceph-mon[36345]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 243 MiB/s wr, 88 op/s 2024-03-18T01:40:26.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.072+0000 7f8429e00640 1 -- 172.21.15.7:0/643257882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84241087e0 msgr2=0x7f842410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:26.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.072+0000 7f8429e00640 1 --2- 172.21.15.7:0/643257882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84241087e0 0x7f842410ac10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f841401a910 tx=0x7f841403cc80 comp rx=0 tx=0).stop 2024-03-18T01:40:26.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.075+0000 7f8429e00640 1 -- 172.21.15.7:0/643257882 shutdown_connections 2024-03-18T01:40:26.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.075+0000 7f8429e00640 1 --2- 172.21.15.7:0/643257882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f842410db00 0x7f842410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.075+0000 7f8429e00640 1 --2- 172.21.15.7:0/643257882 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f842410b150 0x7f842410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.075+0000 7f8429e00640 1 --2- 172.21.15.7:0/643257882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84241087e0 0x7f842410ac10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.075+0000 7f8429e00640 1 -- 172.21.15.7:0/643257882 >> 172.21.15.7:0/643257882 conn(0x7f842406e780 msgr2=0x7f8424070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:26.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.084+0000 7f8429e00640 1 -- 172.21.15.7:0/643257882 shutdown_connections 2024-03-18T01:40:26.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.085+0000 7f8429e00640 1 -- 172.21.15.7:0/643257882 wait complete. 2024-03-18T01:40:26.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.086+0000 7f8429e00640 1 Processor -- start 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.090+0000 7f8429e00640 1 -- start start 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.090+0000 7f8429e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84241087e0 0x7f842419e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.090+0000 7f8429e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f842410db00 0x7f842419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.090+0000 7f8429e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84241a37c0 0x7f84241a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.090+0000 7f8429e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8424111af0 con 0x7f84241a37c0 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.090+0000 7f8429e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8424111970 con 0x7f84241087e0 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.090+0000 7f8429e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8424111c70 con 0x7f842410db00 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.091+0000 7f8422000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f842410db00 0x7f842419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.091+0000 7f8422000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f842410db00 0x7f842419e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:57922/0 (socket says 172.21.15.7:57922) 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.091+0000 7f8422000640 1 -- 172.21.15.7:0/1442881432 learned_addr learned my addr 172.21.15.7:0/1442881432 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.092+0000 7f8423400640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84241a37c0 0x7f84241a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.092+0000 7f8422000640 1 -- 172.21.15.7:0/1442881432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84241087e0 msgr2=0x7f842419e220 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:40:26.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.092+0000 7f8422000640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84241087e0 0x7f842419e220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.092+0000 7f8422000640 1 -- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84241a37c0 msgr2=0x7f84241a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:26.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.092+0000 7f8422000640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84241a37c0 0x7f84241a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.092+0000 7f8422000640 1 -- 172.21.15.7:0/1442881432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84241a6220 con 0x7f842410db00 2024-03-18T01:40:26.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.095+0000 7f8422000640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f842410db00 0x7f842419e760 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f841000d6d0 tx=0x7f841000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:26.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.096+0000 7f8428a00640 1 -- 172.21.15.7:0/1442881432 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410017070 con 0x7f842410db00 2024-03-18T01:40:26.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.096+0000 7f8428a00640 1 -- 172.21.15.7:0/1442881432 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8410010460 con 0x7f842410db00 2024-03-18T01:40:26.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.096+0000 7f8428a00640 1 -- 172.21.15.7:0/1442881432 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841000f620 con 0x7f842410db00 2024-03-18T01:40:26.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.097+0000 7f8429e00640 1 -- 172.21.15.7:0/1442881432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84241a6510 con 0x7f842410db00 2024-03-18T01:40:26.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.098+0000 7f8429e00640 1 -- 172.21.15.7:0/1442881432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84241a69a0 con 0x7f842410db00 2024-03-18T01:40:26.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.114+0000 7f842cb69740 1 Processor -- start 2024-03-18T01:40:26.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.114+0000 7f842cb69740 1 -- start start 2024-03-18T01:40:26.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.114+0000 7f842cb69740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x559573202c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.116+0000 7f842cb69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573203210 0x559573207d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.116+0000 7f842cb69740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x55957320a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.118 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.116+0000 7f842cb69740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5595731671f0 con 0x559573208380 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.116+0000 7f842cb69740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55957310fa80 con 0x559573202820 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.116+0000 7f842cb69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559573163290 con 0x559573203210 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.117+0000 7f8423400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x559573202c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.117+0000 7f8423400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x559573202c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49192/0 (socket says 172.21.15.7:49192) 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.117+0000 7f8423400640 1 -- 172.21.15.7:0/4013254559 learned_addr learned my addr 172.21.15.7:0/4013254559 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.117+0000 7f8422a00640 1 --2- 172.21.15.7:0/4013254559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573203210 0x559573207d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.118+0000 7f8423400640 1 -- 172.21.15.7:0/4013254559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573203210 msgr2=0x559573207d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.118+0000 7f8423400640 1 --2- 172.21.15.7:0/4013254559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573203210 0x559573207d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.118+0000 7f8423400640 1 -- 172.21.15.7:0/4013254559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 msgr2=0x55957320a810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.118+0000 7f8423400640 1 --2- 172.21.15.7:0/4013254559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x55957320a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.118+0000 7f8423400640 1 -- 172.21.15.7:0/4013254559 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559572f457d0 con 0x559573202820 2024-03-18T01:40:26.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.119+0000 7f8423400640 1 --2- 172.21.15.7:0/4013254559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x559573202c40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f841c008dc0 tx=0x7f841c00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1ab1108c52dff92 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:26.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.119+0000 7f8428a00640 1 -- 172.21.15.7:0/1442881432 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f841001a070 con 0x7f842410db00 2024-03-18T01:40:26.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.120+0000 7f840fe00640 1 -- 172.21.15.7:0/4013254559 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c014070 con 0x559573202820 2024-03-18T01:40:26.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.120+0000 7f840fe00640 1 -- 172.21.15.7:0/4013254559 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f841c003b70 con 0x559573202820 2024-03-18T01:40:26.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.120+0000 7f840fe00640 1 -- 172.21.15.7:0/4013254559 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c010930 con 0x559573202820 2024-03-18T01:40:26.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.122+0000 7f842cb69740 1 -- 172.21.15.7:0/4013254559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 msgr2=0x559573202c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:26.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.122+0000 7f842cb69740 1 --2- 172.21.15.7:0/4013254559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x559573202c40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f841c008dc0 tx=0x7f841c00e720 comp rx=0 tx=0).stop 2024-03-18T01:40:26.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.126+0000 7f842cb69740 1 -- 172.21.15.7:0/4013254559 shutdown_connections 2024-03-18T01:40:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.127+0000 7f842cb69740 1 --2- 172.21.15.7:0/4013254559 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x55957320a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.127+0000 7f842cb69740 1 --2- 172.21.15.7:0/4013254559 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573203210 0x559573207d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.127+0000 7f842cb69740 1 --2- 172.21.15.7:0/4013254559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x559573202c40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.127 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.127+0000 7f842cb69740 1 -- 172.21.15.7:0/4013254559 >> 172.21.15.7:0/4013254559 conn(0x5595731fdd90 msgr2=0x5595731fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:26.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.127+0000 7f842cb69740 1 -- 172.21.15.7:0/4013254559 shutdown_connections 2024-03-18T01:40:26.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.128+0000 7f842cb69740 1 -- 172.21.15.7:0/4013254559 wait complete. 2024-03-18T01:40:26.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.128+0000 7f842cb69740 1 -- start start 2024-03-18T01:40:26.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.129+0000 7f842cb69740 1 Processor -- start 2024-03-18T01:40:26.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.129+0000 7f8428a00640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84080a2910 0x7f84080a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.131+0000 7f8428a00640 1 -- 172.21.15.7:0/1442881432 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f84100ce710 con 0x7f842410db00 2024-03-18T01:40:26.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.133+0000 7f842cb69740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x55957320ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.133+0000 7f842cb69740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 0x55957320d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.134+0000 7f8422a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x55957320ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.134+0000 7f8422a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x55957320ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49196/0 (socket says 172.21.15.7:49196) 2024-03-18T01:40:26.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.135+0000 7f8422a00640 1 -- 172.21.15.7:0/1176451168 learned_addr learned my addr 172.21.15.7:0/1176451168 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:26.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.135+0000 7f842cb69740 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573208380 0x559573214380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.136+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55957320b330 con 0x559573203210 2024-03-18T01:40:26.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.136+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559573122ed0 con 0x559573202820 2024-03-18T01:40:26.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.136+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55957320b4b0 con 0x559573208380 2024-03-18T01:40:26.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.136+0000 7f842cb69740 10 client.0 _ll_register_callbacks cb 0x55957315e220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:26.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.137+0000 7f8422a00640 1 -- 172.21.15.7:0/1176451168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573208380 msgr2=0x559573214380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:26.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.137+0000 7f8422a00640 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573208380 0x559573214380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.137+0000 7f8422a00640 1 -- 172.21.15.7:0/1176451168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 msgr2=0x55957320d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:26.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.137+0000 7f8422a00640 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 0x55957320d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.137+0000 7f8422a00640 1 -- 172.21.15.7:0/1176451168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559572e7b3f0 con 0x559573202820 2024-03-18T01:40:26.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.140+0000 7f8422a00640 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x55957320ce00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f841401a860 tx=0x7f841403da40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc61681b8c7ca3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.141+0000 7f8423400640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84080a2910 0x7f84080a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.141+0000 7f840cc00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.141+0000 7f840cc00640 1 -- 172.21.15.7:0/1176451168 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414053070 con 0x559573202820 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.141+0000 7f840cc00640 1 -- 172.21.15.7:0/1176451168 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f841403ddc0 con 0x559573202820 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.141+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f841c003b70 con 0x559573202820 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.141+0000 7f840cc00640 1 -- 172.21.15.7:0/1176451168 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f841405f050 con 0x559573202820 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.141+0000 7f842cb69740 10 client.26042 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.141+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559572e3b0f0 con 0x559573202820 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.142+0000 7f840cc00640 1 -- 172.21.15.7:0/1176451168 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f841c003b70 con 0x559573202820 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.143+0000 7f840cc00640 1 -- 172.21.15.7:0/1176451168 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f84140035f0 con 0x559573202820 2024-03-18T01:40:26.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.143+0000 7f842cb69740 10 client.26042 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.143+0000 7f842cb69740 10 client.26042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.143+0000 7f842cb69740 20 client.26042 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.143+0000 7f842cb69740 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559573219070 0x55957321b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.143+0000 7f842cb69740 4 client.26042 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.143+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55957317abe0 con 0x559573219070 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.144+0000 7f8422000640 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559573219070 0x55957321b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.144+0000 7f8422000640 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559573219070 0x55957321b530 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3d04f8332ea5e36 server_cookie=f201c098da13c19a in_seq=0 out_seq=0 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.145+0000 7f840cc00640 10 client.26042 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.147+0000 7f840cc00640 1 -- 172.21.15.7:0/1176451168 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55957317abe0 con 0x559573219070 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.147+0000 7f840cc00640 10 client.26042 handle_command_reply: tid=0 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.147+0000 7f842cb69740 1 client.26042 shutdown 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.147+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 msgr2=0x55957320ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.147+0000 7f842cb69740 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x55957320ce00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f841401a860 tx=0x7f841403da40 comp rx=0 tx=0).stop 2024-03-18T01:40:26.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.149+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 shutdown_connections 2024-03-18T01:40:26.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.149+0000 7f842cb69740 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559573219070 0x55957321b530 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.150+0000 7f842cb69740 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573208380 0x559573214380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.150+0000 7f840cc00640 0 client.26042 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:26.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.151+0000 7f842cb69740 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 0x55957320d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.152+0000 7f842cb69740 1 --2- 172.21.15.7:0/1176451168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573202820 0x55957320ce00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.152+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 >> 172.21.15.7:0/1176451168 conn(0x5595731fdd90 msgr2=0x5595731fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:26.162 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.157+0000 7f8423400640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84080a2910 0x7f84080a4dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f841c00ec20 tx=0x7f841c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:26.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.170+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 shutdown_connections 2024-03-18T01:40:26.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.170+0000 7f842cb69740 1 -- 172.21.15.7:0/1176451168 wait complete. 2024-03-18T01:40:26.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.170+0000 7f842cb69740 20 client.26042 trim_cache size 0 max 16384 2024-03-18T01:40:26.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.438+0000 7f842cb69740 1 Processor -- start 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.451+0000 7f842cb69740 1 -- start start 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.453+0000 7f842cb69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x5595731fe170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.453+0000 7f842cb69740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573203210 0x559573214340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.453+0000 7f842cb69740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x559573203c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.453+0000 7f842cb69740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f84241b0fc0 con 0x559573208380 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.453+0000 7f842cb69740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84241b2bc0 con 0x559573203210 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.453+0000 7f842cb69740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f84241b1690 con 0x559573202820 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.454+0000 7f8423400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x559573203c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.454+0000 7f8423400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x559573203c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34576/0 (socket says 172.21.15.7:34576) 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.454+0000 7f8423400640 1 -- 172.21.15.7:0/3385609328 learned_addr learned my addr 172.21.15.7:0/3385609328 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.455+0000 7f8423400640 1 -- 172.21.15.7:0/3385609328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 msgr2=0x5595731fe170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.455+0000 7f8423400640 1 --2- 172.21.15.7:0/3385609328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x5595731fe170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.455+0000 7f8423400640 1 -- 172.21.15.7:0/3385609328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573203210 msgr2=0x559573214340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.455+0000 7f8423400640 1 --2- 172.21.15.7:0/3385609328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573203210 0x559573214340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.455+0000 7f8423400640 1 -- 172.21.15.7:0/3385609328 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559573202460 con 0x559573208380 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.455+0000 7f8423400640 1 --2- 172.21.15.7:0/3385609328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x559573203c30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f841c017000 tx=0x7f841c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cb2e87ed8b676c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.456+0000 7f840f400640 1 -- 172.21.15.7:0/3385609328 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c014070 con 0x559573208380 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.456+0000 7f840f400640 1 -- 172.21.15.7:0/3385609328 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f841c010b50 con 0x559573208380 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.456+0000 7f840f400640 1 -- 172.21.15.7:0/3385609328 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841c0052b0 con 0x559573208380 2024-03-18T01:40:26.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.458+0000 7f842cb69740 1 -- 172.21.15.7:0/3385609328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 msgr2=0x559573203c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:26.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.458+0000 7f842cb69740 1 --2- 172.21.15.7:0/3385609328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x559573203c30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f841c017000 tx=0x7f841c01f000 comp rx=0 tx=0).stop 2024-03-18T01:40:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.477+0000 7f842cb69740 1 -- 172.21.15.7:0/3385609328 shutdown_connections 2024-03-18T01:40:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.477+0000 7f842cb69740 1 --2- 172.21.15.7:0/3385609328 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573208380 0x559573203c30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.477 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.477+0000 7f842cb69740 1 --2- 172.21.15.7:0/3385609328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573203210 0x559573214340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.477+0000 7f842cb69740 1 --2- 172.21.15.7:0/3385609328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x5595731fe170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.477+0000 7f842cb69740 1 -- 172.21.15.7:0/3385609328 >> 172.21.15.7:0/3385609328 conn(0x5595731fdd90 msgr2=0x559573201880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.477+0000 7f842cb69740 1 -- 172.21.15.7:0/3385609328 shutdown_connections 2024-03-18T01:40:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.477+0000 7f842cb69740 1 -- 172.21.15.7:0/3385609328 wait complete. 2024-03-18T01:40:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.478+0000 7f842cb69740 1 -- start start 2024-03-18T01:40:26.478 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.478+0000 7f842cb69740 1 Processor -- start 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.479+0000 7f842cb69740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x559573211330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.479+0000 7f842cb69740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 0x559573211870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.479+0000 7f842cb69740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573208380 0x55957327d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.479+0000 7f8422000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 0x559573211870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.479+0000 7f8422a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x559573211330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f8422a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x559573211330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:57946/0 (socket says 172.21.15.7:57946) 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f8422a00640 1 -- 172.21.15.7:0/1810616411 learned_addr learned my addr 172.21.15.7:0/1810616411 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f84241b2430 con 0x559573203210 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84241b1d60 con 0x559573208380 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f8423400640 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573208380 0x55957327d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.480 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f84241aef60 con 0x559573202820 2024-03-18T01:40:26.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f8422a00640 1 -- 172.21.15.7:0/1810616411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573208380 msgr2=0x55957327d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:26.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f8422a00640 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573208380 0x55957327d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f8422a00640 1 -- 172.21.15.7:0/1810616411 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 msgr2=0x559573211870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:26.488 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f8422a00640 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 0x559573211870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.480+0000 7f8422a00640 1 -- 172.21.15.7:0/1810616411 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55957327d9e0 con 0x559573202820 2024-03-18T01:40:26.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.481+0000 7f842cb69740 10 client.0 _ll_register_callbacks cb 0x7f841401a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:26.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.481+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f841c010b50 con 0x559573202820 2024-03-18T01:40:26.489 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.481+0000 7f8422a00640 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x559573211330 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8414003e30 tx=0x7f8414003e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1073915064c4854c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:26.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.488+0000 7f840cc00640 10 client.26029 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:26.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.488+0000 7f840cc00640 1 -- 172.21.15.7:0/1810616411 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8414068060 con 0x559573202820 2024-03-18T01:40:26.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.489+0000 7f840cc00640 1 -- 172.21.15.7:0/1810616411 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8414003910 con 0x559573202820 2024-03-18T01:40:26.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.489+0000 7f840cc00640 1 -- 172.21.15.7:0/1810616411 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f841405f050 con 0x559573202820 2024-03-18T01:40:26.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.489+0000 7f842cb69740 10 client.26029 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:26.490 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.489+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55957327eab0 con 0x559573202820 2024-03-18T01:40:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.493+0000 7f840cc00640 1 -- 172.21.15.7:0/1810616411 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f84140035f0 con 0x559573202820 2024-03-18T01:40:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.493+0000 7f842cb69740 10 client.26029 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:26.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.493+0000 7f842cb69740 10 client.26029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:40:26.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.493+0000 7f842cb69740 20 client.26029 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:26.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.493+0000 7f842cb69740 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559573219070 0x559573281540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:26.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.494+0000 7f842cb69740 4 client.26029 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:40:26.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.494+0000 7f8422000640 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559573219070 0x559573281540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:26.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.494+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559573174730 con 0x559573219070 2024-03-18T01:40:26.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.495+0000 7f8422000640 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559573219070 0x559573281540 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f30022794fd8665 server_cookie=2910da1f7e7dd907 in_seq=0 out_seq=0 2024-03-18T01:40:26.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.499+0000 7f840cc00640 10 client.26029 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:26.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.507+0000 7f840cc00640 1 -- 172.21.15.7:0/1810616411 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559573174730 con 0x559573219070 2024-03-18T01:40:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.507+0000 7f840cc00640 10 client.26029 handle_command_reply: tid=0 2024-03-18T01:40:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.508+0000 7f842cb69740 1 client.26029 shutdown 2024-03-18T01:40:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.508+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 msgr2=0x559573211330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:26.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.508+0000 7f842cb69740 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x559573211330 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8414003e30 tx=0x7f8414003e60 comp rx=0 tx=0).stop 2024-03-18T01:40:26.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.508+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 shutdown_connections 2024-03-18T01:40:26.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.508+0000 7f842cb69740 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559573219070 0x559573281540 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.510+0000 7f840cc00640 0 client.26029 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:26.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.510+0000 7f842cb69740 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559573208380 0x55957327d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.510+0000 7f842cb69740 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559573203210 0x559573211870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.510+0000 7f842cb69740 1 --2- 172.21.15.7:0/1810616411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559573202820 0x559573211330 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.510+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 >> 172.21.15.7:0/1810616411 conn(0x5595731fdd90 msgr2=0x559573200ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.510+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 shutdown_connections 2024-03-18T01:40:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.510+0000 7f842cb69740 1 -- 172.21.15.7:0/1810616411 wait complete. 2024-03-18T01:40:26.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.511+0000 7f842cb69740 20 client.26029 trim_cache size 0 max 16384 2024-03-18T01:40:26.512 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:40:26.512 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-03-18T01:40:26.512 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-03-18T01:40:26.512 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:40:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.541+0000 7f8429e00640 1 -- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84080a2910 msgr2=0x7f84080a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.541+0000 7f8429e00640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84080a2910 0x7f84080a4dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f841c00ec20 tx=0x7f841c000f40 comp rx=0 tx=0).stop 2024-03-18T01:40:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.541+0000 7f8429e00640 1 -- 172.21.15.7:0/1442881432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f842410db00 msgr2=0x7f842419e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.541+0000 7f8429e00640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f842410db00 0x7f842419e760 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f841000d6d0 tx=0x7f841000db90 comp rx=0 tx=0).stop 2024-03-18T01:40:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.545+0000 7f8429e00640 1 -- 172.21.15.7:0/1442881432 shutdown_connections 2024-03-18T01:40:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.545+0000 7f8429e00640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f84080a2910 0x7f84080a4dd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.545+0000 7f8429e00640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f84241a37c0 0x7f84241a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.545+0000 7f8429e00640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f842410db00 0x7f842419e760 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.545+0000 7f8429e00640 1 --2- 172.21.15.7:0/1442881432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84241087e0 0x7f842419e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:26.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.545+0000 7f8429e00640 1 -- 172.21.15.7:0/1442881432 >> 172.21.15.7:0/1442881432 conn(0x7f842406e780 msgr2=0x7f842410b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:26.570 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.564+0000 7f8429e00640 1 -- 172.21.15.7:0/1442881432 shutdown_connections 2024-03-18T01:40:26.572 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:26.570+0000 7f8429e00640 1 -- 172.21.15.7:0/1442881432 wait complete. 2024-03-18T01:40:26.621 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T01:40:26.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:40:26.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:26 smithi203 ceph-mon[36345]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 273 MiB/s wr, 97 op/s 2024-03-18T01:40:26.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:26 smithi007 ceph-mon[31996]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 273 MiB/s wr, 97 op/s 2024-03-18T01:40:27.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:26 smithi046 ceph-mon[36108]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 273 MiB/s wr, 97 op/s 2024-03-18T01:40:27.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.282+0000 7f084be00640 1 -- 172.21.15.7:0/2006186119 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f084403bce0 con 0x7f084c072d70 2024-03-18T01:40:27.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.282+0000 7f0854200640 1 -- 172.21.15.7:0/2006186119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c072d70 msgr2=0x7f084c0731d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:27.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.282+0000 7f0854200640 1 --2- 172.21.15.7:0/2006186119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c072d70 0x7f084c0731d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0844007290 tx=0x7f0844034520 comp rx=0 tx=0).stop 2024-03-18T01:40:27.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.282+0000 7f0854200640 1 -- 172.21.15.7:0/2006186119 shutdown_connections 2024-03-18T01:40:27.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.282+0000 7f0854200640 1 --2- 172.21.15.7:0/2006186119 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c10fb20 0x7f084c111f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:27.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.282+0000 7f0854200640 1 --2- 172.21.15.7:0/2006186119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c072d70 0x7f084c0731d0 secure :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0844007290 tx=0x7f0844034520 comp rx=0 tx=0).stop 2024-03-18T01:40:27.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.282+0000 7f0854200640 1 --2- 172.21.15.7:0/2006186119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c1093f0 0x7f084c072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:27.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.282+0000 7f0854200640 1 -- 172.21.15.7:0/2006186119 >> 172.21.15.7:0/2006186119 conn(0x7f084c06e6a0 msgr2=0x7f084c070ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:27.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.284+0000 7f0854200640 1 -- 172.21.15.7:0/2006186119 shutdown_connections 2024-03-18T01:40:27.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.286+0000 7f0854200640 1 -- 172.21.15.7:0/2006186119 wait complete. 2024-03-18T01:40:27.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.290+0000 7f0854200640 1 Processor -- start 2024-03-18T01:40:27.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.292+0000 7f0854200640 1 -- start start 2024-03-18T01:40:27.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.292+0000 7f0854200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1093f0 0x7f084c19e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:27.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.292+0000 7f0854200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c10fb20 0x7f084c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:27.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.292+0000 7f0854200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c1a3740 0x7f084c1a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:27.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.292+0000 7f0854200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f084c113910 con 0x7f084c1093f0 2024-03-18T01:40:27.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.292+0000 7f0854200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f084c113790 con 0x7f084c10fb20 2024-03-18T01:40:27.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.292+0000 7f0854200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f084c113a90 con 0x7f084c1a3740 2024-03-18T01:40:27.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.293+0000 7f0850e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c10fb20 0x7f084c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:27.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.294+0000 7f0852200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c1a3740 0x7f084c1a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:27.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.294+0000 7f0852200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c1a3740 0x7f084c1a5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:57964/0 (socket says 172.21.15.7:57964) 2024-03-18T01:40:27.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.294+0000 7f0852200640 1 -- 172.21.15.7:0/1341964057 learned_addr learned my addr 172.21.15.7:0/1341964057 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:27.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.294+0000 7f0852200640 1 -- 172.21.15.7:0/1341964057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c10fb20 msgr2=0x7f084c19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:27.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.294+0000 7f0852200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c10fb20 0x7f084c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:27.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.294+0000 7f0852200640 1 -- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1093f0 msgr2=0x7f084c19e1a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:40:27.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.294+0000 7f0851800640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1093f0 0x7f084c19e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:27.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.295+0000 7f0852200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1093f0 0x7f084c19e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:27.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.295+0000 7f0852200640 1 -- 172.21.15.7:0/1341964057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084c1a61a0 con 0x7f084c1a3740 2024-03-18T01:40:27.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.296+0000 7f0852200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c1a3740 0x7f084c1a5b30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f083800d720 tx=0x7f083800dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:27.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.297+0000 7f084a000640 1 -- 172.21.15.7:0/1341964057 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838017070 con 0x7f084c1a3740 2024-03-18T01:40:27.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.297+0000 7f084a000640 1 -- 172.21.15.7:0/1341964057 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f083800de30 con 0x7f084c1a3740 2024-03-18T01:40:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.297+0000 7f084a000640 1 -- 172.21.15.7:0/1341964057 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0838004d90 con 0x7f084c1a3740 2024-03-18T01:40:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.297+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084c1a6490 con 0x7f084c1a3740 2024-03-18T01:40:27.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.298+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084c1a6990 con 0x7f084c1a3740 2024-03-18T01:40:27.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.304+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f084c10e930 con 0x7f084c1a3740 2024-03-18T01:40:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.309+0000 7f084a000640 1 -- 172.21.15.7:0/1341964057 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f0838004f30 con 0x7f084c1a3740 2024-03-18T01:40:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.315+0000 7f084a000640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f08340a2980 0x7f08340a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:27.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.317+0000 7f0851800640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f08340a2980 0x7f08340a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:27.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.321+0000 7f084a000640 1 -- 172.21.15.7:0/1341964057 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0838098d70 con 0x7f084c1a3740 2024-03-18T01:40:27.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.321+0000 7f0851800640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f08340a2980 0x7f08340a4e40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f08400095f0 tx=0x7f0840005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:27.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.330+0000 7f084a000640 1 -- 172.21.15.7:0/1341964057 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f083809b3d0 con 0x7f084c1a3740 2024-03-18T01:40:27.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.929+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f084c0623e0 con 0x7f084c1a3740 2024-03-18T01:40:27.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.930+0000 7f084a000640 1 -- 172.21.15.7:0/1341964057 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11925 (secure 0 0 0) 0x7f083809acd0 con 0x7f084c1a3740 2024-03-18T01:40:27.931 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:40:27.931 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:39:55.246276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":16,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":13,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24544,24523,24542,24596,24562]},"id":1}]} 2024-03-18T01:40:27.931 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 31 2024-03-18T01:40:27.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f08340a2980 msgr2=0x7f08340a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:27.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f08340a2980 0x7f08340a4e40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f08400095f0 tx=0x7f0840005f90 comp rx=0 tx=0).stop 2024-03-18T01:40:27.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c1a3740 msgr2=0x7f084c1a5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:27.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c1a3740 0x7f084c1a5b30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f083800d720 tx=0x7f083800dbe0 comp rx=0 tx=0).stop 2024-03-18T01:40:27.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 shutdown_connections 2024-03-18T01:40:27.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c1a3740 0x7f084c1a5b30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:27.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f084c10fb20 0x7f084c19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:27.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f08340a2980 0x7f08340a4e40 secure :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f08400095f0 tx=0x7f0840005f90 comp rx=0 tx=0).stop 2024-03-18T01:40:27.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 --2- 172.21.15.7:0/1341964057 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f084c1093f0 0x7f084c19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:27.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.938+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 >> 172.21.15.7:0/1341964057 conn(0x7f084c06e6a0 msgr2=0x7f084c10bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:27.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.939+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 shutdown_connections 2024-03-18T01:40:27.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:27.939+0000 7f0854200640 1 -- 172.21.15.7:0/1341964057 wait complete. 2024-03-18T01:40:27.972 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T01:40:28.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.287+0000 7f77b9e00640 1 -- 172.21.15.7:0/150452979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77b4103a20 msgr2=0x7f77b4103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.287+0000 7f77b9e00640 1 --2- 172.21.15.7:0/150452979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77b4103a20 0x7f77b4103e40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f779c009900 tx=0x7f779c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:40:28.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.288+0000 7f77b9e00640 1 -- 172.21.15.7:0/150452979 shutdown_connections 2024-03-18T01:40:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.288+0000 7f77b9e00640 1 --2- 172.21.15.7:0/150452979 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b41055d0 0x7f77b410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.288+0000 7f77b9e00640 1 --2- 172.21.15.7:0/150452979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b4104c10 0x7f77b4105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.288+0000 7f77b9e00640 1 --2- 172.21.15.7:0/150452979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77b4103a20 0x7f77b4103e40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.288+0000 7f77b9e00640 1 -- 172.21.15.7:0/150452979 >> 172.21.15.7:0/150452979 conn(0x7f77b40fef90 msgr2=0x7f77b41013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.288+0000 7f77b9e00640 1 -- 172.21.15.7:0/150452979 shutdown_connections 2024-03-18T01:40:28.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.288+0000 7f77b9e00640 1 -- 172.21.15.7:0/150452979 wait complete. 2024-03-18T01:40:28.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.292+0000 7f77b9e00640 1 Processor -- start 2024-03-18T01:40:28.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.293+0000 7f77b9e00640 1 -- start start 2024-03-18T01:40:28.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.295+0000 7f77b9e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b4103a20 0x7f77b41959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.296+0000 7f77b9e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b4104c10 0x7f77b4195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.296+0000 7f77b9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77b41055d0 0x7f77b419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.296+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b4103a20 0x7f77b41959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.296+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b4103a20 0x7f77b41959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34628/0 (socket says 172.21.15.7:34628) 2024-03-18T01:40:28.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.297+0000 7f77b2000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b4104c10 0x7f77b4195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.297+0000 7f77b2000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b4104c10 0x7f77b4195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:57986/0 (socket says 172.21.15.7:57986) 2024-03-18T01:40:28.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.298+0000 7f77b2a00640 1 -- 172.21.15.7:0/3972046709 learned_addr learned my addr 172.21.15.7:0/3972046709 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.298+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f77b410f740 con 0x7f77b4103a20 2024-03-18T01:40:28.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.298+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77b410f5c0 con 0x7f77b41055d0 2024-03-18T01:40:28.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.298+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77b410f8c0 con 0x7f77b4104c10 2024-03-18T01:40:28.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.299+0000 7f77b2000640 1 -- 172.21.15.7:0/3972046709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77b41055d0 msgr2=0x7f77b419cf50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:40:28.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.299+0000 7f77b2000640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77b41055d0 0x7f77b419cf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.299+0000 7f77b2000640 1 -- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b4103a20 msgr2=0x7f77b41959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.299+0000 7f77b2000640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b4103a20 0x7f77b41959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.299+0000 7f77b2000640 1 -- 172.21.15.7:0/3972046709 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b41ae640 con 0x7f77b4104c10 2024-03-18T01:40:28.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.301+0000 7f77b2000640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b4104c10 0x7f77b4195ee0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f77a400d6d0 tx=0x7f77a400db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.302+0000 7f77b8a00640 1 -- 172.21.15.7:0/3972046709 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4017070 con 0x7f77b4104c10 2024-03-18T01:40:28.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.302+0000 7f77b8a00640 1 -- 172.21.15.7:0/3972046709 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a4009820 con 0x7f77b4104c10 2024-03-18T01:40:28.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.302+0000 7f77b8a00640 1 -- 172.21.15.7:0/3972046709 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a4004d00 con 0x7f77b4104c10 2024-03-18T01:40:28.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.302+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b41ae8d0 con 0x7f77b4104c10 2024-03-18T01:40:28.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.302+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b41aedd0 con 0x7f77b4104c10 2024-03-18T01:40:28.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.306+0000 7f77bca4a740 1 Processor -- start 2024-03-18T01:40:28.308 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77bca4a740 1 -- start start 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa758c0 0x5613caa75ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa7adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa7d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77bca4a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5613caa3d3b0 con 0x5613caa7b420 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77bca4a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5613ca9da450 con 0x5613caa758c0 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77bca4a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5613ca982cb0 con 0x5613caa762b0 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa7adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.307+0000 7f77b3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa758c0 0x5613caa75ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.308+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa7adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58000/0 (socket says 172.21.15.7:58000) 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.308+0000 7f77b2a00640 1 -- 172.21.15.7:0/3166968236 learned_addr learned my addr 172.21.15.7:0/3166968236 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.308+0000 7f77b2a00640 1 -- 172.21.15.7:0/3166968236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa758c0 msgr2=0x5613caa75ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.308+0000 7f77b2a00640 1 --2- 172.21.15.7:0/3166968236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa758c0 0x5613caa75ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.308+0000 7f77b2a00640 1 -- 172.21.15.7:0/3166968236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 msgr2=0x5613caa7d8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:28.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.308+0000 7f77b2a00640 1 --2- 172.21.15.7:0/3166968236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa7d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.308+0000 7f77b2a00640 1 -- 172.21.15.7:0/3166968236 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613ca9a41f0 con 0x5613caa762b0 2024-03-18T01:40:28.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.309+0000 7f77b2a00640 1 --2- 172.21.15.7:0/3166968236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa7adb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f779c02c8f0 tx=0x7f779c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b18a328309ea9b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.309+0000 7f77a3e00640 1 -- 172.21.15.7:0/3166968236 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c042030 con 0x5613caa762b0 2024-03-18T01:40:28.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.309+0000 7f77a3e00640 1 -- 172.21.15.7:0/3166968236 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f779c004040 con 0x5613caa762b0 2024-03-18T01:40:28.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.309+0000 7f77a3e00640 1 -- 172.21.15.7:0/3166968236 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c0046c0 con 0x5613caa762b0 2024-03-18T01:40:28.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.310+0000 7f77bca4a740 1 -- 172.21.15.7:0/3166968236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 msgr2=0x5613caa7adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.310+0000 7f77bca4a740 1 --2- 172.21.15.7:0/3166968236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa7adb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f779c02c8f0 tx=0x7f779c005d50 comp rx=0 tx=0).stop 2024-03-18T01:40:28.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.314+0000 7f77b8a00640 1 -- 172.21.15.7:0/3972046709 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f77a4005220 con 0x7f77b4104c10 2024-03-18T01:40:28.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.314+0000 7f77bca4a740 1 -- 172.21.15.7:0/3166968236 shutdown_connections 2024-03-18T01:40:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.314+0000 7f77bca4a740 1 --2- 172.21.15.7:0/3166968236 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa7d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.314+0000 7f77bca4a740 1 --2- 172.21.15.7:0/3166968236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa7adb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.314+0000 7f77bca4a740 1 --2- 172.21.15.7:0/3166968236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa758c0 0x5613caa75ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.314+0000 7f77bca4a740 1 -- 172.21.15.7:0/3166968236 >> 172.21.15.7:0/3166968236 conn(0x5613caa70e30 msgr2=0x5613caa71230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.315+0000 7f77bca4a740 1 -- 172.21.15.7:0/3166968236 shutdown_connections 2024-03-18T01:40:28.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.315+0000 7f77bca4a740 1 -- 172.21.15.7:0/3166968236 wait complete. 2024-03-18T01:40:28.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.315+0000 7f77bca4a740 1 -- start start 2024-03-18T01:40:28.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.316+0000 7f77bca4a740 1 Processor -- start 2024-03-18T01:40:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.316+0000 7f77b8a00640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77980a2790 0x7f77980a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.316+0000 7f77b8a00640 1 -- 172.21.15.7:0/3972046709 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f77a40ce790 con 0x7f77b4104c10 2024-03-18T01:40:28.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.317+0000 7f77b3400640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77980a2790 0x7f77980a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.318+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa758c0 0x5613caa7fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.318+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa762b0 0x5613caa800e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.318+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa87120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.318+0000 7f77bca4a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5613ca996110 con 0x5613caa7b420 2024-03-18T01:40:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.318+0000 7f77bca4a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5613ca9d64f0 con 0x5613caa762b0 2024-03-18T01:40:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.318+0000 7f77bca4a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5613caa7e370 con 0x5613caa758c0 2024-03-18T01:40:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.318+0000 7f77bca4a740 10 client.0 _ll_register_callbacks cb 0x5613ca983b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.319+0000 7f77b2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa762b0 0x5613caa800e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.319+0000 7f77b2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa762b0 0x5613caa800e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49272/0 (socket says 172.21.15.7:49272) 2024-03-18T01:40:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.319+0000 7f77b2000640 1 -- 172.21.15.7:0/1692566746 learned_addr learned my addr 172.21.15.7:0/1692566746 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.320+0000 7f77b2a00640 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa758c0 0x5613caa7fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.320+0000 7f77b3400640 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa87120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.321+0000 7f77b3400640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77980a2790 0x7f77980a4c50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x5613caa76b10 tx=0x7f77ac00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.321+0000 7f77b2a00640 1 -- 172.21.15.7:0/1692566746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa762b0 msgr2=0x5613caa800e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.321+0000 7f77b2a00640 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa762b0 0x5613caa800e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.321+0000 7f77b2a00640 1 -- 172.21.15.7:0/1692566746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 msgr2=0x5613caa87120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.321+0000 7f77b2a00640 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa87120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.321+0000 7f77b2a00640 1 -- 172.21.15.7:0/1692566746 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613ca6ee3f0 con 0x5613caa758c0 2024-03-18T01:40:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.322+0000 7f77b2a00640 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa758c0 0x5613caa7fba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x5613caa766b0 tx=0x7f779c02bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82812d75d39c12a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.322+0000 7f77a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.323+0000 7f77a0c00640 1 -- 172.21.15.7:0/1692566746 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c04b070 con 0x5613caa758c0 2024-03-18T01:40:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.323+0000 7f77a0c00640 1 -- 172.21.15.7:0/1692566746 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f779c02cc10 con 0x5613caa758c0 2024-03-18T01:40:28.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.323+0000 7f77bca4a740 1 -- 172.21.15.7:0/1692566746 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f779c004040 con 0x5613caa758c0 2024-03-18T01:40:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.324+0000 7f77a0c00640 1 -- 172.21.15.7:0/1692566746 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f779c055050 con 0x5613caa758c0 2024-03-18T01:40:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.324+0000 7f77bca4a740 10 client.26047 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:28.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.325+0000 7f77bca4a740 1 -- 172.21.15.7:0/1692566746 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5613ca7b87d0 con 0x5613caa758c0 2024-03-18T01:40:28.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.325+0000 7f77a0c00640 1 -- 172.21.15.7:0/1692566746 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f779c004ba0 con 0x5613caa758c0 2024-03-18T01:40:28.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.326+0000 7f77a0c00640 1 -- 172.21.15.7:0/1692566746 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f779c0046c0 con 0x5613caa758c0 2024-03-18T01:40:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.327+0000 7f77bca4a740 10 client.26047 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.327+0000 7f77bca4a740 10 client.26047 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:40:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.327+0000 7f77bca4a740 20 client.26047 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.327+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5613caa9caa0 0x5613caa9ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.327+0000 7f77bca4a740 4 client.26047 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:40:28.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.327+0000 7f77bca4a740 1 -- 172.21.15.7:0/1692566746 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f779c004040 con 0x5613caa9caa0 2024-03-18T01:40:28.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.327+0000 7f77b2000640 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5613caa9caa0 0x5613caa9ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.331+0000 7f77a0c00640 10 client.26047 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:40:28.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.331+0000 7f77b2000640 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5613caa9caa0 0x5613caa9ee90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8f719f8b9b7edf81 server_cookie=a08b743344c8619c in_seq=0 out_seq=0 2024-03-18T01:40:28.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.331+0000 7f77a0c00640 1 -- 172.21.15.7:0/1692566746 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f779c004040 con 0x5613caa9caa0 2024-03-18T01:40:28.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.331+0000 7f77a0c00640 10 client.26047 handle_command_reply: tid=0 2024-03-18T01:40:28.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.332+0000 7f77bca4a740 1 client.26047 shutdown 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 -- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa758c0 msgr2=0x5613caa7fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa758c0 0x5613caa7fba0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x5613caa766b0 tx=0x7f779c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 -- 172.21.15.7:0/1692566746 shutdown_connections 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5613caa9caa0 0x5613caa9ee90 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa87120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa762b0 0x5613caa800e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77a0c00640 0 client.26047 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1692566746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa758c0 0x5613caa7fba0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 -- 172.21.15.7:0/1692566746 >> 172.21.15.7:0/1692566746 conn(0x5613caa70e30 msgr2=0x5613caa701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 -- 172.21.15.7:0/1692566746 shutdown_connections 2024-03-18T01:40:28.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 1 -- 172.21.15.7:0/1692566746 wait complete. 2024-03-18T01:40:28.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.333+0000 7f77bca4a740 20 client.26047 trim_cache size 0 max 16384 2024-03-18T01:40:28.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.363+0000 7f77bca4a740 1 Processor -- start 2024-03-18T01:40:28.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.363+0000 7f77bca4a740 1 -- start start 2024-03-18T01:40:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.363+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa71210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.363+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7b420 0x5613caa870e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.363+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7f1c0 0x5613caa7f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.363+0000 7f77bca4a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b0aa0 con 0x5613caa7f1c0 2024-03-18T01:40:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.363+0000 7f77bca4a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b26a0 con 0x5613caa7b420 2024-03-18T01:40:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.363+0000 7f77bca4a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b1170 con 0x5613caa762b0 2024-03-18T01:40:28.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa71210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7b420 0x5613caa870e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7b420 0x5613caa870e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49274/0 (socket says 172.21.15.7:49274) 2024-03-18T01:40:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2000640 1 -- 172.21.15.7:0/1915288578 learned_addr learned my addr 172.21.15.7:0/1915288578 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b3400640 1 --2- 172.21.15.7:0/1915288578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7f1c0 0x5613caa7f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2000640 1 -- 172.21.15.7:0/1915288578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 msgr2=0x5613caa71210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2000640 1 --2- 172.21.15.7:0/1915288578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa71210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2000640 1 -- 172.21.15.7:0/1915288578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7f1c0 msgr2=0x5613caa7f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2000640 1 --2- 172.21.15.7:0/1915288578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7f1c0 0x5613caa7f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.364+0000 7f77b2000640 1 -- 172.21.15.7:0/1915288578 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613caa6cc10 con 0x5613caa7b420 2024-03-18T01:40:28.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.366+0000 7f77b2000640 1 --2- 172.21.15.7:0/1915288578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7b420 0x5613caa870e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f77a40cc2a0 tx=0x7f77a40ce040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9366f71bcc3924d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.367+0000 7f77a3400640 1 -- 172.21.15.7:0/1915288578 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a40b7020 con 0x5613caa7b420 2024-03-18T01:40:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.367+0000 7f77a3400640 1 -- 172.21.15.7:0/1915288578 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a4005510 con 0x5613caa7b420 2024-03-18T01:40:28.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.367+0000 7f77a3400640 1 -- 172.21.15.7:0/1915288578 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a409e040 con 0x5613caa7b420 2024-03-18T01:40:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.368+0000 7f77bca4a740 1 -- 172.21.15.7:0/1915288578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7b420 msgr2=0x5613caa870e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.368+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1915288578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7b420 0x5613caa870e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f77a40cc2a0 tx=0x7f77a40ce040 comp rx=0 tx=0).stop 2024-03-18T01:40:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.368+0000 7f77bca4a740 1 -- 172.21.15.7:0/1915288578 shutdown_connections 2024-03-18T01:40:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.368+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1915288578 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7f1c0 0x5613caa7f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.368+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1915288578 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7b420 0x5613caa870e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.368+0000 7f77bca4a740 1 --2- 172.21.15.7:0/1915288578 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa71210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.368+0000 7f77bca4a740 1 -- 172.21.15.7:0/1915288578 >> 172.21.15.7:0/1915288578 conn(0x5613caa70e30 msgr2=0x5613caa701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.369+0000 7f77bca4a740 1 -- 172.21.15.7:0/1915288578 shutdown_connections 2024-03-18T01:40:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.369+0000 7f77bca4a740 1 -- 172.21.15.7:0/1915288578 wait complete. 2024-03-18T01:40:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.369+0000 7f77bca4a740 1 -- start start 2024-03-18T01:40:28.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.369+0000 7f77bca4a740 1 Processor -- start 2024-03-18T01:40:28.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.370+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa84ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.370+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa84ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77bca4a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7f1c0 0x5613caa86000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77bca4a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b1f10 con 0x5613caa7b420 2024-03-18T01:40:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77bca4a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77b41b1840 con 0x5613caa7f1c0 2024-03-18T01:40:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77bca4a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77b40733b0 con 0x5613caa762b0 2024-03-18T01:40:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77bca4a740 10 client.0 _ll_register_callbacks cb 0x7f77ac009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77b3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7f1c0 0x5613caa86000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77b3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7f1c0 0x5613caa86000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49280/0 (socket says 172.21.15.7:49280) 2024-03-18T01:40:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77b3400640 1 -- 172.21.15.7:0/2835863209 learned_addr learned my addr 172.21.15.7:0/2835863209 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.371+0000 7f77b2000640 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa84ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.372+0000 7f77b3400640 1 -- 172.21.15.7:0/2835863209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 msgr2=0x5613caa84ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.372+0000 7f77b3400640 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa84ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.372+0000 7f77b3400640 1 -- 172.21.15.7:0/2835863209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 msgr2=0x5613caa84ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.372+0000 7f77b3400640 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa84ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.372+0000 7f77b3400640 1 -- 172.21.15.7:0/2835863209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613caa865f0 con 0x5613caa7f1c0 2024-03-18T01:40:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.372+0000 7f77b3400640 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7f1c0 0x5613caa86000 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f77ac016e10 tx=0x7f77ac004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c0798c2fd605cf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.372+0000 7f77a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:40:28.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.373+0000 7f77a0c00640 1 -- 172.21.15.7:0/2835863209 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac027020 con 0x5613caa7f1c0 2024-03-18T01:40:28.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.373+0000 7f77a0c00640 1 -- 172.21.15.7:0/2835863209 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f77ac023680 con 0x5613caa7f1c0 2024-03-18T01:40:28.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.373+0000 7f77bca4a740 1 -- 172.21.15.7:0/2835863209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77a4005510 con 0x5613caa7f1c0 2024-03-18T01:40:28.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.375+0000 7f77a0c00640 1 -- 172.21.15.7:0/2835863209 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f77ac032050 con 0x5613caa7f1c0 2024-03-18T01:40:28.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.375+0000 7f77bca4a740 10 client.26063 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:28.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.375+0000 7f77bca4a740 1 -- 172.21.15.7:0/2835863209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5613cab030f0 con 0x5613caa7f1c0 2024-03-18T01:40:28.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.376+0000 7f77a0c00640 1 -- 172.21.15.7:0/2835863209 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77a4005510 con 0x5613caa7f1c0 2024-03-18T01:40:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.376+0000 7f77a0c00640 1 -- 172.21.15.7:0/2835863209 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f77ac022a30 con 0x5613caa7f1c0 2024-03-18T01:40:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.376+0000 7f77bca4a740 10 client.26063 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.377+0000 7f77bca4a740 10 client.26063 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:40:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.377+0000 7f77bca4a740 20 client.26063 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.377+0000 7f77bca4a740 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5613cab03b00 0x5613cab05f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.377+0000 7f77bca4a740 4 client.26063 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:40:28.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.377+0000 7f77bca4a740 1 -- 172.21.15.7:0/2835863209 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613ca9e7990 con 0x5613cab03b00 2024-03-18T01:40:28.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.378+0000 7f77b2a00640 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5613cab03b00 0x5613cab05f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.378+0000 7f77b2a00640 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5613cab03b00 0x5613cab05f20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5bb4a4dff29ac14 server_cookie=f4ea6ded42081c83 in_seq=0 out_seq=0 2024-03-18T01:40:28.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.379+0000 7f77a0c00640 10 client.26063 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:40:28.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.379+0000 7f77a0c00640 1 -- 172.21.15.7:0/2835863209 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613ca9e7990 con 0x5613cab03b00 2024-03-18T01:40:28.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.379+0000 7f77a0c00640 10 client.26063 handle_command_reply: tid=0 2024-03-18T01:40:28.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.379+0000 7f77bca4a740 1 client.26063 shutdown 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.380+0000 7f77bca4a740 1 -- 172.21.15.7:0/2835863209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7f1c0 msgr2=0x5613caa86000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.380+0000 7f77bca4a740 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7f1c0 0x5613caa86000 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f77ac016e10 tx=0x7f77ac004b50 comp rx=0 tx=0).stop 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77bca4a740 1 -- 172.21.15.7:0/2835863209 shutdown_connections 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77bca4a740 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5613cab03b00 0x5613cab05f20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77bca4a740 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5613caa7f1c0 0x5613caa86000 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77bca4a740 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5613caa7b420 0x5613caa84ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77bca4a740 1 --2- 172.21.15.7:0/2835863209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5613caa762b0 0x5613caa84ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77bca4a740 1 -- 172.21.15.7:0/2835863209 >> 172.21.15.7:0/2835863209 conn(0x5613caa70e30 msgr2=0x5613caa701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77a0c00640 0 client.26063 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:40:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77bca4a740 1 -- 172.21.15.7:0/2835863209 shutdown_connections 2024-03-18T01:40:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.381+0000 7f77bca4a740 1 -- 172.21.15.7:0/2835863209 wait complete. 2024-03-18T01:40:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.382+0000 7f77bca4a740 20 client.26063 trim_cache size 0 max 16384 2024-03-18T01:40:28.383 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:40:28.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.383+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77980a2790 msgr2=0x7f77980a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.383+0000 7f77b9e00640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77980a2790 0x7f77980a4c50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x5613caa76b10 tx=0x7f77ac00d040 comp rx=0 tx=0).stop 2024-03-18T01:40:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.383+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b4104c10 msgr2=0x7f77b4195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.383+0000 7f77b9e00640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b4104c10 0x7f77b4195ee0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f77a400d6d0 tx=0x7f77a400db90 comp rx=0 tx=0).stop 2024-03-18T01:40:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.385+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 shutdown_connections 2024-03-18T01:40:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.385+0000 7f77b9e00640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77b41055d0 0x7f77b419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.385+0000 7f77b9e00640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77b4104c10 0x7f77b4195ee0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.385+0000 7f77b9e00640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f77980a2790 0x7f77980a4c50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.385+0000 7f77b9e00640 1 --2- 172.21.15.7:0/3972046709 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77b4103a20 0x7f77b41959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.385+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 >> 172.21.15.7:0/3972046709 conn(0x7f77b40fef90 msgr2=0x7f77b4077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.387+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 shutdown_connections 2024-03-18T01:40:28.387 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.387+0000 7f77b9e00640 1 -- 172.21.15.7:0/3972046709 wait complete. 2024-03-18T01:40:28.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T01:40:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 -- 172.21.15.7:0/1951656366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e181087e0 msgr2=0x7f4e1810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 --2- 172.21.15.7:0/1951656366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e181087e0 0x7f4e1810ac10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009860 tx=0x7f4e0802bbb0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 -- 172.21.15.7:0/1951656366 shutdown_connections 2024-03-18T01:40:28.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 --2- 172.21.15.7:0/1951656366 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e1810db00 0x7f4e1810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 --2- 172.21.15.7:0/1951656366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e1810b150 0x7f4e1810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 --2- 172.21.15.7:0/1951656366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e181087e0 0x7f4e1810ac10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 -- 172.21.15.7:0/1951656366 >> 172.21.15.7:0/1951656366 conn(0x7f4e1806e770 msgr2=0x7f4e18070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 -- 172.21.15.7:0/1951656366 shutdown_connections 2024-03-18T01:40:28.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.751+0000 7f4e1d800640 1 -- 172.21.15.7:0/1951656366 wait complete. 2024-03-18T01:40:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.752+0000 7f4e1d800640 1 Processor -- start 2024-03-18T01:40:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.753+0000 7f4e1d800640 1 -- start start 2024-03-18T01:40:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.753+0000 7f4e1d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e181087e0 0x7f4e181a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.753+0000 7f4e1d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1810b150 0x7f4e181a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.753+0000 7f4e1d800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e1810db00 0x7f4e181adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.753+0000 7f4e1d800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1811c2d0 con 0x7f4e1810db00 2024-03-18T01:40:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.753+0000 7f4e1d800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1811c150 con 0x7f4e1810b150 2024-03-18T01:40:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.753+0000 7f4e1d800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1811c450 con 0x7f4e181087e0 2024-03-18T01:40:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1810b150 0x7f4e181a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1810b150 0x7f4e181a6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49298/0 (socket says 172.21.15.7:49298) 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16000640 1 -- 172.21.15.7:0/4066027749 learned_addr learned my addr 172.21.15.7:0/4066027749 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e17400640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e1810db00 0x7f4e181adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16a00640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e181087e0 0x7f4e181a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16000640 1 -- 172.21.15.7:0/4066027749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e181087e0 msgr2=0x7f4e181a6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16000640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e181087e0 0x7f4e181a6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16000640 1 -- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e1810db00 msgr2=0x7f4e181adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16000640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e1810db00 0x7f4e181adea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.754+0000 7f4e16000640 1 -- 172.21.15.7:0/4066027749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e181ae510 con 0x7f4e1810b150 2024-03-18T01:40:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.755+0000 7f4e16000640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1810b150 0x7f4e181a6ed0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400e9c0 tx=0x7f4e0400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.756+0000 7f4e0f400640 1 -- 172.21.15.7:0/4066027749 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0400cde0 con 0x7f4e1810b150 2024-03-18T01:40:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.756+0000 7f4e0f400640 1 -- 172.21.15.7:0/4066027749 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e040035f0 con 0x7f4e1810b150 2024-03-18T01:40:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.756+0000 7f4e0f400640 1 -- 172.21.15.7:0/4066027749 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04010690 con 0x7f4e1810b150 2024-03-18T01:40:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.756+0000 7f4e1d800640 1 -- 172.21.15.7:0/4066027749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e181ae800 con 0x7f4e1810b150 2024-03-18T01:40:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.756+0000 7f4e1d800640 1 -- 172.21.15.7:0/4066027749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e181aed00 con 0x7f4e1810b150 2024-03-18T01:40:28.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.761+0000 7f4e20442740 1 Processor -- start 2024-03-18T01:40:28.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.766+0000 7f4e20442740 1 -- start start 2024-03-18T01:40:28.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e20442740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1d1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e20442740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1d6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e20442740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 0x5648fc1d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e20442740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5648fc1993b0 con 0x5648fc1d18c0 2024-03-18T01:40:28.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e20442740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5648fc136450 con 0x5648fc1d22b0 2024-03-18T01:40:28.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e20442740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5648fc0decb0 con 0x5648fc1d7420 2024-03-18T01:40:28.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e16a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1d6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e16a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1d6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49300/0 (socket says 172.21.15.7:49300) 2024-03-18T01:40:28.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.767+0000 7f4e16a00640 1 -- 172.21.15.7:0/3680932481 learned_addr learned my addr 172.21.15.7:0/3680932481 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.768+0000 7f4e17400640 1 --2- 172.21.15.7:0/3680932481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1d1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.768+0000 7f4e16a00640 1 -- 172.21.15.7:0/3680932481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 msgr2=0x5648fc1d98b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:28.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.768+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3680932481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 0x5648fc1d98b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.768+0000 7f4e16a00640 1 -- 172.21.15.7:0/3680932481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 msgr2=0x5648fc1d1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.768+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3680932481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1d1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.769+0000 7f4e16a00640 1 -- 172.21.15.7:0/3680932481 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648fc1001f0 con 0x5648fc1d22b0 2024-03-18T01:40:28.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.770+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3680932481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1d6db0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4e08000c00 tx=0x7f4e08009400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67488c087aa28625 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.771+0000 7f4e03e00640 1 -- 172.21.15.7:0/3680932481 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08039040 con 0x5648fc1d22b0 2024-03-18T01:40:28.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.771+0000 7f4e03e00640 1 -- 172.21.15.7:0/3680932481 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e08034d50 con 0x5648fc1d22b0 2024-03-18T01:40:28.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.771+0000 7f4e03e00640 1 -- 172.21.15.7:0/3680932481 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08005470 con 0x5648fc1d22b0 2024-03-18T01:40:28.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.771+0000 7f4e0f400640 1 -- 172.21.15.7:0/4066027749 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f4e040037a0 con 0x7f4e1810b150 2024-03-18T01:40:28.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.772+0000 7f4e20442740 1 -- 172.21.15.7:0/3680932481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 msgr2=0x5648fc1d6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.772+0000 7f4e20442740 1 --2- 172.21.15.7:0/3680932481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1d6db0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4e08000c00 tx=0x7f4e08009400 comp rx=0 tx=0).stop 2024-03-18T01:40:28.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.775+0000 7f4e0f400640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4dfc0a28d0 0x7f4dfc0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.776+0000 7f4e17400640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4dfc0a28d0 0x7f4dfc0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.776+0000 7f4e20442740 1 -- 172.21.15.7:0/3680932481 shutdown_connections 2024-03-18T01:40:28.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.776+0000 7f4e20442740 1 --2- 172.21.15.7:0/3680932481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 0x5648fc1d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.776+0000 7f4e20442740 1 --2- 172.21.15.7:0/3680932481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1d6db0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.776+0000 7f4e20442740 1 --2- 172.21.15.7:0/3680932481 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1d1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.776+0000 7f4e20442740 1 -- 172.21.15.7:0/3680932481 >> 172.21.15.7:0/3680932481 conn(0x5648fc1cce30 msgr2=0x5648fc1cd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.778+0000 7f4e0f400640 1 -- 172.21.15.7:0/4066027749 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4e040cf2c0 con 0x7f4e1810b150 2024-03-18T01:40:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.778+0000 7f4e17400640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4dfc0a28d0 0x7f4dfc0a4d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x5648fc1d2b10 tx=0x7f4e10009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.778+0000 7f4e20442740 1 -- 172.21.15.7:0/3680932481 shutdown_connections 2024-03-18T01:40:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.778+0000 7f4e20442740 1 -- 172.21.15.7:0/3680932481 wait complete. 2024-03-18T01:40:28.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.779+0000 7f4e20442740 1 -- start start 2024-03-18T01:40:28.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.779+0000 7f4e20442740 1 Processor -- start 2024-03-18T01:40:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.782+0000 7f4e20442740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1eca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.782+0000 7f4e20442740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1ecf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e20442740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 0x5648fc1f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e20442740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5648fc0f2110 con 0x5648fc1d18c0 2024-03-18T01:40:28.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e20442740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5648fc1324f0 con 0x5648fc1d22b0 2024-03-18T01:40:28.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e20442740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5648fc1eb380 con 0x5648fc1d7420 2024-03-18T01:40:28.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e16000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1ecf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e16000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1ecf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49308/0 (socket says 172.21.15.7:49308) 2024-03-18T01:40:28.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e16000640 1 -- 172.21.15.7:0/417648788 learned_addr learned my addr 172.21.15.7:0/417648788 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e16a00640 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1eca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.783+0000 7f4e17400640 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 0x5648fc1f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.784+0000 7f4e20442740 10 client.0 _ll_register_callbacks cb 0x5648fc0dfb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.785+0000 7f4e16000640 1 -- 172.21.15.7:0/417648788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 msgr2=0x5648fc1f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.785+0000 7f4e16000640 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 0x5648fc1f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.785+0000 7f4e16000640 1 -- 172.21.15.7:0/417648788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 msgr2=0x5648fc1eca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.785+0000 7f4e16000640 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1eca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.785+0000 7f4e16000640 1 -- 172.21.15.7:0/417648788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648fbe4a3f0 con 0x5648fc1d22b0 2024-03-18T01:40:28.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.785+0000 7f4e17400640 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 0x5648fc1f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:28.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.786+0000 7f4e16000640 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1ecf60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4e0409b5a0 tx=0x7f4e0409b2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4229fd5d9ccb5ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.786+0000 7f4e00c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:40:28.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.788+0000 7f4e00c00640 1 -- 172.21.15.7:0/417648788 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e040b4060 con 0x5648fc1d22b0 2024-03-18T01:40:28.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.788+0000 7f4e20442740 1 -- 172.21.15.7:0/417648788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e08034d50 con 0x5648fc1d22b0 2024-03-18T01:40:28.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.788+0000 7f4e00c00640 1 -- 172.21.15.7:0/417648788 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4e04097f10 con 0x5648fc1d22b0 2024-03-18T01:40:28.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.790+0000 7f4e00c00640 1 -- 172.21.15.7:0/417648788 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4e040b9050 con 0x5648fc1d22b0 2024-03-18T01:40:28.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.792+0000 7f4e20442740 10 client.26075 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:28.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.792+0000 7f4e20442740 1 -- 172.21.15.7:0/417648788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5648fbf147d0 con 0x5648fc1d22b0 2024-03-18T01:40:28.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.792+0000 7f4e00c00640 1 -- 172.21.15.7:0/417648788 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e08034d50 con 0x5648fc1d22b0 2024-03-18T01:40:28.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.796+0000 7f4e00c00640 1 -- 172.21.15.7:0/417648788 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f4e040860d0 con 0x5648fc1d22b0 2024-03-18T01:40:28.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.796+0000 7f4e20442740 10 client.26075 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:28.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.796+0000 7f4e20442740 10 client.26075 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:40:28.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.796+0000 7f4e20442740 20 client.26075 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:28.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.796+0000 7f4e20442740 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5648fc1f8840 0x5648fc1fac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.797+0000 7f4e20442740 4 client.26075 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:40:28.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.797+0000 7f4e20442740 1 -- 172.21.15.7:0/417648788 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648fc143990 con 0x5648fc1f8840 2024-03-18T01:40:28.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.797+0000 7f4e16a00640 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5648fc1f8840 0x5648fc1fac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.798+0000 7f4e00c00640 10 client.26075 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:40:28.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.798+0000 7f4e16a00640 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5648fc1f8840 0x5648fc1fac30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=84dbe3d7ed29b843 server_cookie=b20edd31937f08a4 in_seq=0 out_seq=0 2024-03-18T01:40:28.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.801+0000 7f4e00c00640 1 -- 172.21.15.7:0/417648788 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648fc143990 con 0x5648fc1f8840 2024-03-18T01:40:28.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.801+0000 7f4e00c00640 10 client.26075 handle_command_reply: tid=0 2024-03-18T01:40:28.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.801+0000 7f4e20442740 1 client.26075 shutdown 2024-03-18T01:40:28.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e20442740 1 -- 172.21.15.7:0/417648788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 msgr2=0x5648fc1ecf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e20442740 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1ecf60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4e0409b5a0 tx=0x7f4e0409b2c0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e20442740 1 -- 172.21.15.7:0/417648788 shutdown_connections 2024-03-18T01:40:28.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e20442740 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5648fc1f8840 0x5648fc1fac30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e20442740 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d7420 0x5648fc1f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e20442740 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d22b0 0x5648fc1ecf60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e20442740 1 --2- 172.21.15.7:0/417648788 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1eca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e00c00640 0 client.26075 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:40:28.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.802+0000 7f4e20442740 1 -- 172.21.15.7:0/417648788 >> 172.21.15.7:0/417648788 conn(0x5648fc1cce30 msgr2=0x5648fc1cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:28 smithi007 ceph-mon[31996]: pgmap v486: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 249 MiB/s wr, 206 op/s 2024-03-18T01:40:28.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:28 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1341964057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:40:28.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.803+0000 7f4e20442740 1 -- 172.21.15.7:0/417648788 shutdown_connections 2024-03-18T01:40:28.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.803+0000 7f4e20442740 1 -- 172.21.15.7:0/417648788 wait complete. 2024-03-18T01:40:28.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.803+0000 7f4e20442740 20 client.26075 trim_cache size 0 max 16384 2024-03-18T01:40:28.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.849+0000 7f4e20442740 1 Processor -- start 2024-03-18T01:40:28.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.850+0000 7f4e20442740 1 -- start start 2024-03-18T01:40:28.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.850+0000 7f4e20442740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d18c0 0x5648fc1cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.850+0000 7f4e20442740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d7420 0x5648fc1f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.850+0000 7f4e20442740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1cb960 0x5648fc1cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.850+0000 7f4e20442740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b09a0 con 0x5648fc1d7420 2024-03-18T01:40:28.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.850+0000 7f4e20442740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b25a0 con 0x5648fc1cb960 2024-03-18T01:40:28.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.850+0000 7f4e20442740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b1070 con 0x5648fc1d18c0 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.851+0000 7f4e16a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d18c0 0x5648fc1cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.851+0000 7f4e16a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d18c0 0x5648fc1cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58060/0 (socket says 172.21.15.7:58060) 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.851+0000 7f4e16a00640 1 -- 172.21.15.7:0/3146175019 learned_addr learned my addr 172.21.15.7:0/3146175019 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.851+0000 7f4e16000640 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d7420 0x5648fc1f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.851+0000 7f4e17400640 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1cb960 0x5648fc1cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.852+0000 7f4e16a00640 1 -- 172.21.15.7:0/3146175019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1cb960 msgr2=0x5648fc1cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.852+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1cb960 0x5648fc1cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.852+0000 7f4e16a00640 1 -- 172.21.15.7:0/3146175019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d7420 msgr2=0x5648fc1f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.852+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d7420 0x5648fc1f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.852+0000 7f4e16a00640 1 -- 172.21.15.7:0/3146175019 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648fc1fafa0 con 0x5648fc1d18c0 2024-03-18T01:40:28.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.853+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d18c0 0x5648fc1cd210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4e08034980 tx=0x7f4e08034880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=420f25b6d8ae949c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.853+0000 7f4e03400640 1 -- 172.21.15.7:0/3146175019 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08039040 con 0x5648fc1d18c0 2024-03-18T01:40:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.853+0000 7f4e03400640 1 -- 172.21.15.7:0/3146175019 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e08034d50 con 0x5648fc1d18c0 2024-03-18T01:40:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.853+0000 7f4e03400640 1 -- 172.21.15.7:0/3146175019 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e08039040 con 0x5648fc1d18c0 2024-03-18T01:40:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.854+0000 7f4e20442740 1 -- 172.21.15.7:0/3146175019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d18c0 msgr2=0x5648fc1cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.854+0000 7f4e20442740 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d18c0 0x5648fc1cd210 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4e08034980 tx=0x7f4e08034880 comp rx=0 tx=0).stop 2024-03-18T01:40:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.855+0000 7f4e20442740 1 -- 172.21.15.7:0/3146175019 shutdown_connections 2024-03-18T01:40:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.855+0000 7f4e20442740 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1cb960 0x5648fc1cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.855+0000 7f4e20442740 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d7420 0x5648fc1f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.855+0000 7f4e20442740 1 --2- 172.21.15.7:0/3146175019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1d18c0 0x5648fc1cd210 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.855+0000 7f4e20442740 1 -- 172.21.15.7:0/3146175019 >> 172.21.15.7:0/3146175019 conn(0x5648fc1cce30 msgr2=0x5648fc1c8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.856+0000 7f4e20442740 1 -- 172.21.15.7:0/3146175019 shutdown_connections 2024-03-18T01:40:28.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.856+0000 7f4e20442740 1 -- 172.21.15.7:0/3146175019 wait complete. 2024-03-18T01:40:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.857+0000 7f4e20442740 1 -- start start 2024-03-18T01:40:28.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.857+0000 7f4e20442740 1 Processor -- start 2024-03-18T01:40:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.859+0000 7f4e20442740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1cb960 0x5648fc1f1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.859+0000 7f4e20442740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.859+0000 7f4e20442740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d7420 0x5648fc1f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.859+0000 7f4e16a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1cb960 0x5648fc1f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.859 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.859+0000 7f4e20442740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b1e10 con 0x5648fc1d18c0 2024-03-18T01:40:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.859+0000 7f4e20442740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181b1740 con 0x5648fc1d7420 2024-03-18T01:40:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.859+0000 7f4e20442740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181180e0 con 0x5648fc1cb960 2024-03-18T01:40:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.859+0000 7f4e17400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d7420 0x5648fc1f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.860 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e17400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d7420 0x5648fc1f2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49338/0 (socket says 172.21.15.7:49338) 2024-03-18T01:40:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e17400640 1 -- 172.21.15.7:0/3065252446 learned_addr learned my addr 172.21.15.7:0/3065252446 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e20442740 10 client.0 _ll_register_callbacks cb 0x7f4e04004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e17400640 1 -- 172.21.15.7:0/3065252446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1cb960 msgr2=0x5648fc1f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e17400640 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1cb960 0x5648fc1f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e17400640 1 -- 172.21.15.7:0/3065252446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 msgr2=0x5648fc1f1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e17400640 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e17400640 1 -- 172.21.15.7:0/3065252446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648fc1f33d0 con 0x5648fc1d7420 2024-03-18T01:40:28.861 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.860+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1cb960 0x5648fc1f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:28.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.861+0000 7f4e00c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:40:28.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.861+0000 7f4e17400640 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d7420 0x5648fc1f2de0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4e100129c0 tx=0x7f4e10004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76441a0209004c22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:28.862 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.862+0000 7f4e00c00640 1 -- 172.21.15.7:0/3065252446 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10020dc0 con 0x5648fc1d7420 2024-03-18T01:40:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.862+0000 7f4e00c00640 1 -- 172.21.15.7:0/3065252446 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4e100283e0 con 0x5648fc1d7420 2024-03-18T01:40:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.862+0000 7f4e20442740 1 -- 172.21.15.7:0/3065252446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e08034d50 con 0x5648fc1d7420 2024-03-18T01:40:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.862+0000 7f4e00c00640 1 -- 172.21.15.7:0/3065252446 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4e100285e0 con 0x5648fc1d7420 2024-03-18T01:40:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.863+0000 7f4e20442740 10 client.26081 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:28.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.863+0000 7f4e20442740 1 -- 172.21.15.7:0/3065252446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5648fc25edf0 con 0x5648fc1d7420 2024-03-18T01:40:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.863+0000 7f4e00c00640 1 -- 172.21.15.7:0/3065252446 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e08034d50 con 0x5648fc1d7420 2024-03-18T01:40:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.864+0000 7f4e00c00640 1 -- 172.21.15.7:0/3065252446 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f4e10027070 con 0x5648fc1d7420 2024-03-18T01:40:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.864+0000 7f4e20442740 10 client.26081 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.864+0000 7f4e20442740 10 client.26081 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:40:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.864+0000 7f4e20442740 20 client.26081 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:28.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.864+0000 7f4e20442740 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5648fc25f800 0x5648fc261c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:28.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.864+0000 7f4e20442740 4 client.26081 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:40:28.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.864+0000 7f4e20442740 1 -- 172.21.15.7:0/3065252446 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648fc13dce0 con 0x5648fc25f800 2024-03-18T01:40:28.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.864+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5648fc25f800 0x5648fc261c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:28.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.867+0000 7f4e16a00640 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5648fc25f800 0x5648fc261c60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bd2a3fb6ccd3df20 server_cookie=63029bf1748f561d in_seq=0 out_seq=0 2024-03-18T01:40:28.868 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.867+0000 7f4e00c00640 10 client.26081 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:40:28.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.871+0000 7f4e00c00640 1 -- 172.21.15.7:0/3065252446 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648fc13dce0 con 0x5648fc25f800 2024-03-18T01:40:28.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e00c00640 10 client.26081 handle_command_reply: tid=0 2024-03-18T01:40:28.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 client.26081 shutdown 2024-03-18T01:40:28.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 -- 172.21.15.7:0/3065252446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d7420 msgr2=0x5648fc1f2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d7420 0x5648fc1f2de0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4e100129c0 tx=0x7f4e10004ab0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 -- 172.21.15.7:0/3065252446 shutdown_connections 2024-03-18T01:40:28.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x5648fc25f800 0x5648fc261c60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5648fc1d7420 0x5648fc1f2de0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5648fc1d18c0 0x5648fc1f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 --2- 172.21.15.7:0/3065252446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5648fc1cb960 0x5648fc1f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e20442740 1 -- 172.21.15.7:0/3065252446 >> 172.21.15.7:0/3065252446 conn(0x5648fc1cce30 msgr2=0x5648fc1cd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.872+0000 7f4e00c00640 0 client.26081 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:40:28.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.873+0000 7f4e20442740 1 -- 172.21.15.7:0/3065252446 shutdown_connections 2024-03-18T01:40:28.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.874+0000 7f4e20442740 1 -- 172.21.15.7:0/3065252446 wait complete. 2024-03-18T01:40:28.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.874+0000 7f4e20442740 20 client.26081 trim_cache size 0 max 16384 2024-03-18T01:40:28.874 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:40:28.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.875+0000 7f4e1d800640 1 -- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4dfc0a28d0 msgr2=0x7f4dfc0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.875+0000 7f4e1d800640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4dfc0a28d0 0x7f4dfc0a4d90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x5648fc1d2b10 tx=0x7f4e10009500 comp rx=0 tx=0).stop 2024-03-18T01:40:28.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.875+0000 7f4e1d800640 1 -- 172.21.15.7:0/4066027749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1810b150 msgr2=0x7f4e181a6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.875+0000 7f4e1d800640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1810b150 0x7f4e181a6ed0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400e9c0 tx=0x7f4e0400ee80 comp rx=0 tx=0).stop 2024-03-18T01:40:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.876+0000 7f4e1d800640 1 -- 172.21.15.7:0/4066027749 shutdown_connections 2024-03-18T01:40:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.876+0000 7f4e1d800640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f4dfc0a28d0 0x7f4dfc0a4d90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.876+0000 7f4e1d800640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4e1810db00 0x7f4e181adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.876+0000 7f4e1d800640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e1810b150 0x7f4e181a6ed0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.876+0000 7f4e1d800640 1 --2- 172.21.15.7:0/4066027749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e181087e0 0x7f4e181a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:28.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.876+0000 7f4e1d800640 1 -- 172.21.15.7:0/4066027749 >> 172.21.15.7:0/4066027749 conn(0x7f4e1806e770 msgr2=0x7f4e1810b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:28.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.876+0000 7f4e1d800640 1 -- 172.21.15.7:0/4066027749 shutdown_connections 2024-03-18T01:40:28.877 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:28.876+0000 7f4e1d800640 1 -- 172.21.15.7:0/4066027749 wait complete. 2024-03-18T01:40:28.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:28 smithi203 ceph-mon[36345]: pgmap v486: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 249 MiB/s wr, 206 op/s 2024-03-18T01:40:28.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:28 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1341964057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:40:28.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T01:40:29.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:28 smithi046 ceph-mon[36108]: pgmap v486: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 249 MiB/s wr, 206 op/s 2024-03-18T01:40:29.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:28 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1341964057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:40:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.251+0000 7f397ca00640 1 -- 172.21.15.7:0/2405263619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 msgr2=0x7f3978110340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.251+0000 7f397ca00640 1 --2- 172.21.15.7:0/2405263619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 0x7f3978110340 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3970005fd0 tx=0x7f397002cc30 comp rx=0 tx=0).stop 2024-03-18T01:40:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.251+0000 7f397ca00640 1 -- 172.21.15.7:0/2405263619 shutdown_connections 2024-03-18T01:40:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.251+0000 7f397ca00640 1 --2- 172.21.15.7:0/2405263619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 0x7f3978110340 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.252+0000 7f397ca00640 1 --2- 172.21.15.7:0/2405263619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f397810b380 0x7f397810d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.252+0000 7f397ca00640 1 --2- 172.21.15.7:0/2405263619 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f39781089c0 0x7f397810adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.252 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.252+0000 7f397ca00640 1 -- 172.21.15.7:0/2405263619 >> 172.21.15.7:0/2405263619 conn(0x7f397806e770 msgr2=0x7f3978070bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.252+0000 7f397ca00640 1 -- 172.21.15.7:0/2405263619 shutdown_connections 2024-03-18T01:40:29.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.254+0000 7f397ca00640 1 -- 172.21.15.7:0/2405263619 wait complete. 2024-03-18T01:40:29.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.255+0000 7f397ca00640 1 Processor -- start 2024-03-18T01:40:29.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.255+0000 7f397ca00640 1 -- start start 2024-03-18T01:40:29.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.257+0000 7f397ca00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39781089c0 0x7f397819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.257+0000 7f397ca00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397810b380 0x7f397819e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.258+0000 7f397ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 0x7f39781a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.258+0000 7f397ca00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3978113a60 con 0x7f397810b380 2024-03-18T01:40:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.258+0000 7f397ca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f39781138e0 con 0x7f397810de80 2024-03-18T01:40:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.258+0000 7f397ca00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3978113be0 con 0x7f39781089c0 2024-03-18T01:40:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.258+0000 7f3976800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 0x7f39781a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.258+0000 7f3976800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 0x7f39781a5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49358/0 (socket says 172.21.15.7:49358) 2024-03-18T01:40:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.258+0000 7f3976800640 1 -- 172.21.15.7:0/3144789694 learned_addr learned my addr 172.21.15.7:0/3144789694 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.258+0000 7f3975e00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39781089c0 0x7f397819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.259+0000 7f3976800640 1 -- 172.21.15.7:0/3144789694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39781089c0 msgr2=0x7f397819dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.259+0000 7f3976800640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39781089c0 0x7f397819dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.259+0000 7f3976800640 1 -- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397810b380 msgr2=0x7f397819e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.259+0000 7f3976800640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397810b380 0x7f397819e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.259+0000 7f3976800640 1 -- 172.21.15.7:0/3144789694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39781b6d30 con 0x7f397810de80 2024-03-18T01:40:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.259+0000 7f3976800640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 0x7f39781a5590 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f397002d170 tx=0x7f397000aa40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.260+0000 7f396ea00640 1 -- 172.21.15.7:0/3144789694 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970044070 con 0x7f397810de80 2024-03-18T01:40:29.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.260+0000 7f396ea00640 1 -- 172.21.15.7:0/3144789694 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3970007d60 con 0x7f397810de80 2024-03-18T01:40:29.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.260+0000 7f396ea00640 1 -- 172.21.15.7:0/3144789694 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970044070 con 0x7f397810de80 2024-03-18T01:40:29.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.260+0000 7f397ca00640 1 -- 172.21.15.7:0/3144789694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39781b6f00 con 0x7f397810de80 2024-03-18T01:40:29.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.260+0000 7f397ca00640 1 -- 172.21.15.7:0/3144789694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39781b7410 con 0x7f397810de80 2024-03-18T01:40:29.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.265+0000 7f397f7bc740 1 Processor -- start 2024-03-18T01:40:29.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.267+0000 7f396ea00640 1 -- 172.21.15.7:0/3144789694 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f3970047070 con 0x7f397810de80 2024-03-18T01:40:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.267+0000 7f397f7bc740 1 -- start start 2024-03-18T01:40:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29e6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e72b0 0x55f7b29ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f397f7bc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f7b29ae3b0 con 0x55f7b29e72b0 2024-03-18T01:40:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f397f7bc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f7b294b450 con 0x55f7b29ec420 2024-03-18T01:40:29.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f397f7bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f7b28f3cb0 con 0x55f7b29e68c0 2024-03-18T01:40:29.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f3975400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29e6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f3975400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29e6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58112/0 (socket says 172.21.15.7:58112) 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f3975e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e72b0 0x55f7b29ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.268+0000 7f3975400640 1 -- 172.21.15.7:0/2023873771 learned_addr learned my addr 172.21.15.7:0/2023873771 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.269+0000 7f3975400640 1 --2- 172.21.15.7:0/2023873771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.269+0000 7f3975e00640 1 -- 172.21.15.7:0/2023873771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 msgr2=0x55f7b29e6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.269+0000 7f3975e00640 1 --2- 172.21.15.7:0/2023873771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29e6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.269+0000 7f3975e00640 1 -- 172.21.15.7:0/2023873771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 msgr2=0x55f7b29ee8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.269+0000 7f3975e00640 1 --2- 172.21.15.7:0/2023873771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29ee8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.269+0000 7f3975e00640 1 -- 172.21.15.7:0/2023873771 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7b29151f0 con 0x55f7b29e72b0 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.270+0000 7f3975e00640 1 --2- 172.21.15.7:0/2023873771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e72b0 0x55f7b29ebdb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3978073910 tx=0x7f396800e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a081b864c6dd6c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.271+0000 7f3963000640 1 -- 172.21.15.7:0/2023873771 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3968004020 con 0x55f7b29e72b0 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.271+0000 7f3963000640 1 -- 172.21.15.7:0/2023873771 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39680041c0 con 0x55f7b29e72b0 2024-03-18T01:40:29.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.271+0000 7f3963000640 1 -- 172.21.15.7:0/2023873771 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39680107b0 con 0x55f7b29e72b0 2024-03-18T01:40:29.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.272+0000 7f397f7bc740 1 -- 172.21.15.7:0/2023873771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e72b0 msgr2=0x55f7b29ebdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.272+0000 7f397f7bc740 1 --2- 172.21.15.7:0/2023873771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e72b0 0x55f7b29ebdb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3978073910 tx=0x7f396800e350 comp rx=0 tx=0).stop 2024-03-18T01:40:29.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.273+0000 7f397f7bc740 1 -- 172.21.15.7:0/2023873771 shutdown_connections 2024-03-18T01:40:29.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.273+0000 7f397f7bc740 1 --2- 172.21.15.7:0/2023873771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29ee8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.273+0000 7f397f7bc740 1 --2- 172.21.15.7:0/2023873771 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e72b0 0x55f7b29ebdb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.273+0000 7f397f7bc740 1 --2- 172.21.15.7:0/2023873771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29e6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.273+0000 7f397f7bc740 1 -- 172.21.15.7:0/2023873771 >> 172.21.15.7:0/2023873771 conn(0x55f7b29e1e30 msgr2=0x55f7b29e2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.274+0000 7f397f7bc740 1 -- 172.21.15.7:0/2023873771 shutdown_connections 2024-03-18T01:40:29.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.274+0000 7f397f7bc740 1 -- 172.21.15.7:0/2023873771 wait complete. 2024-03-18T01:40:29.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.274+0000 7f396ea00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f395c0a29a0 0x7f395c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.275+0000 7f396ea00640 1 -- 172.21.15.7:0/3144789694 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f39700efe80 con 0x7f397810de80 2024-03-18T01:40:29.275 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.275+0000 7f3975e00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f395c0a29a0 0x7f395c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.276+0000 7f3975e00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f395c0a29a0 0x7f395c0a4e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f396800e850 tx=0x7f3968000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.276+0000 7f397f7bc740 1 -- start start 2024-03-18T01:40:29.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.276+0000 7f397f7bc740 1 Processor -- start 2024-03-18T01:40:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.277+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e68c0 0x55f7b29f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.278+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.278+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29f8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.278+0000 7f397f7bc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f7b2907110 con 0x55f7b29e68c0 2024-03-18T01:40:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.278+0000 7f397f7bc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f7b29474f0 con 0x55f7b29ec420 2024-03-18T01:40:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.278+0000 7f397f7bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f7b29ef3d0 con 0x55f7b29e72b0 2024-03-18T01:40:29.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.278+0000 7f397f7bc740 10 client.0 _ll_register_callbacks cb 0x55f7b28f4b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.278+0000 7f3975400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.278+0000 7f3976800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.279+0000 7f3976800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29f8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49382/0 (socket says 172.21.15.7:49382) 2024-03-18T01:40:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.279+0000 7f3976800640 1 -- 172.21.15.7:0/563668672 learned_addr learned my addr 172.21.15.7:0/563668672 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.279+0000 7f3976800640 1 -- 172.21.15.7:0/563668672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 msgr2=0x55f7b29f1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.279+0000 7f3976800640 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.279+0000 7f3976800640 1 -- 172.21.15.7:0/563668672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e68c0 msgr2=0x55f7b29f0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.279+0000 7f3976800640 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e68c0 0x55f7b29f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.279+0000 7f3976800640 1 -- 172.21.15.7:0/563668672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7b265f3f0 con 0x55f7b29ec420 2024-03-18T01:40:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.280+0000 7f3976800640 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29f8150 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f397000ab70 tx=0x7f39700bbee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10f1d04c237d6d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.280+0000 7f3957400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:40:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.280+0000 7f3957400640 1 -- 172.21.15.7:0/563668672 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39700c4c60 con 0x55f7b29ec420 2024-03-18T01:40:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.280+0000 7f3957400640 1 -- 172.21.15.7:0/563668672 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f39700b8b30 con 0x55f7b29ec420 2024-03-18T01:40:29.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.280+0000 7f397f7bc740 1 -- 172.21.15.7:0/563668672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39680041c0 con 0x55f7b29ec420 2024-03-18T01:40:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.281+0000 7f3957400640 1 -- 172.21.15.7:0/563668672 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f39700d5140 con 0x55f7b29ec420 2024-03-18T01:40:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.281+0000 7f397f7bc740 10 client.26093 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.281+0000 7f397f7bc740 1 -- 172.21.15.7:0/563668672 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f7b27297d0 con 0x55f7b29ec420 2024-03-18T01:40:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.282+0000 7f3957400640 1 -- 172.21.15.7:0/563668672 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39700d52e0 con 0x55f7b29ec420 2024-03-18T01:40:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.282+0000 7f3957400640 1 -- 172.21.15.7:0/563668672 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f39700c01e0 con 0x55f7b29ec420 2024-03-18T01:40:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.282+0000 7f397f7bc740 10 client.26093 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:29.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.282+0000 7f397f7bc740 10 client.26093 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:40:29.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.282+0000 7f397f7bc740 20 client.26093 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:29.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.282+0000 7f397f7bc740 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55f7b29fca70 0x55f7b29feef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.282+0000 7f397f7bc740 4 client.26093 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:40:29.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.282+0000 7f397f7bc740 1 -- 172.21.15.7:0/563668672 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f39680041c0 con 0x55f7b29fca70 2024-03-18T01:40:29.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.283+0000 7f3975e00640 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55f7b29fca70 0x55f7b29feef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.286+0000 7f3975e00640 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55f7b29fca70 0x55f7b29feef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=31d6a3163d805e6d server_cookie=f2ba633a8aab6986 in_seq=0 out_seq=0 2024-03-18T01:40:29.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.288+0000 7f3957400640 10 client.26093 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:40:29.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.288+0000 7f3957400640 1 -- 172.21.15.7:0/563668672 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f39680041c0 con 0x55f7b29fca70 2024-03-18T01:40:29.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.288+0000 7f3957400640 10 client.26093 handle_command_reply: tid=0 2024-03-18T01:40:29.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.288+0000 7f397f7bc740 1 client.26093 shutdown 2024-03-18T01:40:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f397f7bc740 1 -- 172.21.15.7:0/563668672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 msgr2=0x55f7b29f8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f397f7bc740 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29f8150 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f397000ab70 tx=0x7f39700bbee0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f397f7bc740 1 -- 172.21.15.7:0/563668672 shutdown_connections 2024-03-18T01:40:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f397f7bc740 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29ec420 0x55f7b29f8150 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f397f7bc740 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55f7b29fca70 0x55f7b29feef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f397f7bc740 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f397f7bc740 1 --2- 172.21.15.7:0/563668672 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29e68c0 0x55f7b29f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f3957400640 0 client.26093 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:40:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.289+0000 7f397f7bc740 1 -- 172.21.15.7:0/563668672 >> 172.21.15.7:0/563668672 conn(0x55f7b29e1e30 msgr2=0x55f7b29e11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.290+0000 7f397f7bc740 1 -- 172.21.15.7:0/563668672 shutdown_connections 2024-03-18T01:40:29.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.290+0000 7f397f7bc740 1 -- 172.21.15.7:0/563668672 wait complete. 2024-03-18T01:40:29.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.292+0000 7f397f7bc740 20 client.26093 trim_cache size 0 max 16384 2024-03-18T01:40:29.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.325+0000 7f397f7bc740 1 Processor -- start 2024-03-18T01:40:29.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.325+0000 7f397f7bc740 1 -- start start 2024-03-18T01:40:29.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e68c0 0x55f7b29e0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29e0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f397f7bc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f39781c1840 con 0x55f7b29ec420 2024-03-18T01:40:29.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f397f7bc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f39781c3440 con 0x55f7b29e68c0 2024-03-18T01:40:29.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f397f7bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f39781c1f10 con 0x55f7b29e72b0 2024-03-18T01:40:29.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f3975e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e68c0 0x55f7b29e0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f3975400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f3975400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58126/0 (socket says 172.21.15.7:58126) 2024-03-18T01:40:29.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.326+0000 7f3975400640 1 -- 172.21.15.7:0/2017200106 learned_addr learned my addr 172.21.15.7:0/2017200106 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.327+0000 7f3975400640 1 -- 172.21.15.7:0/2017200106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e68c0 msgr2=0x55f7b29e0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.327+0000 7f3975400640 1 --2- 172.21.15.7:0/2017200106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e68c0 0x55f7b29e0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.327+0000 7f3975400640 1 -- 172.21.15.7:0/2017200106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 msgr2=0x55f7b29e0ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:29.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.327+0000 7f3976800640 1 --2- 172.21.15.7:0/2017200106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29e0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.327+0000 7f3975400640 1 --2- 172.21.15.7:0/2017200106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29e0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.328+0000 7f3975400640 1 -- 172.21.15.7:0/2017200106 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7b29dd670 con 0x55f7b29e72b0 2024-03-18T01:40:29.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.328+0000 7f3975400640 1 --2- 172.21.15.7:0/2017200106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f8110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f39640126b0 tx=0x7f3964005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89fce34f4b6d7aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.329+0000 7f3962600640 1 -- 172.21.15.7:0/2017200106 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396400e910 con 0x55f7b29e72b0 2024-03-18T01:40:29.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.329+0000 7f3962600640 1 -- 172.21.15.7:0/2017200106 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3964011450 con 0x55f7b29e72b0 2024-03-18T01:40:29.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.329+0000 7f3962600640 1 -- 172.21.15.7:0/2017200106 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396401c650 con 0x55f7b29e72b0 2024-03-18T01:40:29.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.332+0000 7f397f7bc740 1 -- 172.21.15.7:0/2017200106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 msgr2=0x55f7b29f8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.332+0000 7f397f7bc740 1 --2- 172.21.15.7:0/2017200106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f8110 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f39640126b0 tx=0x7f3964005b50 comp rx=0 tx=0).stop 2024-03-18T01:40:29.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.333+0000 7f397f7bc740 1 -- 172.21.15.7:0/2017200106 shutdown_connections 2024-03-18T01:40:29.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.333+0000 7f397f7bc740 1 --2- 172.21.15.7:0/2017200106 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29e0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.333+0000 7f397f7bc740 1 --2- 172.21.15.7:0/2017200106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e72b0 0x55f7b29f8110 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.333+0000 7f397f7bc740 1 --2- 172.21.15.7:0/2017200106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e68c0 0x55f7b29e0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.333+0000 7f397f7bc740 1 -- 172.21.15.7:0/2017200106 >> 172.21.15.7:0/2017200106 conn(0x55f7b29e1e30 msgr2=0x55f7b29e8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.336+0000 7f397f7bc740 1 -- 172.21.15.7:0/2017200106 shutdown_connections 2024-03-18T01:40:29.339 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.339+0000 7f397f7bc740 1 -- 172.21.15.7:0/2017200106 wait complete. 2024-03-18T01:40:29.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.340+0000 7f397f7bc740 1 -- start start 2024-03-18T01:40:29.342 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.340+0000 7f397f7bc740 1 Processor -- start 2024-03-18T01:40:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.342+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29f5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.342+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e72b0 0x55f7b29f6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.342+0000 7f397f7bc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29f7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.342+0000 7f397f7bc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f39781c2cb0 con 0x55f7b29ec420 2024-03-18T01:40:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.342+0000 7f397f7bc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f39781c25e0 con 0x55f7b29e72b0 2024-03-18T01:40:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.342+0000 7f397f7bc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f39781bf820 con 0x55f7b29e68c0 2024-03-18T01:40:29.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.342+0000 7f397f7bc740 10 client.0 _ll_register_callbacks cb 0x7f3970005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.343+0000 7f3975e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29f5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.343+0000 7f3975e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29f5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58138/0 (socket says 172.21.15.7:58138) 2024-03-18T01:40:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.343+0000 7f3975e00640 1 -- 172.21.15.7:0/4124704712 learned_addr learned my addr 172.21.15.7:0/4124704712 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.343+0000 7f3975400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e72b0 0x55f7b29f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.343+0000 7f3976800640 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.344+0000 7f3975e00640 1 -- 172.21.15.7:0/4124704712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e72b0 msgr2=0x55f7b29f6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.344+0000 7f3975e00640 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e72b0 0x55f7b29f6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.344+0000 7f3975e00640 1 -- 172.21.15.7:0/4124704712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 msgr2=0x55f7b29f7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.344+0000 7f3975e00640 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29f7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.344+0000 7f3975e00640 1 -- 172.21.15.7:0/4124704712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7b29f7650 con 0x55f7b29e68c0 2024-03-18T01:40:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.344+0000 7f3976800640 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.345+0000 7f3975e00640 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29f5b10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3978073910 tx=0x7f396800e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9476514b7601460 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.345+0000 7f3957400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:29.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.345+0000 7f3957400640 1 -- 172.21.15.7:0/4124704712 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396800c730 con 0x55f7b29e68c0 2024-03-18T01:40:29.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.345+0000 7f3957400640 1 -- 172.21.15.7:0/4124704712 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f396800ee00 con 0x55f7b29e68c0 2024-03-18T01:40:29.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.346+0000 7f397f7bc740 1 -- 172.21.15.7:0/4124704712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3964011450 con 0x55f7b29e68c0 2024-03-18T01:40:29.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.346+0000 7f3957400640 1 -- 172.21.15.7:0/4124704712 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3968040050 con 0x55f7b29e68c0 2024-03-18T01:40:29.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.347+0000 7f397f7bc740 10 client.26071 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.347+0000 7f397f7bc740 1 -- 172.21.15.7:0/4124704712 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f7b2a631e0 con 0x55f7b29e68c0 2024-03-18T01:40:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f3957400640 1 -- 172.21.15.7:0/4124704712 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3964011450 con 0x55f7b29e68c0 2024-03-18T01:40:29.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f3957400640 1 -- 172.21.15.7:0/4124704712 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f3968003980 con 0x55f7b29e68c0 2024-03-18T01:40:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f397f7bc740 10 client.26071 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f397f7bc740 10 client.26071 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:40:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f397f7bc740 20 client.26071 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f397f7bc740 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55f7b2a63900 0x55f7b2a65d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f397f7bc740 4 client.26071 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:40:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f397f7bc740 1 -- 172.21.15.7:0/4124704712 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7b2958990 con 0x55f7b2a63900 2024-03-18T01:40:29.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.348+0000 7f3975400640 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55f7b2a63900 0x55f7b2a65d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.349+0000 7f3975400640 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55f7b2a63900 0x55f7b2a65d20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a60c447e46b5a96 server_cookie=64d539ee42493878 in_seq=0 out_seq=0 2024-03-18T01:40:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.349+0000 7f3957400640 10 client.26071 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:40:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.349+0000 7f3957400640 1 -- 172.21.15.7:0/4124704712 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7b2958990 con 0x55f7b2a63900 2024-03-18T01:40:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.349+0000 7f3957400640 10 client.26071 handle_command_reply: tid=0 2024-03-18T01:40:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.349+0000 7f397f7bc740 1 client.26071 shutdown 2024-03-18T01:40:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f397f7bc740 1 -- 172.21.15.7:0/4124704712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 msgr2=0x55f7b29f5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.350 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f397f7bc740 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29f5b10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3978073910 tx=0x7f396800e350 comp rx=0 tx=0).stop 2024-03-18T01:40:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f397f7bc740 1 -- 172.21.15.7:0/4124704712 shutdown_connections 2024-03-18T01:40:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f397f7bc740 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f7b29ec420 0x55f7b29f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f397f7bc740 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7b29e72b0 0x55f7b29f6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f397f7bc740 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x55f7b2a63900 0x55f7b2a65d20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f397f7bc740 1 --2- 172.21.15.7:0/4124704712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f7b29e68c0 0x55f7b29f5b10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f397f7bc740 1 -- 172.21.15.7:0/4124704712 >> 172.21.15.7:0/4124704712 conn(0x55f7b29e1e30 msgr2=0x55f7b29e0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.350+0000 7f3957400640 0 client.26071 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:40:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.351+0000 7f397f7bc740 1 -- 172.21.15.7:0/4124704712 shutdown_connections 2024-03-18T01:40:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.351+0000 7f397f7bc740 1 -- 172.21.15.7:0/4124704712 wait complete. 2024-03-18T01:40:29.352 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.351+0000 7f397f7bc740 20 client.26071 trim_cache size 0 max 16384 2024-03-18T01:40:29.352 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:40:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 -- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f395c0a29a0 msgr2=0x7f395c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f395c0a29a0 0x7f395c0a4e60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f396800e850 tx=0x7f3968000bc0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 -- 172.21.15.7:0/3144789694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 msgr2=0x7f39781a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 0x7f39781a5590 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f397002d170 tx=0x7f397000aa40 comp rx=0 tx=0).stop 2024-03-18T01:40:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 -- 172.21.15.7:0/3144789694 shutdown_connections 2024-03-18T01:40:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f397810de80 0x7f39781a5590 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f395c0a29a0 0x7f395c0a4e60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.353 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f397810b380 0x7f397819e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 --2- 172.21.15.7:0/3144789694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39781089c0 0x7f397819dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 -- 172.21.15.7:0/3144789694 >> 172.21.15.7:0/3144789694 conn(0x7f397806e770 msgr2=0x7f397810aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 -- 172.21.15.7:0/3144789694 shutdown_connections 2024-03-18T01:40:29.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.352+0000 7f397ca00640 1 -- 172.21.15.7:0/3144789694 wait complete. 2024-03-18T01:40:29.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T01:40:29.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.729+0000 7f6730200640 1 -- 172.21.15.7:0/161200299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 msgr2=0x7f6728103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.729+0000 7f6730200640 1 --2- 172.21.15.7:0/161200299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 0x7f6728103e40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f671c009900 tx=0x7f671c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:40:29.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.729+0000 7f6730200640 1 -- 172.21.15.7:0/161200299 shutdown_connections 2024-03-18T01:40:29.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.729+0000 7f6730200640 1 --2- 172.21.15.7:0/161200299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67281055d0 0x7f672810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.729+0000 7f6730200640 1 --2- 172.21.15.7:0/161200299 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6728104c10 0x7f6728105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.729+0000 7f6730200640 1 --2- 172.21.15.7:0/161200299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 0x7f6728103e40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.729+0000 7f6730200640 1 -- 172.21.15.7:0/161200299 >> 172.21.15.7:0/161200299 conn(0x7f67280fef90 msgr2=0x7f67281013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.730+0000 7f6730200640 1 -- 172.21.15.7:0/161200299 shutdown_connections 2024-03-18T01:40:29.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.730+0000 7f6730200640 1 -- 172.21.15.7:0/161200299 wait complete. 2024-03-18T01:40:29.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.731+0000 7f6730200640 1 Processor -- start 2024-03-18T01:40:29.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f6730200640 1 -- start start 2024-03-18T01:40:29.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f6730200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 0x7f6728199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f6730200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6728104c10 0x7f672819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f6730200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67281055d0 0x7f67281a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f6730200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f672810f6c0 con 0x7f6728104c10 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f6730200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f672810f540 con 0x7f6728103a20 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f6730200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f672810f840 con 0x7f67281055d0 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f672d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 0x7f6728199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.732+0000 7f672d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 0x7f6728199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49404/0 (socket says 172.21.15.7:49404) 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.733+0000 7f672d800640 1 -- 172.21.15.7:0/2808430747 learned_addr learned my addr 172.21.15.7:0/2808430747 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.733+0000 7f672e200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67281055d0 0x7f67281a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.733+0000 7f672ce00640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6728104c10 0x7f672819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.733+0000 7f672d800640 1 -- 172.21.15.7:0/2808430747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67281055d0 msgr2=0x7f67281a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.733+0000 7f672d800640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67281055d0 0x7f67281a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.733+0000 7f672d800640 1 -- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6728104c10 msgr2=0x7f672819a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.733+0000 7f672d800640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6728104c10 0x7f672819a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.733+0000 7f672d800640 1 -- 172.21.15.7:0/2808430747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67281a18e0 con 0x7f6728103a20 2024-03-18T01:40:29.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.734+0000 7f672d800640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 0x7f6728199ca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f671c02c250 tx=0x7f671c02c5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.735+0000 7f6726000640 1 -- 172.21.15.7:0/2808430747 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c041070 con 0x7f6728103a20 2024-03-18T01:40:29.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.735+0000 7f6726000640 1 -- 172.21.15.7:0/2808430747 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f671c0054e0 con 0x7f6728103a20 2024-03-18T01:40:29.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.735+0000 7f6726000640 1 -- 172.21.15.7:0/2808430747 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c034440 con 0x7f6728103a20 2024-03-18T01:40:29.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.736+0000 7f6730200640 1 -- 172.21.15.7:0/2808430747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67281a1b70 con 0x7f6728103a20 2024-03-18T01:40:29.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.736+0000 7f6730200640 1 -- 172.21.15.7:0/2808430747 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67281a2010 con 0x7f6728103a20 2024-03-18T01:40:29.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.745+0000 7f6726000640 1 -- 172.21.15.7:0/2808430747 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f671c044070 con 0x7f6728103a20 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.747+0000 7f673302b740 1 Processor -- start 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.747+0000 7f673302b740 1 -- start start 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.747+0000 7f673302b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x557533045ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.747+0000 7f673302b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x55753304adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.747+0000 7f673302b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x55753304d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.747+0000 7f673302b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55753300d3b0 con 0x5575330458c0 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.747+0000 7f673302b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557532faa450 con 0x5575330462b0 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.747+0000 7f673302b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557532f52cb0 con 0x55753304b420 2024-03-18T01:40:29.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.748+0000 7f672ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x55753304d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.748+0000 7f672e200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x557533045ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.748+0000 7f672ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x55753304d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58162/0 (socket says 172.21.15.7:58162) 2024-03-18T01:40:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.748+0000 7f672ce00640 1 -- 172.21.15.7:0/2049541393 learned_addr learned my addr 172.21.15.7:0/2049541393 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.749+0000 7f672d800640 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x55753304adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.749+0000 7f672ce00640 1 -- 172.21.15.7:0/2049541393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 msgr2=0x55753304adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.749+0000 7f672ce00640 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x55753304adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.749+0000 7f672ce00640 1 -- 172.21.15.7:0/2049541393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 msgr2=0x557533045ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.750+0000 7f672ce00640 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x557533045ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.750+0000 7f672ce00640 1 -- 172.21.15.7:0/2049541393 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557532f741f0 con 0x55753304b420 2024-03-18T01:40:29.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.750+0000 7f672e200640 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x557533045ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:29.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.750+0000 7f672ce00640 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x55753304d8b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f671800cc40 tx=0x7f671800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2815a3ac29030022 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.751+0000 7f6726000640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67100a2950 0x7f67100a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.751+0000 7f6726000640 1 -- 172.21.15.7:0/2808430747 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f671c0f0a80 con 0x7f6728103a20 2024-03-18T01:40:29.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.752+0000 7f672e200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67100a2950 0x7f67100a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.754+0000 7f672e200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67100a2950 0x7f67100a4e10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x557533046b10 tx=0x7f6720009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.755+0000 7f6716a00640 1 -- 172.21.15.7:0/2049541393 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718014070 con 0x55753304b420 2024-03-18T01:40:29.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.755+0000 7f6716a00640 1 -- 172.21.15.7:0/2049541393 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67180047a0 con 0x55753304b420 2024-03-18T01:40:29.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.755+0000 7f6716a00640 1 -- 172.21.15.7:0/2049541393 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718014070 con 0x55753304b420 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.756+0000 7f673302b740 1 -- 172.21.15.7:0/2049541393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 msgr2=0x55753304d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.756+0000 7f673302b740 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x55753304d8b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f671800cc40 tx=0x7f671800e720 comp rx=0 tx=0).stop 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.756+0000 7f673302b740 1 -- 172.21.15.7:0/2049541393 shutdown_connections 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.756+0000 7f673302b740 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x55753304d8b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.756+0000 7f673302b740 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x55753304adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.756+0000 7f673302b740 1 --2- 172.21.15.7:0/2049541393 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x557533045ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.756+0000 7f673302b740 1 -- 172.21.15.7:0/2049541393 >> 172.21.15.7:0/2049541393 conn(0x557533040e30 msgr2=0x557533041230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.756+0000 7f673302b740 1 -- 172.21.15.7:0/2049541393 shutdown_connections 2024-03-18T01:40:29.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.757+0000 7f673302b740 1 -- 172.21.15.7:0/2049541393 wait complete. 2024-03-18T01:40:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.757+0000 7f673302b740 1 -- start start 2024-03-18T01:40:29.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.758+0000 7f673302b740 1 Processor -- start 2024-03-18T01:40:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f673302b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330458c0 0x55753304fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f673302b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330462b0 0x5575330500d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f673302b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x557533057110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f673302b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557532f66110 con 0x5575330462b0 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f673302b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557532fa64f0 con 0x5575330458c0 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f673302b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55753304e3d0 con 0x55753304b420 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f673302b740 10 client.0 _ll_register_callbacks cb 0x557532f53b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f672e200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x557533057110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f672e200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x557533057110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58172/0 (socket says 172.21.15.7:58172) 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f672e200640 1 -- 172.21.15.7:0/2359250391 learned_addr learned my addr 172.21.15.7:0/2359250391 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.760+0000 7f672d800640 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330458c0 0x55753304fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.761+0000 7f672e200640 1 -- 172.21.15.7:0/2359250391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330458c0 msgr2=0x55753304fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.761+0000 7f672e200640 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330458c0 0x55753304fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.761+0000 7f672e200640 1 -- 172.21.15.7:0/2359250391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330462b0 msgr2=0x5575330500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.761+0000 7f672e200640 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330462b0 0x5575330500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.761+0000 7f672e200640 1 -- 172.21.15.7:0/2359250391 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557532cbe3f0 con 0x55753304b420 2024-03-18T01:40:29.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.761+0000 7f672e200640 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x557533057110 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6720009c90 tx=0x7f6720004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df772ae41368c943 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.762+0000 7f670b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.762+0000 7f670b400640 1 -- 172.21.15.7:0/2359250391 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720044070 con 0x55753304b420 2024-03-18T01:40:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.762+0000 7f670b400640 1 -- 172.21.15.7:0/2359250391 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f672000fe00 con 0x55753304b420 2024-03-18T01:40:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.763+0000 7f673302b740 1 -- 172.21.15.7:0/2359250391 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67180047a0 con 0x55753304b420 2024-03-18T01:40:29.764 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.763+0000 7f670b400640 1 -- 172.21.15.7:0/2359250391 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6720049460 con 0x55753304b420 2024-03-18T01:40:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.763+0000 7f673302b740 10 client.26083 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.763+0000 7f673302b740 1 -- 172.21.15.7:0/2359250391 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557532d887d0 con 0x55753304b420 2024-03-18T01:40:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.763+0000 7f670b400640 1 -- 172.21.15.7:0/2359250391 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67180047a0 con 0x55753304b420 2024-03-18T01:40:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.765+0000 7f670b400640 1 -- 172.21.15.7:0/2359250391 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f672000f050 con 0x55753304b420 2024-03-18T01:40:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.765+0000 7f673302b740 10 client.26083 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.765+0000 7f673302b740 10 client.26083 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:40:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.765+0000 7f673302b740 20 client.26083 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:29.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.765+0000 7f673302b740 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557533059d20 0x55753305c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.766+0000 7f673302b740 4 client.26083 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:40:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.766+0000 7f673302b740 1 -- 172.21.15.7:0/2359250391 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557532fb7990 con 0x557533059d20 2024-03-18T01:40:29.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.766+0000 7f672d800640 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557533059d20 0x55753305c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.770+0000 7f672d800640 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557533059d20 0x55753305c1e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1bd2f24a7cf6873a server_cookie=b5362a151933b023 in_seq=0 out_seq=0 2024-03-18T01:40:29.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.770+0000 7f670b400640 10 client.26083 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:40:29.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.774+0000 7f670b400640 1 -- 172.21.15.7:0/2359250391 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557532fb7990 con 0x557533059d20 2024-03-18T01:40:29.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.774+0000 7f670b400640 10 client.26083 handle_command_reply: tid=0 2024-03-18T01:40:29.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.774+0000 7f673302b740 1 client.26083 shutdown 2024-03-18T01:40:29.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f673302b740 1 -- 172.21.15.7:0/2359250391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 msgr2=0x557533057110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f673302b740 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x557533057110 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6720009c90 tx=0x7f6720004a40 comp rx=0 tx=0).stop 2024-03-18T01:40:29.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f673302b740 1 -- 172.21.15.7:0/2359250391 shutdown_connections 2024-03-18T01:40:29.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f673302b740 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557533059d20 0x55753305c1e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f670b400640 0 client.26083 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:40:29.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f673302b740 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x557533057110 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f673302b740 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330462b0 0x5575330500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f673302b740 1 --2- 172.21.15.7:0/2359250391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330458c0 0x55753304fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.775+0000 7f673302b740 1 -- 172.21.15.7:0/2359250391 >> 172.21.15.7:0/2359250391 conn(0x557533040e30 msgr2=0x5575330401f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.776+0000 7f673302b740 1 -- 172.21.15.7:0/2359250391 shutdown_connections 2024-03-18T01:40:29.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.776+0000 7f673302b740 1 -- 172.21.15.7:0/2359250391 wait complete. 2024-03-18T01:40:29.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.776+0000 7f673302b740 20 client.26083 trim_cache size 0 max 16384 2024-03-18T01:40:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.807+0000 7f673302b740 1 Processor -- start 2024-03-18T01:40:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.808+0000 7f673302b740 1 -- start start 2024-03-18T01:40:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.808+0000 7f673302b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x5575330476d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.808+0000 7f673302b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x557533047c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.808+0000 7f673302b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330570d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.808+0000 7f673302b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f67281a3de0 con 0x5575330458c0 2024-03-18T01:40:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.808+0000 7f673302b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67281a59e0 con 0x5575330462b0 2024-03-18T01:40:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.808+0000 7f673302b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67281a44b0 con 0x55753304b420 2024-03-18T01:40:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.809+0000 7f672ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x557533047c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.809+0000 7f672ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x557533047c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49430/0 (socket says 172.21.15.7:49430) 2024-03-18T01:40:29.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.809+0000 7f672ce00640 1 -- 172.21.15.7:0/685267348 learned_addr learned my addr 172.21.15.7:0/685267348 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.809+0000 7f672d800640 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x5575330476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.810+0000 7f672ce00640 1 -- 172.21.15.7:0/685267348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 msgr2=0x5575330570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.810+0000 7f672e200640 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.810+0000 7f672ce00640 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.810+0000 7f672ce00640 1 -- 172.21.15.7:0/685267348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 msgr2=0x5575330476d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.810+0000 7f672ce00640 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x5575330476d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.810+0000 7f672ce00640 1 -- 172.21.15.7:0/685267348 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55753304b170 con 0x5575330462b0 2024-03-18T01:40:29.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.810+0000 7f672d800640 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x5575330476d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:29.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.810+0000 7f672e200640 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.811+0000 7f672ce00640 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x557533047c10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6718005e10 tx=0x7f671800c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c91e12768627588a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.812+0000 7f6716000640 1 -- 172.21.15.7:0/685267348 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718014070 con 0x5575330462b0 2024-03-18T01:40:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.812+0000 7f6716000640 1 -- 172.21.15.7:0/685267348 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67180035f0 con 0x5575330462b0 2024-03-18T01:40:29.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.812+0000 7f6716000640 1 -- 172.21.15.7:0/685267348 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718010700 con 0x5575330462b0 2024-03-18T01:40:29.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.813+0000 7f673302b740 1 -- 172.21.15.7:0/685267348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 msgr2=0x557533047c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.813+0000 7f673302b740 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x557533047c10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6718005e10 tx=0x7f671800c350 comp rx=0 tx=0).stop 2024-03-18T01:40:29.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.814+0000 7f673302b740 1 -- 172.21.15.7:0/685267348 shutdown_connections 2024-03-18T01:40:29.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.814+0000 7f673302b740 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.814+0000 7f673302b740 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330462b0 0x557533047c10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.814+0000 7f673302b740 1 --2- 172.21.15.7:0/685267348 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x5575330476d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.814+0000 7f673302b740 1 -- 172.21.15.7:0/685267348 >> 172.21.15.7:0/685267348 conn(0x557533040e30 msgr2=0x55753303c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.814+0000 7f673302b740 1 -- 172.21.15.7:0/685267348 shutdown_connections 2024-03-18T01:40:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.814+0000 7f673302b740 1 -- 172.21.15.7:0/685267348 wait complete. 2024-03-18T01:40:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.815+0000 7f673302b740 1 -- start start 2024-03-18T01:40:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.815+0000 7f673302b740 1 Processor -- start 2024-03-18T01:40:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.817+0000 7f673302b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x5575330532a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.817+0000 7f673302b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f673302b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330563f0 0x557533056870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f672ce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f672e200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330563f0 0x557533056870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f672e200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330563f0 0x557533056870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49438/0 (socket says 172.21.15.7:49438) 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f672e200640 1 -- 172.21.15.7:0/3263467855 learned_addr learned my addr 172.21.15.7:0/3263467855 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f67281a5250 con 0x5575330458c0 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f67281a4b80 con 0x5575330563f0 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6728073530 con 0x55753304b420 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.818+0000 7f673302b740 10 client.0 _ll_register_callbacks cb 0x7f6720009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.819+0000 7f672e200640 1 -- 172.21.15.7:0/3263467855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 msgr2=0x5575330537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.819+0000 7f672e200640 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.819+0000 7f672e200640 1 -- 172.21.15.7:0/3263467855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 msgr2=0x5575330532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:29.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.819+0000 7f672e200640 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x5575330532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.819+0000 7f672e200640 1 -- 172.21.15.7:0/3263467855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557533056db0 con 0x5575330563f0 2024-03-18T01:40:29.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.820+0000 7f672e200640 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330563f0 0x557533056870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f672000a010 tx=0x7f672000feb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=145236a6f1bb97c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:29.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.820+0000 7f670b400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:40:29.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.820+0000 7f670b400640 1 -- 172.21.15.7:0/3263467855 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720061060 con 0x5575330563f0 2024-03-18T01:40:29.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.820+0000 7f670b400640 1 -- 172.21.15.7:0/3263467855 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6720053620 con 0x5575330563f0 2024-03-18T01:40:29.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.821+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67180035f0 con 0x5575330563f0 2024-03-18T01:40:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.823+0000 7f670b400640 1 -- 172.21.15.7:0/3263467855 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6720044070 con 0x5575330563f0 2024-03-18T01:40:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.823+0000 7f673302b740 10 client.26117 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.823+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575330c1f90 con 0x5575330563f0 2024-03-18T01:40:29.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.823+0000 7f670b400640 1 -- 172.21.15.7:0/3263467855 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67180035f0 con 0x5575330563f0 2024-03-18T01:40:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.824+0000 7f670b400640 1 -- 172.21.15.7:0/3263467855 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f6720049b90 con 0x5575330563f0 2024-03-18T01:40:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.824+0000 7f673302b740 10 client.26117 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.824+0000 7f673302b740 10 client.26117 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:40:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.824+0000 7f673302b740 20 client.26117 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.824+0000 7f673302b740 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5575330c2920 0x5575330c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.824+0000 7f673302b740 4 client.26117 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:40:29.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.824+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557532fb1ce0 con 0x5575330c2920 2024-03-18T01:40:29.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.827+0000 7f672d800640 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5575330c2920 0x5575330c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:29.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.829+0000 7f672d800640 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5575330c2920 0x5575330c4d40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4780866feff37c51 server_cookie=10ed6896e125d3aa in_seq=0 out_seq=0 2024-03-18T01:40:29.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.829+0000 7f670b400640 10 client.26117 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:40:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.830+0000 7f670b400640 1 -- 172.21.15.7:0/3263467855 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557532fb1ce0 con 0x5575330c2920 2024-03-18T01:40:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.830+0000 7f670b400640 10 client.26117 handle_command_reply: tid=0 2024-03-18T01:40:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.830+0000 7f673302b740 1 client.26117 shutdown 2024-03-18T01:40:29.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.830+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330563f0 msgr2=0x557533056870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.830+0000 7f673302b740 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330563f0 0x557533056870 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f672000a010 tx=0x7f672000feb0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.833+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 shutdown_connections 2024-03-18T01:40:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.833+0000 7f673302b740 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5575330c2920 0x5575330c4d40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.833+0000 7f673302b740 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5575330563f0 0x557533056870 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.833+0000 7f673302b740 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55753304b420 0x5575330537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.833+0000 7f673302b740 1 --2- 172.21.15.7:0/3263467855 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5575330458c0 0x5575330532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.833+0000 7f670b400640 0 client.26117 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:40:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.833+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 >> 172.21.15.7:0/3263467855 conn(0x557533040e30 msgr2=0x55753304fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.834+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 shutdown_connections 2024-03-18T01:40:29.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.834+0000 7f673302b740 1 -- 172.21.15.7:0/3263467855 wait complete. 2024-03-18T01:40:29.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.834+0000 7f673302b740 20 client.26117 trim_cache size 0 max 16384 2024-03-18T01:40:29.835 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:40:29.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 -- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67100a2950 msgr2=0x7f67100a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67100a2950 0x7f67100a4e10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x557533046b10 tx=0x7f6720009500 comp rx=0 tx=0).stop 2024-03-18T01:40:29.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 -- 172.21.15.7:0/2808430747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 msgr2=0x7f6728199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 0x7f6728199ca0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f671c02c250 tx=0x7f671c02c5e0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 -- 172.21.15.7:0/2808430747 shutdown_connections 2024-03-18T01:40:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67281055d0 0x7f67281a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f67100a2950 0x7f67100a4e10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6728104c10 0x7f672819a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 --2- 172.21.15.7:0/2808430747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6728103a20 0x7f6728199ca0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.836+0000 7f6730200640 1 -- 172.21.15.7:0/2808430747 >> 172.21.15.7:0/2808430747 conn(0x7f67280fef90 msgr2=0x7f6728107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.837+0000 7f6730200640 1 -- 172.21.15.7:0/2808430747 shutdown_connections 2024-03-18T01:40:29.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:29.837+0000 7f6730200640 1 -- 172.21.15.7:0/2808430747 wait complete. 2024-03-18T01:40:29.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:40:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.191+0000 7f556a800640 1 -- 172.21.15.7:0/3368551019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55641009d0 msgr2=0x7f5564100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.191+0000 7f556a800640 1 --2- 172.21.15.7:0/3368551019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55641009d0 0x7f5564100df0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f555c009ca0 tx=0x7f555c02c140 comp rx=0 tx=0).stop 2024-03-18T01:40:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.192+0000 7f556a800640 1 -- 172.21.15.7:0/3368551019 shutdown_connections 2024-03-18T01:40:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.192+0000 7f556a800640 1 --2- 172.21.15.7:0/3368551019 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55640ffe60 0x7f5564101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.192+0000 7f556a800640 1 --2- 172.21.15.7:0/3368551019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55640ff4a0 0x7f55640ff920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.192+0000 7f556a800640 1 --2- 172.21.15.7:0/3368551019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55641009d0 0x7f5564100df0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.192+0000 7f556a800640 1 -- 172.21.15.7:0/3368551019 >> 172.21.15.7:0/3368551019 conn(0x7f55640faf60 msgr2=0x7f55640fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:30.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.192+0000 7f556a800640 1 -- 172.21.15.7:0/3368551019 shutdown_connections 2024-03-18T01:40:30.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.192+0000 7f556a800640 1 -- 172.21.15.7:0/3368551019 wait complete. 2024-03-18T01:40:30.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.193+0000 7f556a800640 1 Processor -- start 2024-03-18T01:40:30.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.193+0000 7f556a800640 1 -- start start 2024-03-18T01:40:30.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f556a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55640ff4a0 0x7f55641068e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f556a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640ffe60 0x7f5564106e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f556a800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55641009d0 0x7f5564103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f556a800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f556410f6e0 con 0x7f55641009d0 2024-03-18T01:40:30.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f556a800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f556410f560 con 0x7f55640ffe60 2024-03-18T01:40:30.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f556a800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f556410f860 con 0x7f55640ff4a0 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f5562a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640ffe60 0x7f5564106e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f5562a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640ffe60 0x7f5564106e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49448/0 (socket says 172.21.15.7:49448) 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.194+0000 7f5562a00640 1 -- 172.21.15.7:0/3715670595 learned_addr learned my addr 172.21.15.7:0/3715670595 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.195+0000 7f5562a00640 1 -- 172.21.15.7:0/3715670595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55640ff4a0 msgr2=0x7f55641068e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.195+0000 7f5562a00640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55640ff4a0 0x7f55641068e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.195+0000 7f5562a00640 1 -- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55641009d0 msgr2=0x7f5564103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.195+0000 7f5562a00640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55641009d0 0x7f5564103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.195+0000 7f5562a00640 1 -- 172.21.15.7:0/3715670595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5564103d10 con 0x7f55640ffe60 2024-03-18T01:40:30.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.195+0000 7f5562a00640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640ffe60 0x7f5564106e20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f555000b470 tx=0x7f555000b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.197+0000 7f555be00640 1 -- 172.21.15.7:0/3715670595 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550015020 con 0x7f55640ffe60 2024-03-18T01:40:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.197+0000 7f555be00640 1 -- 172.21.15.7:0/3715670595 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5550011d70 con 0x7f55640ffe60 2024-03-18T01:40:30.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.197+0000 7f555be00640 1 -- 172.21.15.7:0/3715670595 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555000f340 con 0x7f55640ffe60 2024-03-18T01:40:30.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.198+0000 7f556a800640 1 -- 172.21.15.7:0/3715670595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55641999a0 con 0x7f55640ffe60 2024-03-18T01:40:30.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.198+0000 7f556a800640 1 -- 172.21.15.7:0/3715670595 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5564199f20 con 0x7f55640ffe60 2024-03-18T01:40:30.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.211+0000 7f556d57e740 1 Processor -- start 2024-03-18T01:40:30.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.212+0000 7f555be00640 1 -- 172.21.15.7:0/3715670595 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f5550018020 con 0x7f55640ffe60 2024-03-18T01:40:30.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.213+0000 7f556d57e740 1 -- start start 2024-03-18T01:40:30.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.213+0000 7f556d57e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf7fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.214 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.214+0000 7f556d57e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf802b0 0x557f8bf84db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.214+0000 7f556d57e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf85420 0x557f8bf878b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.214+0000 7f556d57e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557f8bf473b0 con 0x557f8bf7f8c0 2024-03-18T01:40:30.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.214+0000 7f556d57e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557f8bee4450 con 0x557f8bf85420 2024-03-18T01:40:30.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.214+0000 7f556d57e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557f8be8ccb0 con 0x557f8bf802b0 2024-03-18T01:40:30.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.214+0000 7f5563e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf7fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.214+0000 7f5563e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf7fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34838/0 (socket says 172.21.15.7:34838) 2024-03-18T01:40:30.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.214+0000 7f5563e00640 1 -- 172.21.15.7:0/3322825342 learned_addr learned my addr 172.21.15.7:0/3322825342 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:30.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.215+0000 7f5562a00640 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf85420 0x557f8bf878b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.215+0000 7f5563e00640 1 -- 172.21.15.7:0/3322825342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf802b0 msgr2=0x557f8bf84db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:30.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.215+0000 7f5563e00640 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf802b0 0x557f8bf84db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.215+0000 7f5563e00640 1 -- 172.21.15.7:0/3322825342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf85420 msgr2=0x557f8bf878b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:30.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.217+0000 7f555be00640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55480a2880 0x7f55480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.217+0000 7f5563e00640 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf85420 0x557f8bf878b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.217+0000 7f5563e00640 1 -- 172.21.15.7:0/3322825342 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f8beae1f0 con 0x557f8bf7f8c0 2024-03-18T01:40:30.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.217+0000 7f5562a00640 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf85420 0x557f8bf878b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:30.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.218+0000 7f555be00640 1 -- 172.21.15.7:0/3715670595 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f55500d1170 con 0x7f55640ffe60 2024-03-18T01:40:30.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.219+0000 7f5563400640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55480a2880 0x7f55480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.220+0000 7f5563e00640 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf7fce0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f555400ba90 tx=0x7f555400bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ee59a446c6821a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:30.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.220+0000 7f5558c00640 1 -- 172.21.15.7:0/3322825342 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554016070 con 0x557f8bf7f8c0 2024-03-18T01:40:30.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.220+0000 7f5558c00640 1 -- 172.21.15.7:0/3322825342 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5554005500 con 0x557f8bf7f8c0 2024-03-18T01:40:30.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.220+0000 7f5558c00640 1 -- 172.21.15.7:0/3322825342 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554010460 con 0x557f8bf7f8c0 2024-03-18T01:40:30.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.222+0000 7f556d57e740 1 -- 172.21.15.7:0/3322825342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 msgr2=0x557f8bf7fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:30.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.222+0000 7f556d57e740 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf7fce0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f555400ba90 tx=0x7f555400bf50 comp rx=0 tx=0).stop 2024-03-18T01:40:30.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.222+0000 7f5563400640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55480a2880 0x7f55480a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f555c0325e0 tx=0x7f555c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.222+0000 7f556d57e740 1 -- 172.21.15.7:0/3322825342 shutdown_connections 2024-03-18T01:40:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.222+0000 7f556d57e740 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf85420 0x557f8bf878b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.223+0000 7f556d57e740 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf802b0 0x557f8bf84db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.223+0000 7f556d57e740 1 --2- 172.21.15.7:0/3322825342 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf7fce0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.223+0000 7f556d57e740 1 -- 172.21.15.7:0/3322825342 >> 172.21.15.7:0/3322825342 conn(0x557f8bf7ae30 msgr2=0x557f8bf7b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.223+0000 7f556d57e740 1 -- 172.21.15.7:0/3322825342 shutdown_connections 2024-03-18T01:40:30.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.224+0000 7f556d57e740 1 -- 172.21.15.7:0/3322825342 wait complete. 2024-03-18T01:40:30.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.224+0000 7f556d57e740 1 -- start start 2024-03-18T01:40:30.225 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.225+0000 7f556d57e740 1 Processor -- start 2024-03-18T01:40:30.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.226+0000 7f556d57e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.226+0000 7f556d57e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.226+0000 7f556d57e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf911a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f5562a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f5562a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49460/0 (socket says 172.21.15.7:49460) 2024-03-18T01:40:30.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f556d57e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557f8bea0110 con 0x557f8bf7f8c0 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f5563400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f5563e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf911a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f5563e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf911a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58202/0 (socket says 172.21.15.7:58202) 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f5563e00640 1 -- 172.21.15.7:0/1349738206 learned_addr learned my addr 172.21.15.7:0/1349738206 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557f8bee04f0 con 0x557f8bf802b0 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.227+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557f8bf883d0 con 0x557f8bf85420 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.228+0000 7f5563e00640 1 -- 172.21.15.7:0/1349738206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 msgr2=0x557f8bf8a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.228+0000 7f5563e00640 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.228+0000 7f5563e00640 1 -- 172.21.15.7:0/1349738206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 msgr2=0x557f8bf89c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:30.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.228+0000 7f556d57e740 10 client.0 _ll_register_callbacks cb 0x557f8be8db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.228+0000 7f5563e00640 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.229+0000 7f5563e00640 1 -- 172.21.15.7:0/1349738206 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f8bbf83f0 con 0x557f8bf85420 2024-03-18T01:40:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.229+0000 7f5563400640 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.229+0000 7f5563e00640 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf911a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f555400b7d0 tx=0x7f55540047a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f0fadffa092bded server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.230+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5554005500 con 0x557f8bf85420 2024-03-18T01:40:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.230+0000 7f554d600640 10 client.26089 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:30.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.230+0000 7f554d600640 1 -- 172.21.15.7:0/1349738206 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554016030 con 0x557f8bf85420 2024-03-18T01:40:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.230+0000 7f554d600640 1 -- 172.21.15.7:0/1349738206 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5554005230 con 0x557f8bf85420 2024-03-18T01:40:30.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.231+0000 7f554d600640 1 -- 172.21.15.7:0/1349738206 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5554016030 con 0x557f8bf85420 2024-03-18T01:40:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.232+0000 7f556d57e740 10 client.26089 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.232+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=31}) -- 0x557f8bcc27d0 con 0x557f8bf85420 2024-03-18T01:40:30.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.232+0000 7f554d600640 1 -- 172.21.15.7:0/1349738206 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f555400daa0 con 0x557f8bf85420 2024-03-18T01:40:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.233+0000 7f556d57e740 10 client.26089 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.233+0000 7f556d57e740 10 client.26089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:40:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.233+0000 7f556d57e740 20 client.26089 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:30.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.233+0000 7f556d57e740 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557f8bf95460 0x557f8bf978e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.233+0000 7f556d57e740 4 client.26089 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:40:30.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.233+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f8bef1990 con 0x557f8bf95460 2024-03-18T01:40:30.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.233+0000 7f5563400640 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557f8bf95460 0x557f8bf978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.234+0000 7f5563400640 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557f8bf95460 0x557f8bf978e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66da99c317e0ed1d server_cookie=9e19ccac6733dd14 in_seq=0 out_seq=0 2024-03-18T01:40:30.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.234+0000 7f554d600640 10 client.26089 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.235+0000 7f554d600640 1 -- 172.21.15.7:0/1349738206 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f8bef1990 con 0x557f8bf95460 2024-03-18T01:40:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.235+0000 7f554d600640 10 client.26089 handle_command_reply: tid=0 2024-03-18T01:40:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 client.26089 shutdown 2024-03-18T01:40:30.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 msgr2=0x557f8bf911a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf911a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f555400b7d0 tx=0x7f55540047a0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 shutdown_connections 2024-03-18T01:40:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557f8bf95460 0x557f8bf978e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf911a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f554d600640 0 client.26089 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 --2- 172.21.15.7:0/1349738206 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.236+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 >> 172.21.15.7:0/1349738206 conn(0x557f8bf7ae30 msgr2=0x557f8bf7a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:30.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.237+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 shutdown_connections 2024-03-18T01:40:30.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.237+0000 7f556d57e740 1 -- 172.21.15.7:0/1349738206 wait complete. 2024-03-18T01:40:30.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.239+0000 7f556d57e740 20 client.26089 trim_cache size 0 max 16384 2024-03-18T01:40:30.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.288+0000 7f556d57e740 1 Processor -- start 2024-03-18T01:40:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.289+0000 7f556d57e740 1 -- start start 2024-03-18T01:40:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.289+0000 7f556d57e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.289+0000 7f556d57e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf802b0 0x557f8bf91160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.289+0000 7f556d57e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.289+0000 7f556d57e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f55641b93a0 con 0x557f8bf802b0 2024-03-18T01:40:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.289+0000 7f556d57e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f55641bafa0 con 0x557f8bf7f8c0 2024-03-18T01:40:30.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.289+0000 7f556d57e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55641b9a70 con 0x557f8bf85420 2024-03-18T01:40:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49462/0 (socket says 172.21.15.7:49462) 2024-03-18T01:40:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563400640 1 -- 172.21.15.7:0/3772613737 learned_addr learned my addr 172.21.15.7:0/3772613737 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563e00640 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563400640 1 -- 172.21.15.7:0/3772613737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 msgr2=0x557f8bf8a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:30.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563400640 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563400640 1 -- 172.21.15.7:0/3772613737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf802b0 msgr2=0x557f8bf91160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:40:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563400640 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf802b0 0x557f8bf91160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.290+0000 7f5563400640 1 -- 172.21.15.7:0/3772613737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f8bf76c10 con 0x557f8bf7f8c0 2024-03-18T01:40:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.291+0000 7f5563e00640 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.291+0000 7f5563400640 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f555c0324b0 tx=0x7f555c03b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ce0a4f72d5c7072 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.291+0000 7f554fe00640 1 -- 172.21.15.7:0/3772613737 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c039040 con 0x557f8bf7f8c0 2024-03-18T01:40:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.291+0000 7f554fe00640 1 -- 172.21.15.7:0/3772613737 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f555c053030 con 0x557f8bf7f8c0 2024-03-18T01:40:30.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.291+0000 7f554fe00640 1 -- 172.21.15.7:0/3772613737 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c034640 con 0x557f8bf7f8c0 2024-03-18T01:40:30.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.295+0000 7f556d57e740 1 -- 172.21.15.7:0/3772613737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf7f8c0 msgr2=0x557f8bf89c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:30.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.295+0000 7f556d57e740 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f555c0324b0 tx=0x7f555c03b000 comp rx=0 tx=0).stop 2024-03-18T01:40:30.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.295+0000 7f556d57e740 1 -- 172.21.15.7:0/3772613737 shutdown_connections 2024-03-18T01:40:30.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.295+0000 7f556d57e740 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.295+0000 7f556d57e740 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf802b0 0x557f8bf91160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.295+0000 7f556d57e740 1 --2- 172.21.15.7:0/3772613737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf7f8c0 0x557f8bf89c20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.295+0000 7f556d57e740 1 -- 172.21.15.7:0/3772613737 >> 172.21.15.7:0/3772613737 conn(0x557f8bf7ae30 msgr2=0x557f8bf7b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:30.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.296+0000 7f556d57e740 1 -- 172.21.15.7:0/3772613737 shutdown_connections 2024-03-18T01:40:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.296+0000 7f556d57e740 1 -- 172.21.15.7:0/3772613737 wait complete. 2024-03-18T01:40:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.297+0000 7f556d57e740 1 -- start start 2024-03-18T01:40:30.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.297+0000 7f556d57e740 1 Processor -- start 2024-03-18T01:40:30.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.298+0000 7f556d57e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf8ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.299+0000 7f556d57e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.299+0000 7f556d57e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.299+0000 7f5563e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.299+0000 7f5563e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58220/0 (socket says 172.21.15.7:58220) 2024-03-18T01:40:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.299+0000 7f5563e00640 1 -- 172.21.15.7:0/1470637260 learned_addr learned my addr 172.21.15.7:0/1470637260 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.300+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f55641ba810 con 0x557f8bf7f8c0 2024-03-18T01:40:30.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.300+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f55641ba140 con 0x557f8bf802b0 2024-03-18T01:40:30.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.300+0000 7f5562a00640 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.300+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55641b7380 con 0x557f8bf85420 2024-03-18T01:40:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.301+0000 7f5563e00640 1 -- 172.21.15.7:0/1470637260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 msgr2=0x557f8bf8ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.301+0000 7f5563e00640 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.301+0000 7f5563e00640 1 -- 172.21.15.7:0/1470637260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 msgr2=0x557f8bf8ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.301+0000 7f5563e00640 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf8ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.301+0000 7f5563e00640 1 -- 172.21.15.7:0/1470637260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f8bf904f0 con 0x557f8bf85420 2024-03-18T01:40:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.301+0000 7f556d57e740 10 client.0 _ll_register_callbacks cb 0x7f555400b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:30.302 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.301+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f555c053030 con 0x557f8bf85420 2024-03-18T01:40:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.302+0000 7f554d600640 10 client.26092 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.302+0000 7f5563e00640 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8ff00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f555400b820 tx=0x7f555400b380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8abd509bfafb7af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:30.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.303+0000 7f554d600640 1 -- 172.21.15.7:0/1470637260 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554036030 con 0x557f8bf85420 2024-03-18T01:40:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.304+0000 7f554d600640 1 -- 172.21.15.7:0/1470637260 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f555402a070 con 0x557f8bf85420 2024-03-18T01:40:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.304+0000 7f554d600640 1 -- 172.21.15.7:0/1470637260 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5554004a80 con 0x557f8bf85420 2024-03-18T01:40:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.304+0000 7f556d57e740 10 client.26092 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:30.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.304+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=31}) -- 0x557f8bffb850 con 0x557f8bf85420 2024-03-18T01:40:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.306+0000 7f554d600640 1 -- 172.21.15.7:0/1470637260 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f55540040a0 con 0x557f8bf85420 2024-03-18T01:40:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.306+0000 7f556d57e740 10 client.26092 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.306+0000 7f556d57e740 10 client.26092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:40:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.306+0000 7f556d57e740 20 client.26092 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.306+0000 7f556d57e740 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557f8bffc120 0x557f8bffe540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:30.306 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.306+0000 7f556d57e740 4 client.26092 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:40:30.307 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.306+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f8beebce0 con 0x557f8bffc120 2024-03-18T01:40:30.309 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.309+0000 7f5563400640 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557f8bffc120 0x557f8bffe540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.309+0000 7f554d600640 10 client.26092 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.309+0000 7f5563400640 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557f8bffc120 0x557f8bffe540 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2c849748b7f93ef server_cookie=a2aaf91102e265f6 in_seq=0 out_seq=0 2024-03-18T01:40:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.310+0000 7f554d600640 1 -- 172.21.15.7:0/1470637260 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f8beebce0 con 0x557f8bffc120 2024-03-18T01:40:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.310+0000 7f554d600640 10 client.26092 handle_command_reply: tid=0 2024-03-18T01:40:30.310 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.310+0000 7f556d57e740 1 client.26092 shutdown 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.310+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 msgr2=0x557f8bf8ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.310+0000 7f556d57e740 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8ff00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f555400b820 tx=0x7f555400b380 comp rx=0 tx=0).stop 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 shutdown_connections 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f556d57e740 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557f8bffc120 0x557f8bffe540 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f556d57e740 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8bf85420 0x557f8bf8ff00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f556d57e740 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557f8bf802b0 0x557f8bf8ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f556d57e740 1 --2- 172.21.15.7:0/1470637260 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557f8bf7f8c0 0x557f8bf8ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 >> 172.21.15.7:0/1470637260 conn(0x557f8bf7ae30 msgr2=0x557f8bf89c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:30.311 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f554d600640 0 client.26092 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 shutdown_connections 2024-03-18T01:40:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.311+0000 7f556d57e740 1 -- 172.21.15.7:0/1470637260 wait complete. 2024-03-18T01:40:30.312 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.312+0000 7f556d57e740 20 client.26092 trim_cache size 0 max 16384 2024-03-18T01:40:30.313 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:40:30.313 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.313+0000 7f556a800640 1 -- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55480a2880 msgr2=0x7f55480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:30.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.313+0000 7f556a800640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55480a2880 0x7f55480a4d40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f555c0325e0 tx=0x7f555c000f40 comp rx=0 tx=0).stop 2024-03-18T01:40:30.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.313+0000 7f556a800640 1 -- 172.21.15.7:0/3715670595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640ffe60 msgr2=0x7f5564106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:30.314 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.313+0000 7f556a800640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640ffe60 0x7f5564106e20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f555000b470 tx=0x7f555000b930 comp rx=0 tx=0).stop 2024-03-18T01:40:30.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.314+0000 7f556a800640 1 -- 172.21.15.7:0/3715670595 shutdown_connections 2024-03-18T01:40:30.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.314+0000 7f556a800640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f55480a2880 0x7f55480a4d40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.314+0000 7f556a800640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f55641009d0 0x7f5564103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.314+0000 7f556a800640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55640ffe60 0x7f5564106e20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.314+0000 7f556a800640 1 --2- 172.21.15.7:0/3715670595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55640ff4a0 0x7f55641068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:30.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.314+0000 7f556a800640 1 -- 172.21.15.7:0/3715670595 >> 172.21.15.7:0/3715670595 conn(0x7f55640faf60 msgr2=0x7f55640fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:30.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.315+0000 7f556a800640 1 -- 172.21.15.7:0/3715670595 shutdown_connections 2024-03-18T01:40:30.315 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:30.315+0000 7f556a800640 1 -- 172.21.15.7:0/3715670595 wait complete. 2024-03-18T01:40:30.611 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:30 smithi007 ceph-mon[31996]: pgmap v487: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 223 MiB/s wr, 929 op/s 2024-03-18T01:40:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:30 smithi203 ceph-mon[36345]: pgmap v487: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 223 MiB/s wr, 929 op/s 2024-03-18T01:40:31.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:30 smithi046 ceph-mon[36108]: pgmap v487: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 223 MiB/s wr, 929 op/s 2024-03-18T01:40:31.343 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T01:40:31.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:40:31.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T01:40:31.684 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 -- 172.21.15.7:0/3692539741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0761a0 msgr2=0x7fc98c0765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 --2- 172.21.15.7:0/3692539741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0761a0 0x7fc98c0765c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc974009900 tx=0x7fc97402bc30 comp rx=0 tx=0).stop 2024-03-18T01:40:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 -- 172.21.15.7:0/3692539741 shutdown_connections 2024-03-18T01:40:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 --2- 172.21.15.7:0/3692539741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc98c077d80 0x7fc98c078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 --2- 172.21.15.7:0/3692539741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98c0773e0 0x7fc98c077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 --2- 172.21.15.7:0/3692539741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0761a0 0x7fc98c0765c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 -- 172.21.15.7:0/3692539741 >> 172.21.15.7:0/3692539741 conn(0x7fc98c0fefa0 msgr2=0x7fc98c1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 -- 172.21.15.7:0/3692539741 shutdown_connections 2024-03-18T01:40:31.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.684+0000 7fc992400640 1 -- 172.21.15.7:0/3692539741 wait complete. 2024-03-18T01:40:31.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.685+0000 7fc992400640 1 Processor -- start 2024-03-18T01:40:31.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.686+0000 7fc992400640 1 -- start start 2024-03-18T01:40:31.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.686+0000 7fc992400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc98c0761a0 0x7fc98c19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.686+0000 7fc992400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0773e0 0x7fc98c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.686+0000 7fc992400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98c077d80 0x7fc98c1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.686+0000 7fc992400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c113aa0 con 0x7fc98c0761a0 2024-03-18T01:40:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.686+0000 7fc992400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c113920 con 0x7fc98c0773e0 2024-03-18T01:40:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.686+0000 7fc992400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c113c20 con 0x7fc98c077d80 2024-03-18T01:40:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.687+0000 7fc98aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0773e0 0x7fc98c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.687+0000 7fc98aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0773e0 0x7fc98c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49490/0 (socket says 172.21.15.7:49490) 2024-03-18T01:40:31.687 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.687+0000 7fc98aa00640 1 -- 172.21.15.7:0/965194214 learned_addr learned my addr 172.21.15.7:0/965194214 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.687+0000 7fc98be00640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98c077d80 0x7fc98c1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.688+0000 7fc98aa00640 1 -- 172.21.15.7:0/965194214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98c077d80 msgr2=0x7fc98c1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.688+0000 7fc98aa00640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98c077d80 0x7fc98c1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.688+0000 7fc98aa00640 1 -- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc98c0761a0 msgr2=0x7fc98c19e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.688+0000 7fc98aa00640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc98c0761a0 0x7fc98c19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.688+0000 7fc98aa00640 1 -- 172.21.15.7:0/965194214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98c1a5d30 con 0x7fc98c0773e0 2024-03-18T01:40:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.688+0000 7fc98aa00640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0773e0 0x7fc98c19e650 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00d950 tx=0x7fc97c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.689+0000 7fc983e00640 1 -- 172.21.15.7:0/965194214 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c017070 con 0x7fc98c0773e0 2024-03-18T01:40:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.689+0000 7fc983e00640 1 -- 172.21.15.7:0/965194214 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc97c009c30 con 0x7fc98c0773e0 2024-03-18T01:40:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.689+0000 7fc983e00640 1 -- 172.21.15.7:0/965194214 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c00f6b0 con 0x7fc98c0773e0 2024-03-18T01:40:31.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.689+0000 7fc992400640 1 -- 172.21.15.7:0/965194214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98c1a5fc0 con 0x7fc98c0773e0 2024-03-18T01:40:31.690 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.689+0000 7fc992400640 1 -- 172.21.15.7:0/965194214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc98c1a6490 con 0x7fc98c0773e0 2024-03-18T01:40:31.699 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.697+0000 7fc983e00640 1 -- 172.21.15.7:0/965194214 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fc97c0109d0 con 0x7fc98c0773e0 2024-03-18T01:40:31.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.702+0000 7fc995174740 1 Processor -- start 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.703+0000 7fc983e00640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc9700a2880 0x7fc9700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.703+0000 7fc995174740 1 -- start start 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.704+0000 7fc983e00640 1 -- 172.21.15.7:0/965194214 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc97c0ce960 con 0x7fc98c0773e0 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.704+0000 7fc995174740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a33364c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.704+0000 7fc995174740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333651e0 0x560a33369ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.704+0000 7fc995174740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3336c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.704+0000 7fc995174740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x560a332b6f80 con 0x560a333647f0 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.704+0000 7fc995174740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560a3325fad0 con 0x560a3336a350 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.704+0000 7fc995174740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560a332b3020 con 0x560a333651e0 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.704+0000 7fc98be00640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc9700a2880 0x7fc9700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.705+0000 7fc98aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333651e0 0x560a33369ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.705+0000 7fc98be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3336c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.705+0000 7fc98aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333651e0 0x560a33369ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58246/0 (socket says 172.21.15.7:58246) 2024-03-18T01:40:31.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.705+0000 7fc98aa00640 1 -- 172.21.15.7:0/1038906680 learned_addr learned my addr 172.21.15.7:0/1038906680 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:31.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.705+0000 7fc98b400640 1 --2- 172.21.15.7:0/1038906680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a33364c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.706+0000 7fc98aa00640 1 -- 172.21.15.7:0/1038906680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 msgr2=0x560a3336c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:31.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.706+0000 7fc98aa00640 1 --2- 172.21.15.7:0/1038906680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3336c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.706+0000 7fc98aa00640 1 -- 172.21.15.7:0/1038906680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 msgr2=0x560a33364c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:31.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.706+0000 7fc98aa00640 1 --2- 172.21.15.7:0/1038906680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a33364c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.706+0000 7fc98aa00640 1 -- 172.21.15.7:0/1038906680 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a332776f0 con 0x560a333651e0 2024-03-18T01:40:31.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.707+0000 7fc98aa00640 1 --2- 172.21.15.7:0/1038906680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333651e0 0x560a33369ce0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc97c09af30 tx=0x7fc97c010d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aacfc5d3b24d2609 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:31.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.709+0000 7fc980c00640 1 -- 172.21.15.7:0/1038906680 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c020040 con 0x560a333651e0 2024-03-18T01:40:31.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.709+0000 7fc980c00640 1 -- 172.21.15.7:0/1038906680 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc97c09f0f0 con 0x560a333651e0 2024-03-18T01:40:31.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.709+0000 7fc980c00640 1 -- 172.21.15.7:0/1038906680 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c09a6e0 con 0x560a333651e0 2024-03-18T01:40:31.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.710+0000 7fc995174740 1 -- 172.21.15.7:0/1038906680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333651e0 msgr2=0x560a33369ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.710+0000 7fc995174740 1 --2- 172.21.15.7:0/1038906680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333651e0 0x560a33369ce0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc97c09af30 tx=0x7fc97c010d70 comp rx=0 tx=0).stop 2024-03-18T01:40:31.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.711+0000 7fc98be00640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc9700a2880 0x7fc9700a4d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc984005fd0 tx=0x7fc984009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:31.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.711+0000 7fc995174740 1 -- 172.21.15.7:0/1038906680 shutdown_connections 2024-03-18T01:40:31.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.711+0000 7fc995174740 1 --2- 172.21.15.7:0/1038906680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3336c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.711+0000 7fc995174740 1 --2- 172.21.15.7:0/1038906680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333651e0 0x560a33369ce0 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc97c09af30 tx=0x7fc97c010d70 comp rx=0 tx=0).stop 2024-03-18T01:40:31.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.711+0000 7fc995174740 1 --2- 172.21.15.7:0/1038906680 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a33364c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.711+0000 7fc995174740 1 -- 172.21.15.7:0/1038906680 >> 172.21.15.7:0/1038906680 conn(0x560a3331b930 msgr2=0x560a3334f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:31.716 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.716+0000 7fc995174740 1 -- 172.21.15.7:0/1038906680 shutdown_connections 2024-03-18T01:40:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.716+0000 7fc995174740 1 -- 172.21.15.7:0/1038906680 wait complete. 2024-03-18T01:40:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.717+0000 7fc995174740 1 -- start start 2024-03-18T01:40:31.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.717+0000 7fc995174740 1 Processor -- start 2024-03-18T01:40:31.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.719+0000 7fc995174740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a333647f0 0x560a333551d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.719+0000 7fc995174740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3336a350 0x560a33355710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.720+0000 7fc995174740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3335cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.720+0000 7fc995174740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x560a33353620 con 0x560a3336a350 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.720+0000 7fc995174740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560a33272f30 con 0x560a333647f0 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.720+0000 7fc995174740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560a333537a0 con 0x560a3335a680 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.720+0000 7fc98be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3335cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.720+0000 7fc98be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3335cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58248/0 (socket says 172.21.15.7:58248) 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.720+0000 7fc98be00640 1 -- 172.21.15.7:0/2492716650 learned_addr learned my addr 172.21.15.7:0/2492716650 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98b400640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a333647f0 0x560a333551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98be00640 1 -- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a333647f0 msgr2=0x560a333551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98aa00640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3336a350 0x560a33355710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc995174740 10 client.0 _ll_register_callbacks cb 0x560a33260bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98be00640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a333647f0 0x560a333551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98be00640 1 -- 172.21.15.7:0/2492716650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3336a350 msgr2=0x560a33355710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98be00640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3336a350 0x560a33355710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98be00640 1 -- 172.21.15.7:0/2492716650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a330937e0 con 0x560a3335a680 2024-03-18T01:40:31.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98b400640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a333647f0 0x560a333551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.721+0000 7fc98aa00640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3336a350 0x560a33355710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:31.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.722+0000 7fc995174740 1 -- 172.21.15.7:0/2492716650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc97c09f0f0 con 0x560a3335a680 2024-03-18T01:40:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.722+0000 7fc98be00640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3335cb10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc984005f40 tx=0x7fc984004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=163fd57618af6e95 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.722+0000 7fc979600640 10 client.26104 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:31.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.723+0000 7fc979600640 1 -- 172.21.15.7:0/2492716650 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984024030 con 0x560a3335a680 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.723+0000 7fc979600640 1 -- 172.21.15.7:0/2492716650 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc98400cbb0 con 0x560a3335a680 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.723+0000 7fc979600640 1 -- 172.21.15.7:0/2492716650 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc98400cdb0 con 0x560a3335a680 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.723+0000 7fc995174740 10 client.26104 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.723+0000 7fc995174740 1 -- 172.21.15.7:0/2492716650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=31}) -- 0x560a330f3f70 con 0x560a3335a680 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.724+0000 7fc979600640 1 -- 172.21.15.7:0/2492716650 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fc98401e040 con 0x560a3335a680 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.724+0000 7fc995174740 10 client.26104 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.724+0000 7fc995174740 10 client.26104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.724+0000 7fc995174740 20 client.26104 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.724+0000 7fc995174740 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560a33361330 0x560a333637d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.724+0000 7fc995174740 4 client.26104 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:40:31.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.724+0000 7fc995174740 1 -- 172.21.15.7:0/2492716650 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a332c44c0 con 0x560a33361330 2024-03-18T01:40:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.725+0000 7fc98b400640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560a33361330 0x560a333637d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.726+0000 7fc98b400640 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560a33361330 0x560a333637d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1caa32ed6142a035 server_cookie=aa8f82a1bd572353 in_seq=0 out_seq=0 2024-03-18T01:40:31.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.726+0000 7fc979600640 10 client.26104 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:31.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.727+0000 7fc979600640 1 -- 172.21.15.7:0/2492716650 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560a332c44c0 con 0x560a33361330 2024-03-18T01:40:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.727+0000 7fc979600640 10 client.26104 handle_command_reply: tid=0 2024-03-18T01:40:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 client.26104 shutdown 2024-03-18T01:40:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 -- 172.21.15.7:0/2492716650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 msgr2=0x560a3335cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3335cb10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc984005f40 tx=0x7fc984004a40 comp rx=0 tx=0).stop 2024-03-18T01:40:31.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 -- 172.21.15.7:0/2492716650 shutdown_connections 2024-03-18T01:40:31.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560a33361330 0x560a333637d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3335cb10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3336a350 0x560a33355710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 --2- 172.21.15.7:0/2492716650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a333647f0 0x560a333551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc995174740 1 -- 172.21.15.7:0/2492716650 >> 172.21.15.7:0/2492716650 conn(0x560a3331b930 msgr2=0x560a3334e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:31.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.728+0000 7fc979600640 0 client.26104 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:31.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.729+0000 7fc995174740 1 -- 172.21.15.7:0/2492716650 shutdown_connections 2024-03-18T01:40:31.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.729+0000 7fc995174740 1 -- 172.21.15.7:0/2492716650 wait complete. 2024-03-18T01:40:31.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.730+0000 7fc995174740 20 client.26104 trim_cache size 0 max 16384 2024-03-18T01:40:31.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.774+0000 7fc995174740 1 Processor -- start 2024-03-18T01:40:31.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.774+0000 7fc995174740 1 -- start start 2024-03-18T01:40:31.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.774+0000 7fc995174740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3335a680 0x560a333544a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.774+0000 7fc995174740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333647f0 0x560a333549e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.774+0000 7fc995174740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3334dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.774+0000 7fc995174740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1a81e0 con 0x560a3335a680 2024-03-18T01:40:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.774+0000 7fc995174740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1a9de0 con 0x560a3336a350 2024-03-18T01:40:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.774+0000 7fc995174740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1a88b0 con 0x560a333647f0 2024-03-18T01:40:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.775+0000 7fc98aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333647f0 0x560a333549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.775+0000 7fc98be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3334dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.775+0000 7fc98be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3334dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49520/0 (socket says 172.21.15.7:49520) 2024-03-18T01:40:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.775+0000 7fc98be00640 1 -- 172.21.15.7:0/4049508550 learned_addr learned my addr 172.21.15.7:0/4049508550 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.776+0000 7fc98be00640 1 -- 172.21.15.7:0/4049508550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333647f0 msgr2=0x560a333549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.776+0000 7fc98be00640 1 --2- 172.21.15.7:0/4049508550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333647f0 0x560a333549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.776+0000 7fc98be00640 1 -- 172.21.15.7:0/4049508550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3335a680 msgr2=0x560a333544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.776+0000 7fc98be00640 1 --2- 172.21.15.7:0/4049508550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3335a680 0x560a333544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.776+0000 7fc98be00640 1 -- 172.21.15.7:0/4049508550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a32f8b170 con 0x560a3336a350 2024-03-18T01:40:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.777+0000 7fc98be00640 1 --2- 172.21.15.7:0/4049508550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3334dac0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc98400c890 tx=0x7fc984020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=794bd547de11437a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:31.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.777+0000 7fc97be00640 1 -- 172.21.15.7:0/4049508550 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984024070 con 0x560a3336a350 2024-03-18T01:40:31.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.778+0000 7fc97be00640 1 -- 172.21.15.7:0/4049508550 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984038070 con 0x560a3336a350 2024-03-18T01:40:31.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.778+0000 7fc97be00640 1 -- 172.21.15.7:0/4049508550 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9840336c0 con 0x560a3336a350 2024-03-18T01:40:31.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.779+0000 7fc995174740 1 -- 172.21.15.7:0/4049508550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 msgr2=0x560a3334dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.779+0000 7fc995174740 1 --2- 172.21.15.7:0/4049508550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3334dac0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc98400c890 tx=0x7fc984020000 comp rx=0 tx=0).stop 2024-03-18T01:40:31.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.780+0000 7fc995174740 1 -- 172.21.15.7:0/4049508550 shutdown_connections 2024-03-18T01:40:31.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.780+0000 7fc995174740 1 --2- 172.21.15.7:0/4049508550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a3334dac0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.780+0000 7fc995174740 1 --2- 172.21.15.7:0/4049508550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a333647f0 0x560a333549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.780+0000 7fc995174740 1 --2- 172.21.15.7:0/4049508550 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a3335a680 0x560a333544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.780+0000 7fc995174740 1 -- 172.21.15.7:0/4049508550 >> 172.21.15.7:0/4049508550 conn(0x560a3331b930 msgr2=0x560a333667c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:31.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.780+0000 7fc995174740 1 -- 172.21.15.7:0/4049508550 shutdown_connections 2024-03-18T01:40:31.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.780+0000 7fc995174740 1 -- 172.21.15.7:0/4049508550 wait complete. 2024-03-18T01:40:31.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.781+0000 7fc995174740 1 -- start start 2024-03-18T01:40:31.782 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.782+0000 7fc995174740 1 Processor -- start 2024-03-18T01:40:31.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.782+0000 7fc995174740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3336dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.782+0000 7fc995174740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a3336e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.782+0000 7fc995174740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a33371200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.782+0000 7fc995174740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1a9650 con 0x560a333647f0 2024-03-18T01:40:31.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.782+0000 7fc995174740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c1a8f80 con 0x560a3336a350 2024-03-18T01:40:31.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.782+0000 7fc995174740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c10b750 con 0x560a3335a680 2024-03-18T01:40:31.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.783+0000 7fc995174740 10 client.0 _ll_register_callbacks cb 0x7fc984012510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:31.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.784+0000 7fc98b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3336dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.784+0000 7fc98aa00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a3336e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.784+0000 7fc98b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3336dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58262/0 (socket says 172.21.15.7:58262) 2024-03-18T01:40:31.784 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.784+0000 7fc98aa00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a3336e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34906/0 (socket says 172.21.15.7:34906) 2024-03-18T01:40:31.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.784+0000 7fc98aa00640 1 -- 172.21.15.7:0/783948723 learned_addr learned my addr 172.21.15.7:0/783948723 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:31.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.785+0000 7fc98b400640 1 -- 172.21.15.7:0/783948723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 msgr2=0x560a33371200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:31.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.785+0000 7fc98b400640 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a33371200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.785+0000 7fc98b400640 1 -- 172.21.15.7:0/783948723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 msgr2=0x560a3336e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:31.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.785+0000 7fc98b400640 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a3336e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.785+0000 7fc98b400640 1 -- 172.21.15.7:0/783948723 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a333717f0 con 0x560a3335a680 2024-03-18T01:40:31.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.786+0000 7fc98b400640 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3336dcf0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc97402c170 tx=0x7fc97402cab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3048b11b27d8016 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:31.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.786+0000 7fc979600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:31.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.787+0000 7fc979600640 1 -- 172.21.15.7:0/783948723 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974049060 con 0x560a3335a680 2024-03-18T01:40:31.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.787+0000 7fc979600640 1 -- 172.21.15.7:0/783948723 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc974004230 con 0x560a3335a680 2024-03-18T01:40:31.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.787+0000 7fc995174740 1 -- 172.21.15.7:0/783948723 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc984038070 con 0x560a3335a680 2024-03-18T01:40:31.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.788+0000 7fc979600640 1 -- 172.21.15.7:0/783948723 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc974004430 con 0x560a3335a680 2024-03-18T01:40:31.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.788+0000 7fc995174740 10 client.26110 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:31.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.788+0000 7fc995174740 1 -- 172.21.15.7:0/783948723 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560a333e1860 con 0x560a3335a680 2024-03-18T01:40:31.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.788+0000 7fc979600640 1 -- 172.21.15.7:0/783948723 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984038070 con 0x560a3335a680 2024-03-18T01:40:31.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.789+0000 7fc979600640 1 -- 172.21.15.7:0/783948723 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fc9740046c0 con 0x560a3335a680 2024-03-18T01:40:31.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.789+0000 7fc995174740 10 client.26110 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:31.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.789+0000 7fc995174740 10 client.26110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:40:31.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.789+0000 7fc995174740 20 client.26110 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:31.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.789+0000 7fc995174740 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560a333e2250 0x560a333e4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:31.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.789+0000 7fc995174740 4 client.26110 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T01:40:31.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.789+0000 7fc995174740 1 -- 172.21.15.7:0/783948723 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560a332be810 con 0x560a333e2250 2024-03-18T01:40:31.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.789+0000 7fc98aa00640 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560a333e2250 0x560a333e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:31.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.790+0000 7fc979600640 10 client.26110 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:31.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.790+0000 7fc98aa00640 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560a333e2250 0x560a333e4670 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ad59d49913d7784 server_cookie=85d0a36571ee80a5 in_seq=0 out_seq=0 2024-03-18T01:40:31.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.793+0000 7fc979600640 1 -- 172.21.15.7:0/783948723 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560a332be810 con 0x560a333e2250 2024-03-18T01:40:31.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.793+0000 7fc979600640 10 client.26110 handle_command_reply: tid=0 2024-03-18T01:40:31.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.793+0000 7fc995174740 1 client.26110 shutdown 2024-03-18T01:40:31.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.794+0000 7fc995174740 1 -- 172.21.15.7:0/783948723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 msgr2=0x560a3336dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.794+0000 7fc995174740 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3336dcf0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc97402c170 tx=0x7fc97402cab0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 1 -- 172.21.15.7:0/783948723 shutdown_connections 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x560a333e2250 0x560a333e4670 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560a3336a350 0x560a33371200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x560a333647f0 0x560a3336e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 1 --2- 172.21.15.7:0/783948723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560a3335a680 0x560a3336dcf0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc979600640 0 client.26110 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 1 -- 172.21.15.7:0/783948723 >> 172.21.15.7:0/783948723 conn(0x560a3331b930 msgr2=0x560a333544a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 1 -- 172.21.15.7:0/783948723 shutdown_connections 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 1 -- 172.21.15.7:0/783948723 wait complete. 2024-03-18T01:40:31.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.795+0000 7fc995174740 20 client.26110 trim_cache size 0 max 16384 2024-03-18T01:40:31.797 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:40:31.797 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-03-18T01:40:31.797 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:40:31.797 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-03-18T01:40:31.797 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:40:31.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.796+0000 7fc992400640 1 -- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc9700a2880 msgr2=0x7fc9700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.796+0000 7fc992400640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc9700a2880 0x7fc9700a4d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc984005fd0 tx=0x7fc984009600 comp rx=0 tx=0).stop 2024-03-18T01:40:31.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.796+0000 7fc992400640 1 -- 172.21.15.7:0/965194214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0773e0 msgr2=0x7fc98c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:31.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.796+0000 7fc992400640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0773e0 0x7fc98c19e650 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00d950 tx=0x7fc97c00de10 comp rx=0 tx=0).stop 2024-03-18T01:40:31.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.797+0000 7fc992400640 1 -- 172.21.15.7:0/965194214 shutdown_connections 2024-03-18T01:40:31.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.797+0000 7fc992400640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98c077d80 0x7fc98c1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.797+0000 7fc992400640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc98c0773e0 0x7fc98c19e650 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.797+0000 7fc992400640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc9700a2880 0x7fc9700a4d40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.797+0000 7fc992400640 1 --2- 172.21.15.7:0/965194214 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc98c0761a0 0x7fc98c19e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:31.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.797+0000 7fc992400640 1 -- 172.21.15.7:0/965194214 >> 172.21.15.7:0/965194214 conn(0x7fc98c0fefa0 msgr2=0x7fc98c1006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:31.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.797+0000 7fc992400640 1 -- 172.21.15.7:0/965194214 shutdown_connections 2024-03-18T01:40:31.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:31.797+0000 7fc992400640 1 -- 172.21.15.7:0/965194214 wait complete. 2024-03-18T01:40:31.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:40:31.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:40:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.148+0000 7f9ac0200640 1 -- 172.21.15.7:0/3118939157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ab8107e20 msgr2=0x7f9ab8108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.148+0000 7f9ac0200640 1 --2- 172.21.15.7:0/3118939157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ab8107e20 0x7f9ab8108240 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009900 tx=0x7f9aac02bc30 comp rx=0 tx=0).stop 2024-03-18T01:40:32.149 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.149+0000 7f9ac0200640 1 -- 172.21.15.7:0/3118939157 shutdown_connections 2024-03-18T01:40:32.150 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.149+0000 7f9ac0200640 1 --2- 172.21.15.7:0/3118939157 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8109930 0x7f9ab8110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.149+0000 7f9ac0200640 1 --2- 172.21.15.7:0/3118939157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab8108f70 0x7f9ab81093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.149+0000 7f9ac0200640 1 --2- 172.21.15.7:0/3118939157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ab8107e20 0x7f9ab8108240 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.149+0000 7f9ac0200640 1 -- 172.21.15.7:0/3118939157 >> 172.21.15.7:0/3118939157 conn(0x7f9ab8076dd0 msgr2=0x7f9ab80791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:32.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.151+0000 7f9ac0200640 1 -- 172.21.15.7:0/3118939157 shutdown_connections 2024-03-18T01:40:32.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.151+0000 7f9ac0200640 1 -- 172.21.15.7:0/3118939157 wait complete. 2024-03-18T01:40:32.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.153+0000 7f9ac0200640 1 Processor -- start 2024-03-18T01:40:32.154 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.154+0000 7f9ac0200640 1 -- start start 2024-03-18T01:40:32.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.155+0000 7f9ac0200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8107e20 0x7f9ab807ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.155 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.156+0000 7f9ac0200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ab8108f70 0x7f9ab807f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.156 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.156+0000 7f9ac0200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab8109930 0x7f9ab81ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.156+0000 7f9ac0200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab8113af0 con 0x7f9ab8107e20 2024-03-18T01:40:32.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.156+0000 7f9ac0200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab8113970 con 0x7f9ab8108f70 2024-03-18T01:40:32.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.156+0000 7f9ac0200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab8113c70 con 0x7f9ab8109930 2024-03-18T01:40:32.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.156+0000 7f9abd800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8107e20 0x7f9ab807ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.157 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.156+0000 7f9abd800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8107e20 0x7f9ab807ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34918/0 (socket says 172.21.15.7:34918) 2024-03-18T01:40:32.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.156+0000 7f9abd800640 1 -- 172.21.15.7:0/735412370 learned_addr learned my addr 172.21.15.7:0/735412370 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:32.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.157+0000 7f9abe200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab8109930 0x7f9ab81ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.157+0000 7f9abce00640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ab8108f70 0x7f9ab807f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.157+0000 7f9abe200640 1 -- 172.21.15.7:0/735412370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ab8108f70 msgr2=0x7f9ab807f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.157+0000 7f9abe200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ab8108f70 0x7f9ab807f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.157+0000 7f9abe200640 1 -- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8107e20 msgr2=0x7f9ab807ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.158 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.157+0000 7f9abe200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8107e20 0x7f9ab807ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.157+0000 7f9abe200640 1 -- 172.21.15.7:0/735412370 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab81ae740 con 0x7f9ab8109930 2024-03-18T01:40:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.158+0000 7f9abe200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab8109930 0x7f9ab81ae200 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000e940 tx=0x7f9ab000ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.159+0000 7f9ab6000640 1 -- 172.21.15.7:0/735412370 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0004d20 con 0x7f9ab8109930 2024-03-18T01:40:32.159 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.159+0000 7f9ab6000640 1 -- 172.21.15.7:0/735412370 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ab0005020 con 0x7f9ab8109930 2024-03-18T01:40:32.160 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.159+0000 7f9ab6000640 1 -- 172.21.15.7:0/735412370 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0004d20 con 0x7f9ab8109930 2024-03-18T01:40:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.160+0000 7f9ac0200640 1 -- 172.21.15.7:0/735412370 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab81ae910 con 0x7f9ab8109930 2024-03-18T01:40:32.161 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.161+0000 7f9ac0200640 1 -- 172.21.15.7:0/735412370 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab81aed80 con 0x7f9ab8109930 2024-03-18T01:40:32.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.170+0000 7f9ab6000640 1 -- 172.21.15.7:0/735412370 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9ab000cb20 con 0x7f9ab8109930 2024-03-18T01:40:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.173+0000 7f9ab6000640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9aa00a2b00 0x7f9aa00a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.174+0000 7f9ac2bfd740 1 Processor -- start 2024-03-18T01:40:32.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.174+0000 7f9ac2bfd740 1 -- start start 2024-03-18T01:40:32.174 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.174+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae5802c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.175+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5807d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.175+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae580a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.175+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x563ae57671f0 con 0x563ae5808370 2024-03-18T01:40:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.175+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563ae570fa80 con 0x563ae5802810 2024-03-18T01:40:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.175+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563ae5763290 con 0x563ae5803200 2024-03-18T01:40:32.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.175+0000 7f9abce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5807d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.176+0000 7f9abce00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5807d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58288/0 (socket says 172.21.15.7:58288) 2024-03-18T01:40:32.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.176+0000 7f9abce00640 1 -- 172.21.15.7:0/3356126361 learned_addr learned my addr 172.21.15.7:0/3356126361 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:32.176 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.176+0000 7f9abd800640 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae5802c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.176+0000 7f9abd800640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9aa00a2b00 0x7f9aa00a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.177+0000 7f9abe200640 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae580a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.177+0000 7f9abce00640 1 -- 172.21.15.7:0/3356126361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 msgr2=0x563ae5802c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:32.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.177+0000 7f9abce00640 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae5802c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.177+0000 7f9abce00640 1 -- 172.21.15.7:0/3356126361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 msgr2=0x563ae580a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:32.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.177+0000 7f9abce00640 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae580a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.177+0000 7f9abce00640 1 -- 172.21.15.7:0/3356126361 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ae55457d0 con 0x563ae5803200 2024-03-18T01:40:32.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.177+0000 7f9abe200640 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae580a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:40:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.177+0000 7f9ab6000640 1 -- 172.21.15.7:0/735412370 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9ab00ce030 con 0x7f9ab8109930 2024-03-18T01:40:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.179+0000 7f9abce00640 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5807d00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009c00 tx=0x7f9aa4010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9be998199a85f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.179+0000 7f9abd800640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9aa00a2b00 0x7f9aa00a4fc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9aac02c8e0 tx=0x7f9aac02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:32.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.180+0000 7f9aaaa00640 1 -- 172.21.15.7:0/3356126361 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4004030 con 0x563ae5803200 2024-03-18T01:40:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.185+0000 7f9aaaa00640 1 -- 172.21.15.7:0/3356126361 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa40041d0 con 0x563ae5803200 2024-03-18T01:40:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.185+0000 7f9aaaa00640 1 -- 172.21.15.7:0/3356126361 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa40126b0 con 0x563ae5803200 2024-03-18T01:40:32.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.187+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/3356126361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 msgr2=0x563ae5807d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.188+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5807d00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009c00 tx=0x7f9aa4010740 comp rx=0 tx=0).stop 2024-03-18T01:40:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.189+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/3356126361 shutdown_connections 2024-03-18T01:40:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.189+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae580a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.189+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5807d00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.189+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/3356126361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae5802c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.189+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/3356126361 >> 172.21.15.7:0/3356126361 conn(0x563ae57fdd80 msgr2=0x563ae57fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:32.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.190+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/3356126361 shutdown_connections 2024-03-18T01:40:32.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.190+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/3356126361 wait complete. 2024-03-18T01:40:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.191+0000 7f9ac2bfd740 1 -- start start 2024-03-18T01:40:32.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.191+0000 7f9ac2bfd740 1 Processor -- start 2024-03-18T01:40:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.194+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae580cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.194+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5803200 0x563ae580d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.194+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5808370 0x563ae5814300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.194+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x563ae580b320 con 0x563ae5803200 2024-03-18T01:40:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.194+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563ae5722ed0 con 0x563ae5802810 2024-03-18T01:40:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.194+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563ae580b4a0 con 0x563ae5808370 2024-03-18T01:40:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.194+0000 7f9abe200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5808370 0x563ae5814300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.194+0000 7f9abd800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae580cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.195+0000 7f9abce00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5803200 0x563ae580d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.195+0000 7f9ac2bfd740 10 client.0 _ll_register_callbacks cb 0x563ae575e220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.195+0000 7f9abce00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5803200 0x563ae580d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:34928/0 (socket says 172.21.15.7:34928) 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.195+0000 7f9abce00640 1 -- 172.21.15.7:0/2140116857 learned_addr learned my addr 172.21.15.7:0/2140116857 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.196+0000 7f9abe200640 1 -- 172.21.15.7:0/2140116857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 msgr2=0x563ae580cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.196+0000 7f9abe200640 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae580cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.196+0000 7f9abe200640 1 -- 172.21.15.7:0/2140116857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5803200 msgr2=0x563ae580d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.196+0000 7f9abe200640 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5803200 0x563ae580d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.196+0000 7f9abe200640 1 -- 172.21.15.7:0/2140116857 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ae547b3f0 con 0x563ae5808370 2024-03-18T01:40:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.196+0000 7f9abce00640 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5803200 0x563ae580d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:40:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.198+0000 7f9abe200640 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5808370 0x563ae5814300 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9ab009a5a0 tx=0x7f9ab009a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8966c6bf4f9f976 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.198+0000 7f9a9f400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:40:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.198+0000 7f9a9f400640 1 -- 172.21.15.7:0/2140116857 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab009a610 con 0x563ae5808370 2024-03-18T01:40:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.198+0000 7f9a9f400640 1 -- 172.21.15.7:0/2140116857 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9ab00051d0 con 0x563ae5808370 2024-03-18T01:40:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.199+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2140116857 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aa40041d0 con 0x563ae5808370 2024-03-18T01:40:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.199+0000 7f9a9f400640 1 -- 172.21.15.7:0/2140116857 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9ab009e760 con 0x563ae5808370 2024-03-18T01:40:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.200+0000 7f9ac2bfd740 10 client.26122 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.200+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2140116857 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563ae543b0f0 con 0x563ae5808370 2024-03-18T01:40:32.200 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.200+0000 7f9a9f400640 1 -- 172.21.15.7:0/2140116857 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa40041d0 con 0x563ae5808370 2024-03-18T01:40:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.202+0000 7f9a9f400640 1 -- 172.21.15.7:0/2140116857 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f9ab000f040 con 0x563ae5808370 2024-03-18T01:40:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.202+0000 7f9ac2bfd740 10 client.26122 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:32.202 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.202+0000 7f9ac2bfd740 10 client.26122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:40:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.202+0000 7f9ac2bfd740 20 client.26122 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.202+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563ae5818c90 0x563ae581b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.202+0000 7f9ac2bfd740 4 client.26122 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:40:32.203 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.202+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2140116857 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ae577abe0 con 0x563ae5818c90 2024-03-18T01:40:32.204 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.204+0000 7f9abd800640 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563ae5818c90 0x563ae581b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.206+0000 7f9abd800640 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563ae5818c90 0x563ae581b0d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e4503a2f0d1d398 server_cookie=b26dad89b38592c in_seq=0 out_seq=0 2024-03-18T01:40:32.206 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.207+0000 7f9a9f400640 10 client.26122 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:32.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.215+0000 7f9a9f400640 1 -- 172.21.15.7:0/2140116857 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ae577abe0 con 0x563ae5818c90 2024-03-18T01:40:32.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.215+0000 7f9a9f400640 10 client.26122 handle_command_reply: tid=0 2024-03-18T01:40:32.215 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.215+0000 7f9ac2bfd740 1 client.26122 shutdown 2024-03-18T01:40:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2140116857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5808370 msgr2=0x563ae5814300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5808370 0x563ae5814300 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9ab009a5a0 tx=0x7f9ab009a150 comp rx=0 tx=0).stop 2024-03-18T01:40:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2140116857 shutdown_connections 2024-03-18T01:40:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563ae5818c90 0x563ae581b0d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5808370 0x563ae5814300 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5803200 0x563ae580d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2140116857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae580cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9a9f400640 0 client.26122 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.216+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2140116857 >> 172.21.15.7:0/2140116857 conn(0x563ae57fdd80 msgr2=0x563ae57fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.217+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2140116857 shutdown_connections 2024-03-18T01:40:32.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.217+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2140116857 wait complete. 2024-03-18T01:40:32.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.217+0000 7f9ac2bfd740 20 client.26122 trim_cache size 0 max 16384 2024-03-18T01:40:32.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.274+0000 7f9ac2bfd740 1 Processor -- start 2024-03-18T01:40:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.277+0000 7f9ac2bfd740 1 -- start start 2024-03-18T01:40:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.277+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5802810 0x563ae58047a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.277+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5803200 0x563ae58142c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.277+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae5804ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.277+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab81b9190 con 0x563ae5808370 2024-03-18T01:40:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.277+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab81bad90 con 0x563ae5803200 2024-03-18T01:40:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.277+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab81b9860 con 0x563ae5802810 2024-03-18T01:40:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5803200 0x563ae58142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5803200 0x563ae58142c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49560/0 (socket says 172.21.15.7:49560) 2024-03-18T01:40:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abce00640 1 -- 172.21.15.7:0/4154150403 learned_addr learned my addr 172.21.15.7:0/4154150403 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abd800640 1 --2- 172.21.15.7:0/4154150403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5802810 0x563ae58047a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abce00640 1 -- 172.21.15.7:0/4154150403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5802810 msgr2=0x563ae58047a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abce00640 1 --2- 172.21.15.7:0/4154150403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5802810 0x563ae58047a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abce00640 1 -- 172.21.15.7:0/4154150403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 msgr2=0x563ae5804ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abce00640 1 --2- 172.21.15.7:0/4154150403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae5804ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.278+0000 7f9abce00640 1 -- 172.21.15.7:0/4154150403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ae57fe7d0 con 0x563ae5803200 2024-03-18T01:40:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.279+0000 7f9abce00640 1 --2- 172.21.15.7:0/4154150403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5803200 0x563ae58142c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4019000 tx=0x7f9aa4009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf294f54e58c3ed8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.280+0000 7f9aaa000640 1 -- 172.21.15.7:0/4154150403 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4028070 con 0x563ae5803200 2024-03-18T01:40:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.280+0000 7f9aaa000640 1 -- 172.21.15.7:0/4154150403 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa4004520 con 0x563ae5803200 2024-03-18T01:40:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.280+0000 7f9aaa000640 1 -- 172.21.15.7:0/4154150403 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4003e60 con 0x563ae5803200 2024-03-18T01:40:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.283+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/4154150403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5803200 msgr2=0x563ae58142c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.283+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/4154150403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5803200 0x563ae58142c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4019000 tx=0x7f9aa4009620 comp rx=0 tx=0).stop 2024-03-18T01:40:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.284+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/4154150403 shutdown_connections 2024-03-18T01:40:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.284+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/4154150403 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae5804ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.284+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/4154150403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5803200 0x563ae58142c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.284+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/4154150403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5802810 0x563ae58047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.284+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/4154150403 >> 172.21.15.7:0/4154150403 conn(0x563ae57fdd80 msgr2=0x563ae580cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.284+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/4154150403 shutdown_connections 2024-03-18T01:40:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.284+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/4154150403 wait complete. 2024-03-18T01:40:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.285+0000 7f9ac2bfd740 1 -- start start 2024-03-18T01:40:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.285+0000 7f9ac2bfd740 1 Processor -- start 2024-03-18T01:40:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae58119b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5811ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9ac2bfd740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae5812f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab81ba600 con 0x563ae5808370 2024-03-18T01:40:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab81b9f30 con 0x563ae5802810 2024-03-18T01:40:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9ac2bfd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab81b7170 con 0x563ae5803200 2024-03-18T01:40:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9ac2bfd740 10 client.0 _ll_register_callbacks cb 0x7f9ab000ef30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:40:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9abd800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae58119b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9abd800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae58119b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:49572/0 (socket says 172.21.15.7:49572) 2024-03-18T01:40:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.286+0000 7f9abd800640 1 -- 172.21.15.7:0/2944978100 learned_addr learned my addr 172.21.15.7:0/2944978100 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:40:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.287+0000 7f9abce00640 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5811ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.287+0000 7f9abe200640 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae5812f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.287+0000 7f9abd800640 1 -- 172.21.15.7:0/2944978100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 msgr2=0x563ae5811ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.287+0000 7f9abd800640 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5811ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.287+0000 7f9abd800640 1 -- 172.21.15.7:0/2944978100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 msgr2=0x563ae5812f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:40:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.287+0000 7f9abd800640 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae5812f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.287+0000 7f9abd800640 1 -- 172.21.15.7:0/2944978100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ae58134f0 con 0x563ae5802810 2024-03-18T01:40:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.288+0000 7f9abd800640 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae58119b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9aac02c240 tx=0x7f9aac02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee6bb00d863b9319 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:40:32.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.288+0000 7f9a9f400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:40:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.290+0000 7f9a9f400640 1 -- 172.21.15.7:0/2944978100 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac047040 con 0x563ae5802810 2024-03-18T01:40:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.290+0000 7f9a9f400640 1 -- 172.21.15.7:0/2944978100 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9aac004ef0 con 0x563ae5802810 2024-03-18T01:40:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.290+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2944978100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9aa4004520 con 0x563ae5802810 2024-03-18T01:40:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.292+0000 7f9a9f400640 1 -- 172.21.15.7:0/2944978100 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9aac068050 con 0x563ae5802810 2024-03-18T01:40:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.292+0000 7f9ac2bfd740 10 client.26159 fetch_fsmap learned FSMap version 31 2024-03-18T01:40:32.292 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.292+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2944978100 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563ae587f070 con 0x563ae5802810 2024-03-18T01:40:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.293+0000 7f9a9f400640 1 -- 172.21.15.7:0/2944978100 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9aa4004520 con 0x563ae5802810 2024-03-18T01:40:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.293+0000 7f9a9f400640 1 -- 172.21.15.7:0/2944978100 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f9aac003aa0 con 0x563ae5802810 2024-03-18T01:40:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.293+0000 7f9ac2bfd740 10 client.26159 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:40:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.293+0000 7f9ac2bfd740 10 client.26159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:40:32.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.293+0000 7f9ac2bfd740 20 client.26159 populate_metadata read hostname 'smithi007' 2024-03-18T01:40:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.293+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563ae587fab0 0x563ae5881ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:40:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.293+0000 7f9ac2bfd740 4 client.26159 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:40:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.293+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2944978100 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ae5774730 con 0x563ae587fab0 2024-03-18T01:40:32.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.294+0000 7f9abce00640 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563ae587fab0 0x563ae5881ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:40:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.295+0000 7f9abce00640 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563ae587fab0 0x563ae5881ed0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=746459247ea131af server_cookie=688572ac80431ed2 in_seq=0 out_seq=0 2024-03-18T01:40:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.295+0000 7f9a9f400640 10 client.26159 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.295+0000 7f9a9f400640 1 -- 172.21.15.7:0/2944978100 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x563ae5774730 con 0x563ae587fab0 2024-03-18T01:40:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.295+0000 7f9a9f400640 10 client.26159 handle_command_reply: tid=0 2024-03-18T01:40:32.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.295+0000 7f9ac2bfd740 1 client.26159 shutdown 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2944978100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 msgr2=0x563ae58119b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae58119b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9aac02c240 tx=0x7f9aac02bc30 comp rx=0 tx=0).stop 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2944978100 shutdown_connections 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563ae587fab0 0x563ae5881ed0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563ae5808370 0x563ae5812f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563ae5803200 0x563ae5811ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 --2- 172.21.15.7:0/2944978100 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563ae5802810 0x563ae58119b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9a9f400640 0 client.26159 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:40:32.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2944978100 >> 172.21.15.7:0/2944978100 conn(0x563ae57fdd80 msgr2=0x563ae580cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:32.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2944978100 shutdown_connections 2024-03-18T01:40:32.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.297+0000 7f9ac2bfd740 1 -- 172.21.15.7:0/2944978100 wait complete. 2024-03-18T01:40:32.298 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.298+0000 7f9ac2bfd740 20 client.26159 trim_cache size 0 max 16384 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:40:32.299 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:40:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 -- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9aa00a2b00 msgr2=0x7f9aa00a4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9aa00a2b00 0x7f9aa00a4fc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9aac02c8e0 tx=0x7f9aac02df90 comp rx=0 tx=0).stop 2024-03-18T01:40:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 -- 172.21.15.7:0/735412370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab8109930 msgr2=0x7f9ab81ae200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:40:32.300 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab8109930 0x7f9ab81ae200 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000e940 tx=0x7f9ab000ee00 comp rx=0 tx=0).stop 2024-03-18T01:40:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 -- 172.21.15.7:0/735412370 shutdown_connections 2024-03-18T01:40:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab8109930 0x7f9ab81ae200 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ab8108f70 0x7f9ab807f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9aa00a2b00 0x7f9aa00a4fc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 --2- 172.21.15.7:0/735412370 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9ab8107e20 0x7f9ab807ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:40:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.300+0000 7f9ac0200640 1 -- 172.21.15.7:0/735412370 >> 172.21.15.7:0/735412370 conn(0x7f9ab8076dd0 msgr2=0x7f9ab810c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:40:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.301+0000 7f9ac0200640 1 -- 172.21.15.7:0/735412370 shutdown_connections 2024-03-18T01:40:32.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:40:32.301+0000 7f9ac0200640 1 -- 172.21.15.7:0/735412370 wait complete. 2024-03-18T01:40:32.337 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:40:32.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:32 smithi203 ceph-mon[36345]: pgmap v488: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 189 MiB/s wr, 2.09k op/s 2024-03-18T01:40:32.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:32 smithi203 ceph-mon[36345]: scrub queued for path: / 2024-03-18T01:40:32.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:32 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:40:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:32 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:40:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:32 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:40:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:32 smithi046 ceph-mon[36108]: pgmap v488: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 189 MiB/s wr, 2.09k op/s 2024-03-18T01:40:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:32 smithi046 ceph-mon[36108]: scrub queued for path: / 2024-03-18T01:40:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:32 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:40:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:32 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:40:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:32 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:40:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:32 smithi007 ceph-mon[31996]: pgmap v488: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 189 MiB/s wr, 2.09k op/s 2024-03-18T01:40:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:32 smithi007 ceph-mon[31996]: scrub queued for path: / 2024-03-18T01:40:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:32 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:40:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:32 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:40:33.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:32 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:40:33.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:33 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:40:33.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:33 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:40:33.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:33 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:40:33.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:33 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:40:33.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:33 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:40:34.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:33 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:40:34.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:33 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:40:34.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:33 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:40:34.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:33 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:40:34.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:33 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:40:34.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:33 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:40:34.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:33 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:40:34.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:33 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:40:34.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:33 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:40:34.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:33 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:40:34.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:34 smithi203 ceph-mon[36345]: pgmap v489: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 128 MiB/s wr, 2.63k op/s 2024-03-18T01:40:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:34 smithi046 ceph-mon[36108]: pgmap v489: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 128 MiB/s wr, 2.63k op/s 2024-03-18T01:40:35.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:34 smithi007 ceph-mon[31996]: pgmap v489: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 128 MiB/s wr, 2.63k op/s 2024-03-18T01:40:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:36 smithi046 ceph-mon[36108]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 109 MiB/s wr, 5.05k op/s 2024-03-18T01:40:37.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:36 smithi007 ceph-mon[31996]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 109 MiB/s wr, 5.05k op/s 2024-03-18T01:40:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:36 smithi203 ceph-mon[36345]: pgmap v490: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 109 MiB/s wr, 5.05k op/s 2024-03-18T01:40:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:38 smithi046 ceph-mon[36108]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 64 MiB/s wr, 5.69k op/s 2024-03-18T01:40:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:38 smithi007 ceph-mon[31996]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 64 MiB/s wr, 5.69k op/s 2024-03-18T01:40:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:38 smithi203 ceph-mon[36345]: pgmap v491: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 64 MiB/s wr, 5.69k op/s 2024-03-18T01:40:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:40 smithi046 ceph-mon[36108]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 55 MiB/s wr, 7.47k op/s 2024-03-18T01:40:41.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:40 smithi007 ceph-mon[31996]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 55 MiB/s wr, 7.47k op/s 2024-03-18T01:40:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:40 smithi203 ceph-mon[36345]: pgmap v492: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 55 MiB/s wr, 7.47k op/s 2024-03-18T01:40:42.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:42 smithi203 ceph-mon[36345]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 54 MiB/s wr, 8.57k op/s 2024-03-18T01:40:43.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:42 smithi046 ceph-mon[36108]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 54 MiB/s wr, 8.57k op/s 2024-03-18T01:40:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:42 smithi007 ceph-mon[31996]: pgmap v493: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 54 MiB/s wr, 8.57k op/s 2024-03-18T01:40:45.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:44 smithi046 ceph-mon[36108]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 53 MiB/s wr, 8.15k op/s 2024-03-18T01:40:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:44 smithi007 ceph-mon[31996]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 53 MiB/s wr, 8.15k op/s 2024-03-18T01:40:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:44 smithi203 ceph-mon[36345]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 53 MiB/s wr, 8.15k op/s 2024-03-18T01:40:47.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:46 smithi046 ceph-mon[36108]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 62 MiB/s wr, 9.47k op/s 2024-03-18T01:40:47.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:46 smithi007 ceph-mon[31996]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 62 MiB/s wr, 9.47k op/s 2024-03-18T01:40:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:46 smithi203 ceph-mon[36345]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 62 MiB/s wr, 9.47k op/s 2024-03-18T01:40:49.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:48 smithi046 ceph-mon[36108]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 55 MiB/s wr, 8.17k op/s 2024-03-18T01:40:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:48 smithi007 ceph-mon[31996]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 55 MiB/s wr, 8.17k op/s 2024-03-18T01:40:49.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:48 smithi203 ceph-mon[36345]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 55 MiB/s wr, 8.17k op/s 2024-03-18T01:40:51.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:50 smithi046 ceph-mon[36108]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 59 MiB/s wr, 8.84k op/s 2024-03-18T01:40:51.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:50 smithi007 ceph-mon[31996]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 59 MiB/s wr, 8.84k op/s 2024-03-18T01:40:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:50 smithi203 ceph-mon[36345]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 59 MiB/s wr, 8.84k op/s 2024-03-18T01:40:53.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:52 smithi046 ceph-mon[36108]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 58 MiB/s wr, 8.76k op/s 2024-03-18T01:40:53.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:52 smithi007 ceph-mon[31996]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 58 MiB/s wr, 8.76k op/s 2024-03-18T01:40:53.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:52 smithi203 ceph-mon[36345]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 58 MiB/s wr, 8.76k op/s 2024-03-18T01:40:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:54 smithi046 ceph-mon[36108]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 48 MiB/s wr, 7.21k op/s 2024-03-18T01:40:55.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:54 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:40:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:54 smithi007 ceph-mon[31996]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 48 MiB/s wr, 7.21k op/s 2024-03-18T01:40:55.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:54 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:40:55.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:54 smithi203 ceph-mon[36345]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 48 MiB/s wr, 7.21k op/s 2024-03-18T01:40:55.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:54 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:40:56.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:40:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:40:56.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:55 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:40:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:40:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:40:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:55 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:40:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:40:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:40:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:55 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:40:57.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:56 smithi007 ceph-mon[31996]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 58 MiB/s wr, 8.49k op/s 2024-03-18T01:40:57.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:56 smithi203 ceph-mon[36345]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 58 MiB/s wr, 8.49k op/s 2024-03-18T01:40:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:56 smithi046 ceph-mon[36108]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 58 MiB/s wr, 8.49k op/s 2024-03-18T01:40:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:40:58 smithi203 ceph-mon[36345]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 52 MiB/s wr, 7.52k op/s 2024-03-18T01:40:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:40:58 smithi046 ceph-mon[36108]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 52 MiB/s wr, 7.52k op/s 2024-03-18T01:40:59.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:40:58 smithi007 ceph-mon[31996]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 52 MiB/s wr, 7.52k op/s 2024-03-18T01:41:01.104 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:00 smithi007 ceph-mon[31996]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 MiB/s wr, 7.87k op/s 2024-03-18T01:41:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:00 smithi203 ceph-mon[36345]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 MiB/s wr, 7.87k op/s 2024-03-18T01:41:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:00 smithi046 ceph-mon[36108]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 MiB/s wr, 7.87k op/s 2024-03-18T01:41:02.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:41:02.338 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:41:02.704 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.703+0000 7f83ca000640 1 -- 172.21.15.7:0/2766770900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c4072b50 msgr2=0x7f83c4072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.703+0000 7f83ca000640 1 --2- 172.21.15.7:0/2766770900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c4072b50 0x7f83c4072f70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009860 tx=0x7f83b802bb90 comp rx=0 tx=0).stop 2024-03-18T01:41:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.703+0000 7f83ca000640 1 -- 172.21.15.7:0/2766770900 shutdown_connections 2024-03-18T01:41:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.703+0000 7f83ca000640 1 --2- 172.21.15.7:0/2766770900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c41097c0 0x7f83c410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.703+0000 7f83ca000640 1 --2- 172.21.15.7:0/2766770900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83c4108e00 0x7f83c4109280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.703+0000 7f83ca000640 1 --2- 172.21.15.7:0/2766770900 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c4072b50 0x7f83c4072f70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.703+0000 7f83ca000640 1 -- 172.21.15.7:0/2766770900 >> 172.21.15.7:0/2766770900 conn(0x7f83c406e620 msgr2=0x7f83c4070a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.704+0000 7f83ca000640 1 -- 172.21.15.7:0/2766770900 shutdown_connections 2024-03-18T01:41:02.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.705+0000 7f83ca000640 1 -- 172.21.15.7:0/2766770900 wait complete. 2024-03-18T01:41:02.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.706+0000 7f83ca000640 1 Processor -- start 2024-03-18T01:41:02.710 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.709+0000 7f83ca000640 1 -- start start 2024-03-18T01:41:02.711 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.710+0000 7f83ca000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c4108e00 0x7f83c41a6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.710+0000 7f83ca000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c41097c0 0x7f83c41a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.711+0000 7f83ca000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83c41abfd0 0x7f83c41ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.711+0000 7f83ca000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f83c411c170 con 0x7f83c41097c0 2024-03-18T01:41:02.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.711+0000 7f83ca000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f83c411bff0 con 0x7f83c4108e00 2024-03-18T01:41:02.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.711+0000 7f83ca000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83c411c2f0 con 0x7f83c41abfd0 2024-03-18T01:41:02.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.711+0000 7f83c9600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83c41abfd0 0x7f83c41ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.711+0000 7f83c3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c41097c0 0x7f83c41a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.711+0000 7f83c9600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83c41abfd0 0x7f83c41ae3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44510/0 (socket says 172.21.15.7:44510) 2024-03-18T01:41:02.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.711+0000 7f83c9600640 1 -- 172.21.15.7:0/2196196123 learned_addr learned my addr 172.21.15.7:0/2196196123 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:02.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.712+0000 7f83c9600640 1 -- 172.21.15.7:0/2196196123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c4108e00 msgr2=0x7f83c41a6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.712+0000 7f83c9600640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c4108e00 0x7f83c41a6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.712+0000 7f83c9600640 1 -- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c41097c0 msgr2=0x7f83c41a6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.712+0000 7f83c9600640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c41097c0 0x7f83c41a6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.712+0000 7f83c9600640 1 -- 172.21.15.7:0/2196196123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c41aea30 con 0x7f83c41abfd0 2024-03-18T01:41:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.712+0000 7f83c9600640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83c41abfd0 0x7f83c41ae3c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009fd0 tx=0x7f83bc00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.713+0000 7f83c1600640 1 -- 172.21.15.7:0/2196196123 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc004b20 con 0x7f83c41abfd0 2024-03-18T01:41:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.713+0000 7f83c1600640 1 -- 172.21.15.7:0/2196196123 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83bc003e50 con 0x7f83c41abfd0 2024-03-18T01:41:02.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.713+0000 7f83ca000640 1 -- 172.21.15.7:0/2196196123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c41aed20 con 0x7f83c41abfd0 2024-03-18T01:41:02.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.713+0000 7f83c1600640 1 -- 172.21.15.7:0/2196196123 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc00b630 con 0x7f83c41abfd0 2024-03-18T01:41:02.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.713+0000 7f83ca000640 1 -- 172.21.15.7:0/2196196123 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c41af290 con 0x7f83c41abfd0 2024-03-18T01:41:02.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.718+0000 7f83ce366740 1 Processor -- start 2024-03-18T01:41:02.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.718+0000 7f83ce366740 1 -- start start 2024-03-18T01:41:02.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.719+0000 7f83ce366740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c513ffe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.719+0000 7f83ce366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51400400 0x557c51404f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.719+0000 7f83ce366740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51407a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.719+0000 7f83ce366740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557c513641f0 con 0x557c51405570 2024-03-18T01:41:02.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.719+0000 7f83ce366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c5130ca80 con 0x557c51400400 2024-03-18T01:41:02.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.719+0000 7f83ce366740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557c51360290 con 0x557c513ffa10 2024-03-18T01:41:02.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.719+0000 7f83c3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51407a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.719+0000 7f83c3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c513ffe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.720+0000 7f83c3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c513ffe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44526/0 (socket says 172.21.15.7:44526) 2024-03-18T01:41:02.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.720+0000 7f83c3e00640 1 -- 172.21.15.7:0/585960527 learned_addr learned my addr 172.21.15.7:0/585960527 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:02.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.721+0000 7f83c8c00640 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51400400 0x557c51404f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.721+0000 7f83c3e00640 1 -- 172.21.15.7:0/585960527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 msgr2=0x557c513ffe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:02.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.721+0000 7f83c3e00640 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c513ffe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.721+0000 7f83c3e00640 1 -- 172.21.15.7:0/585960527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51400400 msgr2=0x557c51404f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.721+0000 7f83c3e00640 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51400400 0x557c51404f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.721+0000 7f83c3e00640 1 -- 172.21.15.7:0/585960527 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c511427d0 con 0x557c51405570 2024-03-18T01:41:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.721+0000 7f83c8c00640 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51400400 0x557c51404f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:41:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.721+0000 7f83c3e00640 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51407a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f83b000bed0 tx=0x7f83b0006bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fdd9ec35e4eb3d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.722+0000 7f83b5c00640 1 -- 172.21.15.7:0/585960527 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b001d070 con 0x557c51405570 2024-03-18T01:41:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.723+0000 7f83b5c00640 1 -- 172.21.15.7:0/585960527 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83b0018b50 con 0x557c51405570 2024-03-18T01:41:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.723+0000 7f83b5c00640 1 -- 172.21.15.7:0/585960527 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b000f040 con 0x557c51405570 2024-03-18T01:41:02.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.723+0000 7f83c1600640 1 -- 172.21.15.7:0/2196196123 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f83bc01d020 con 0x7f83c41abfd0 2024-03-18T01:41:02.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.724+0000 7f83ce366740 1 -- 172.21.15.7:0/585960527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 msgr2=0x557c51407a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.724+0000 7f83ce366740 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51407a00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f83b000bed0 tx=0x7f83b0006bc0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.727+0000 7f83ce366740 1 -- 172.21.15.7:0/585960527 shutdown_connections 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.727+0000 7f83ce366740 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51407a00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.727+0000 7f83ce366740 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51400400 0x557c51404f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.727+0000 7f83ce366740 1 --2- 172.21.15.7:0/585960527 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c513ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.727+0000 7f83ce366740 1 -- 172.21.15.7:0/585960527 >> 172.21.15.7:0/585960527 conn(0x557c513faf80 msgr2=0x557c513fb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.727+0000 7f83ce366740 1 -- 172.21.15.7:0/585960527 shutdown_connections 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.727+0000 7f83c1600640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f83ac0a29b0 0x7f83ac0a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.727+0000 7f83ce366740 1 -- 172.21.15.7:0/585960527 wait complete. 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.728+0000 7f83c1600640 1 -- 172.21.15.7:0/2196196123 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f83bc0ced60 con 0x7f83c41abfd0 2024-03-18T01:41:02.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.728+0000 7f83c8c00640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f83ac0a29b0 0x7f83ac0a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.729+0000 7f83ce366740 1 -- start start 2024-03-18T01:41:02.730 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.729+0000 7f83ce366740 1 Processor -- start 2024-03-18T01:41:02.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.731+0000 7f83c8c00640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f83ac0a29b0 0x7f83ac0a4e70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x557c51400800 tx=0x7f83b8009490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:02.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.731+0000 7f83ce366740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c51409f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.731+0000 7f83ce366740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51400400 0x557c5140a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.731+0000 7f83ce366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51405570 0x557c51411510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.731+0000 7f83c8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c51409f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.731+0000 7f83c8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c51409f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44538/0 (socket says 172.21.15.7:44538) 2024-03-18T01:41:02.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.732+0000 7f83c8c00640 1 -- 172.21.15.7:0/994986375 learned_addr learned my addr 172.21.15.7:0/994986375 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:02.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.732+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557c51408520 con 0x557c51400400 2024-03-18T01:41:02.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.732+0000 7f83c9600640 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51405570 0x557c51411510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.732+0000 7f83c3e00640 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51400400 0x557c5140a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.733+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557c5131fed0 con 0x557c51405570 2024-03-18T01:41:02.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.733+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557c514086a0 con 0x557c513ffa10 2024-03-18T01:41:02.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.733+0000 7f83ce366740 10 client.0 _ll_register_callbacks cb 0x557c5135b220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:41:02.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.733+0000 7f83c9600640 1 -- 172.21.15.7:0/994986375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 msgr2=0x557c51409f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:02.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.733+0000 7f83c9600640 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c51409f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.733+0000 7f83c9600640 1 -- 172.21.15.7:0/994986375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51400400 msgr2=0x557c5140a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:02.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.733+0000 7f83c9600640 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51400400 0x557c5140a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.733+0000 7f83c9600640 1 -- 172.21.15.7:0/994986375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c510783f0 con 0x557c51405570 2024-03-18T01:41:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.734+0000 7f83c9600640 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51405570 0x557c51411510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f83bc09b040 tx=0x7f83bc09ae80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=139ec773ebf6594f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.734+0000 7f83aa000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:41:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.734+0000 7f83aa000640 1 -- 172.21.15.7:0/994986375 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc01a040 con 0x557c51405570 2024-03-18T01:41:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.735+0000 7f83aa000640 1 -- 172.21.15.7:0/994986375 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f83bc003970 con 0x557c51405570 2024-03-18T01:41:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.735+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83b0018b50 con 0x557c51405570 2024-03-18T01:41:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.735+0000 7f83aa000640 1 -- 172.21.15.7:0/994986375 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f83bc004cc0 con 0x557c51405570 2024-03-18T01:41:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83ce366740 10 client.26165 fetch_fsmap learned FSMap version 31 2024-03-18T01:41:02.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557c510380f0 con 0x557c51405570 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83aa000640 1 -- 172.21.15.7:0/994986375 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83b0018b50 con 0x557c51405570 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83aa000640 1 -- 172.21.15.7:0/994986375 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f83bc00b040 con 0x557c51405570 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83ce366740 10 client.26165 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83ce366740 10 client.26165 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83ce366740 20 client.26165 populate_metadata read hostname 'smithi007' 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83ce366740 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c51415e30 0x557c51418290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.736+0000 7f83ce366740 4 client.26165 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.737+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c51377be0 con 0x557c51415e30 2024-03-18T01:41:02.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.737+0000 7f83c8c00640 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c51415e30 0x557c51418290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.738+0000 7f83c8c00640 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c51415e30 0x557c51418290 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96c2e080cd394e7a server_cookie=5fedd3b76e17828a in_seq=0 out_seq=0 2024-03-18T01:41:02.739 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.738+0000 7f83aa000640 10 client.26165 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:41:02.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83aa000640 1 -- 172.21.15.7:0/994986375 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c51377be0 con 0x557c51415e30 2024-03-18T01:41:02.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83aa000640 10 client.26165 handle_command_reply: tid=0 2024-03-18T01:41:02.741 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 client.26165 shutdown 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51405570 msgr2=0x557c51411510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51405570 0x557c51411510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f83bc09b040 tx=0x7f83bc09ae80 comp rx=0 tx=0).stop 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 shutdown_connections 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c51415e30 0x557c51418290 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c51405570 0x557c51411510 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83aa000640 0 client.26165 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51400400 0x557c5140a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 --2- 172.21.15.7:0/994986375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c513ffa10 0x557c51409f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.740+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 >> 172.21.15.7:0/994986375 conn(0x557c513faf80 msgr2=0x557c513fa110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:02.742 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.741+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 shutdown_connections 2024-03-18T01:41:02.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.742+0000 7f83ce366740 1 -- 172.21.15.7:0/994986375 wait complete. 2024-03-18T01:41:02.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.742+0000 7f83ce366740 20 client.26165 trim_cache size 0 max 16384 2024-03-18T01:41:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.792+0000 7f83ce366740 1 Processor -- start 2024-03-18T01:41:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.792+0000 7f83ce366740 1 -- start start 2024-03-18T01:41:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.793+0000 7f83ce366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c513ffa10 0x557c51408a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.793+0000 7f83ce366740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c514114d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.793+0000 7f83ce366740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51408f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.793+0000 7f83ce366740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41b96e0 con 0x557c51405570 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.793+0000 7f83ce366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41bb2e0 con 0x557c513ffa10 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.793+0000 7f83ce366740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41b9db0 con 0x557c51400400 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.793+0000 7f83c3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c514114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c514114d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44542/0 (socket says 172.21.15.7:44542) 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c3e00640 1 -- 172.21.15.7:0/1992674179 learned_addr learned my addr 172.21.15.7:0/1992674179 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c8c00640 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c513ffa10 0x557c51408a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c9600640 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51408f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c8c00640 1 -- 172.21.15.7:0/1992674179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 msgr2=0x557c514114d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c8c00640 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c514114d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c8c00640 1 -- 172.21.15.7:0/1992674179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 msgr2=0x557c51408f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c8c00640 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51408f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c8c00640 1 -- 172.21.15.7:0/1992674179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c514087b0 con 0x557c513ffa10 2024-03-18T01:41:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.794+0000 7f83c9600640 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51408f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:41:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.795+0000 7f83c8c00640 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c513ffa10 0x557c51408a40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009780 tx=0x7f83b803c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d8df0702d127895 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.795+0000 7f83b5200640 1 -- 172.21.15.7:0/1992674179 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8004020 con 0x557c513ffa10 2024-03-18T01:41:02.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.796+0000 7f83b5200640 1 -- 172.21.15.7:0/1992674179 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83b803a040 con 0x557c513ffa10 2024-03-18T01:41:02.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.796+0000 7f83b5200640 1 -- 172.21.15.7:0/1992674179 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b804d6c0 con 0x557c513ffa10 2024-03-18T01:41:02.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 -- 172.21.15.7:0/1992674179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c513ffa10 msgr2=0x557c51408a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.798 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c513ffa10 0x557c51408a40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009780 tx=0x7f83b803c000 comp rx=0 tx=0).stop 2024-03-18T01:41:02.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 -- 172.21.15.7:0/1992674179 shutdown_connections 2024-03-18T01:41:02.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c51408f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c514114d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 --2- 172.21.15.7:0/1992674179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c513ffa10 0x557c51408a40 secure :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009780 tx=0x7f83b803c000 comp rx=0 tx=0).stop 2024-03-18T01:41:02.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 -- 172.21.15.7:0/1992674179 >> 172.21.15.7:0/1992674179 conn(0x557c513faf80 msgr2=0x557c514019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:02.799 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 -- 172.21.15.7:0/1992674179 shutdown_connections 2024-03-18T01:41:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.797+0000 7f83ce366740 1 -- 172.21.15.7:0/1992674179 wait complete. 2024-03-18T01:41:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.798+0000 7f83ce366740 1 -- start start 2024-03-18T01:41:02.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.798+0000 7f83ce366740 1 Processor -- start 2024-03-18T01:41:02.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.799+0000 7f83ce366740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c5140efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.799+0000 7f83ce366740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c5140f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.800+0000 7f83ce366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c514104c0 0x557c51410940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.800+0000 7f83ce366740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41bab50 con 0x557c51405570 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.800+0000 7f83c8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c5140efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.800+0000 7f83c8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c5140efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44552/0 (socket says 172.21.15.7:44552) 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.800+0000 7f83c8c00640 1 -- 172.21.15.7:0/1640207347 learned_addr learned my addr 172.21.15.7:0/1640207347 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.800+0000 7f83c9600640 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c514104c0 0x557c51410940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.800+0000 7f83ce366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41ba480 con 0x557c514104c0 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.803+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41b76c0 con 0x557c51400400 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.803+0000 7f83ce366740 10 client.0 _ll_register_callbacks cb 0x7f83bc004ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:41:02.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.803+0000 7f83c3e00640 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c5140f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.804+0000 7f83c3e00640 1 -- 172.21.15.7:0/1640207347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 msgr2=0x557c5140efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:02.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.804+0000 7f83c3e00640 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c5140efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.804+0000 7f83c3e00640 1 -- 172.21.15.7:0/1640207347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c514104c0 msgr2=0x557c51410940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:02.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.804+0000 7f83c3e00640 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c514104c0 0x557c51410940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.804+0000 7f83c3e00640 1 -- 172.21.15.7:0/1640207347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c51410e80 con 0x557c51405570 2024-03-18T01:41:02.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.805+0000 7f83c3e00640 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c5140f500 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f83b0006b90 tx=0x7f83b00186a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972607e3270a18d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:02.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.805+0000 7f83aa000640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:41:02.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.806+0000 7f83aa000640 1 -- 172.21.15.7:0/1640207347 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b001d040 con 0x557c51405570 2024-03-18T01:41:02.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.806+0000 7f83aa000640 1 -- 172.21.15.7:0/1640207347 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f83b0018a10 con 0x557c51405570 2024-03-18T01:41:02.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.806+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83b803a040 con 0x557c51405570 2024-03-18T01:41:02.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.806+0000 7f83aa000640 1 -- 172.21.15.7:0/1640207347 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f83b0018c10 con 0x557c51405570 2024-03-18T01:41:02.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.806+0000 7f83ce366740 10 client.15909 fetch_fsmap learned FSMap version 31 2024-03-18T01:41:02.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.806+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557c5147c940 con 0x557c51405570 2024-03-18T01:41:02.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.807+0000 7f83aa000640 1 -- 172.21.15.7:0/1640207347 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83b803a040 con 0x557c51405570 2024-03-18T01:41:02.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.807+0000 7f83aa000640 1 -- 172.21.15.7:0/1640207347 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f83b0015040 con 0x557c51405570 2024-03-18T01:41:02.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.807+0000 7f83ce366740 10 client.15909 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:41:02.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.807+0000 7f83ce366740 10 client.15909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:41:02.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.807+0000 7f83ce366740 20 client.15909 populate_metadata read hostname 'smithi007' 2024-03-18T01:41:02.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.807+0000 7f83ce366740 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c5147d000 0x557c5147f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:02.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.807+0000 7f83ce366740 4 client.15909 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:41:02.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.807+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557c51371730 con 0x557c5147d000 2024-03-18T01:41:02.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.808+0000 7f83c8c00640 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c5147d000 0x557c5147f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:02.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.812+0000 7f83aa000640 10 client.15909 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:41:02.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.812+0000 7f83c8c00640 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c5147d000 0x557c5147f420 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2d06996d6498f42 server_cookie=4cefa0955c72413b in_seq=0 out_seq=0 2024-03-18T01:41:02.820 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:41:02.820 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (16 inodes in the stack)", 2024-03-18T01:41:02.820 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:41:02.821 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:41:02.821 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:41:02.821 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:41:02.821 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:41:02.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:41:02.821 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:41:02.821 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:41:02.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.815+0000 7f83aa000640 1 -- 172.21.15.7:0/1640207347 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x557c51371730 con 0x557c5147d000 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.815+0000 7f83aa000640 10 client.15909 handle_command_reply: tid=0 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.815+0000 7f83ce366740 1 client.15909 shutdown 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.815+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 msgr2=0x557c5140f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.815+0000 7f83ce366740 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c5140f500 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f83b0006b90 tx=0x7f83b00186a0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.816+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 shutdown_connections 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.816+0000 7f83ce366740 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x557c5147d000 0x557c5147f420 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.816+0000 7f83ce366740 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c514104c0 0x557c51410940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.816+0000 7f83aa000640 0 client.15909 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.816+0000 7f83ce366740 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557c51405570 0x557c5140f500 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.816+0000 7f83ce366740 1 --2- 172.21.15.7:0/1640207347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557c51400400 0x557c5140efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.816+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 >> 172.21.15.7:0/1640207347 conn(0x557c513faf80 msgr2=0x557c51408a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.816+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 shutdown_connections 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.817+0000 7f83ce366740 1 -- 172.21.15.7:0/1640207347 wait complete. 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.817+0000 7f83ce366740 20 client.15909 trim_cache size 0 max 16384 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.818+0000 7f83ca000640 1 -- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f83ac0a29b0 msgr2=0x7f83ac0a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.818+0000 7f83ca000640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f83ac0a29b0 0x7f83ac0a4e70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x557c51400800 tx=0x7f83b8009490 comp rx=0 tx=0).stop 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.818+0000 7f83ca000640 1 -- 172.21.15.7:0/2196196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83c41abfd0 msgr2=0x7f83c41ae3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.818+0000 7f83ca000640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83c41abfd0 0x7f83c41ae3c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009fd0 tx=0x7f83bc00c800 comp rx=0 tx=0).stop 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.819+0000 7f83ca000640 1 -- 172.21.15.7:0/2196196123 shutdown_connections 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.819+0000 7f83ca000640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83c41abfd0 0x7f83c41ae3c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.819+0000 7f83ca000640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f83ac0a29b0 0x7f83ac0a4e70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.819+0000 7f83ca000640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f83c41097c0 0x7f83c41a6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.819+0000 7f83ca000640 1 --2- 172.21.15.7:0/2196196123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c4108e00 0x7f83c41a6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.819+0000 7f83ca000640 1 -- 172.21.15.7:0/2196196123 >> 172.21.15.7:0/2196196123 conn(0x7f83c406e620 msgr2=0x7f83c4070330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.820+0000 7f83ca000640 1 -- 172.21.15.7:0/2196196123 shutdown_connections 2024-03-18T01:41:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:02.821+0000 7f83ca000640 1 -- 172.21.15.7:0/2196196123 wait complete. 2024-03-18T01:41:02.853 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:41:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:02 smithi203 ceph-mon[36345]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 57 MiB/s wr, 7.91k op/s 2024-03-18T01:41:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:02 smithi046 ceph-mon[36108]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 57 MiB/s wr, 7.91k op/s 2024-03-18T01:41:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:02 smithi007 ceph-mon[31996]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 57 MiB/s wr, 7.91k op/s 2024-03-18T01:41:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:04 smithi203 ceph-mon[36345]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 52 MiB/s wr, 6.92k op/s 2024-03-18T01:41:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:04 smithi046 ceph-mon[36108]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 52 MiB/s wr, 6.92k op/s 2024-03-18T01:41:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:04 smithi007 ceph-mon[31996]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 52 MiB/s wr, 6.92k op/s 2024-03-18T01:41:07.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:06 smithi203 ceph-mon[36345]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 60 MiB/s wr, 8.08k op/s 2024-03-18T01:41:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:06 smithi046 ceph-mon[36108]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 60 MiB/s wr, 8.08k op/s 2024-03-18T01:41:07.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:06 smithi007 ceph-mon[31996]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 60 MiB/s wr, 8.08k op/s 2024-03-18T01:41:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:08 smithi203 ceph-mon[36345]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 58 MiB/s wr, 7.76k op/s 2024-03-18T01:41:09.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:08 smithi046 ceph-mon[36108]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 58 MiB/s wr, 7.76k op/s 2024-03-18T01:41:09.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:08 smithi007 ceph-mon[31996]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 58 MiB/s wr, 7.76k op/s 2024-03-18T01:41:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:10 smithi203 ceph-mon[36345]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 62 MiB/s wr, 8.41k op/s 2024-03-18T01:41:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:10 smithi046 ceph-mon[36108]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 62 MiB/s wr, 8.41k op/s 2024-03-18T01:41:11.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:10 smithi007 ceph-mon[31996]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 62 MiB/s wr, 8.41k op/s 2024-03-18T01:41:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:12 smithi203 ceph-mon[36345]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 58 MiB/s wr, 8.13k op/s 2024-03-18T01:41:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:12 smithi046 ceph-mon[36108]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 58 MiB/s wr, 8.13k op/s 2024-03-18T01:41:13.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:12 smithi007 ceph-mon[31996]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 58 MiB/s wr, 8.13k op/s 2024-03-18T01:41:15.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:14 smithi046 ceph-mon[36108]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 52 MiB/s wr, 7.34k op/s 2024-03-18T01:41:15.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:14 smithi007 ceph-mon[31996]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 52 MiB/s wr, 7.34k op/s 2024-03-18T01:41:15.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:14 smithi203 ceph-mon[36345]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 52 MiB/s wr, 7.34k op/s 2024-03-18T01:41:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:16 smithi046 ceph-mon[36108]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 55 MiB/s wr, 7.89k op/s 2024-03-18T01:41:17.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:16 smithi007 ceph-mon[31996]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 55 MiB/s wr, 7.89k op/s 2024-03-18T01:41:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:16 smithi203 ceph-mon[36345]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 55 MiB/s wr, 7.89k op/s 2024-03-18T01:41:19.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:18 smithi046 ceph-mon[36108]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 58 MiB/s wr, 8.14k op/s 2024-03-18T01:41:19.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:18 smithi007 ceph-mon[31996]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 58 MiB/s wr, 8.14k op/s 2024-03-18T01:41:19.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:18 smithi203 ceph-mon[36345]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 58 MiB/s wr, 8.14k op/s 2024-03-18T01:41:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:20 smithi046 ceph-mon[36108]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 MiB/s wr, 7.26k op/s 2024-03-18T01:41:21.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:20 smithi007 ceph-mon[31996]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 MiB/s wr, 7.26k op/s 2024-03-18T01:41:21.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:20 smithi203 ceph-mon[36345]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 MiB/s wr, 7.26k op/s 2024-03-18T01:41:23.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:22 smithi046 ceph-mon[36108]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 56 MiB/s wr, 7.84k op/s 2024-03-18T01:41:23.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:22 smithi007 ceph-mon[31996]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 56 MiB/s wr, 7.84k op/s 2024-03-18T01:41:23.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:22 smithi203 ceph-mon[36345]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 56 MiB/s wr, 7.84k op/s 2024-03-18T01:41:25.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:24 smithi046 ceph-mon[36108]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 53 MiB/s wr, 7.29k op/s 2024-03-18T01:41:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:24 smithi007 ceph-mon[31996]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 53 MiB/s wr, 7.29k op/s 2024-03-18T01:41:25.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:24 smithi203 ceph-mon[36345]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 53 MiB/s wr, 7.29k op/s 2024-03-18T01:41:27.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:26 smithi046 ceph-mon[36108]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 58 MiB/s wr, 7.92k op/s 2024-03-18T01:41:27.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:26 smithi007 ceph-mon[31996]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 58 MiB/s wr, 7.92k op/s 2024-03-18T01:41:27.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:26 smithi203 ceph-mon[36345]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 58 MiB/s wr, 7.92k op/s 2024-03-18T01:41:29.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:28 smithi046 ceph-mon[36108]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 65 MiB/s wr, 8.74k op/s 2024-03-18T01:41:29.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:28 smithi007 ceph-mon[31996]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 65 MiB/s wr, 8.74k op/s 2024-03-18T01:41:29.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:28 smithi203 ceph-mon[36345]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 65 MiB/s wr, 8.74k op/s 2024-03-18T01:41:31.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:30 smithi046 ceph-mon[36108]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 63 MiB/s wr, 8.20k op/s 2024-03-18T01:41:31.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:30 smithi007 ceph-mon[31996]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 63 MiB/s wr, 8.20k op/s 2024-03-18T01:41:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:30 smithi203 ceph-mon[36345]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 63 MiB/s wr, 8.20k op/s 2024-03-18T01:41:32.854 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:41:32.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:41:33.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:32 smithi046 ceph-mon[36108]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 72 MiB/s wr, 9.21k op/s 2024-03-18T01:41:33.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:32 smithi007 ceph-mon[31996]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 72 MiB/s wr, 9.21k op/s 2024-03-18T01:41:33.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:32 smithi203 ceph-mon[36345]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 72 MiB/s wr, 9.21k op/s 2024-03-18T01:41:35.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:34 smithi046 ceph-mon[36108]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 61 MiB/s wr, 7.84k op/s 2024-03-18T01:41:35.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:34 smithi203 ceph-mon[36345]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 61 MiB/s wr, 7.84k op/s 2024-03-18T01:41:35.765 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:34 smithi007 ceph-mon[31996]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 61 MiB/s wr, 7.84k op/s 2024-03-18T01:41:37.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:36 smithi046 ceph-mon[36108]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 62 MiB/s wr, 7.93k op/s 2024-03-18T01:41:37.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:36 smithi007 ceph-mon[31996]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 62 MiB/s wr, 7.93k op/s 2024-03-18T01:41:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:36 smithi203 ceph-mon[36345]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 62 MiB/s wr, 7.93k op/s 2024-03-18T01:41:39.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:39 smithi007 ceph-mon[31996]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 65 MiB/s wr, 8.45k op/s 2024-03-18T01:41:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:39 smithi203 ceph-mon[36345]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 65 MiB/s wr, 8.45k op/s 2024-03-18T01:41:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:39 smithi046 ceph-mon[36108]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 65 MiB/s wr, 8.45k op/s 2024-03-18T01:41:41.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:41 smithi203 ceph-mon[36345]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 59 MiB/s wr, 7.70k op/s 2024-03-18T01:41:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:41 smithi046 ceph-mon[36108]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 59 MiB/s wr, 7.70k op/s 2024-03-18T01:41:41.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:41 smithi007 ceph-mon[31996]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 59 MiB/s wr, 7.70k op/s 2024-03-18T01:41:43.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:43 smithi203 ceph-mon[36345]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 56 MiB/s wr, 7.65k op/s 2024-03-18T01:41:43.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:43 smithi046 ceph-mon[36108]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 56 MiB/s wr, 7.65k op/s 2024-03-18T01:41:43.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:43 smithi007 ceph-mon[31996]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 56 MiB/s wr, 7.65k op/s 2024-03-18T01:41:45.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:45 smithi203 ceph-mon[36345]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 MiB/s wr, 6.98k op/s 2024-03-18T01:41:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:45 smithi046 ceph-mon[36108]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 MiB/s wr, 6.98k op/s 2024-03-18T01:41:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:45 smithi007 ceph-mon[31996]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 52 MiB/s wr, 6.98k op/s 2024-03-18T01:41:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.772+0000 7fba93a00640 1 -- 172.21.15.7:0/1702762683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c100700 msgr2=0x7fba8c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.772+0000 7fba93a00640 1 --2- 172.21.15.7:0/1702762683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c100700 0x7fba8c100b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fba80009900 tx=0x7fba8002bc40 comp rx=0 tx=0).stop 2024-03-18T01:41:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.773+0000 7fba93a00640 1 -- 172.21.15.7:0/1702762683 shutdown_connections 2024-03-18T01:41:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.773+0000 7fba93a00640 1 --2- 172.21.15.7:0/1702762683 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c1022b0 0x7fba8c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.773+0000 7fba93a00640 1 --2- 172.21.15.7:0/1702762683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba8c1018f0 0x7fba8c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.773+0000 7fba93a00640 1 --2- 172.21.15.7:0/1702762683 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c100700 0x7fba8c100b20 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fba80009900 tx=0x7fba8002bc40 comp rx=0 tx=0).stop 2024-03-18T01:41:46.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.773+0000 7fba93a00640 1 -- 172.21.15.7:0/1702762683 >> 172.21.15.7:0/1702762683 conn(0x7fba8c0fbc70 msgr2=0x7fba8c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.774+0000 7fba93a00640 1 -- 172.21.15.7:0/1702762683 shutdown_connections 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.774+0000 7fba93a00640 1 -- 172.21.15.7:0/1702762683 wait complete. 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.776+0000 7fba93a00640 1 Processor -- start 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.776+0000 7fba93a00640 1 -- start start 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba93a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c1018f0 0x7fba8c0703c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba93a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba8c1022b0 0x7fba8c070900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba93a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c192940 0x7fba8c194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba93a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c10b290 con 0x7fba8c1018f0 2024-03-18T01:41:46.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba93a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c10b110 con 0x7fba8c1022b0 2024-03-18T01:41:46.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba93a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c10b410 con 0x7fba8c192940 2024-03-18T01:41:46.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba8c1022b0 0x7fba8c070900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba8c1022b0 0x7fba8c070900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:42942/0 (socket says 172.21.15.7:42942) 2024-03-18T01:41:46.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba8be00640 1 -- 172.21.15.7:0/2574153398 learned_addr learned my addr 172.21.15.7:0/2574153398 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.777+0000 7fba91a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c192940 0x7fba8c194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.778+0000 7fba91000640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c1018f0 0x7fba8c0703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.778+0000 7fba91a00640 1 -- 172.21.15.7:0/2574153398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba8c1022b0 msgr2=0x7fba8c070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.778+0000 7fba91a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba8c1022b0 0x7fba8c070900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.778+0000 7fba91a00640 1 -- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c1018f0 msgr2=0x7fba8c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.778+0000 7fba91a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c1018f0 0x7fba8c0703c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.778+0000 7fba91a00640 1 -- 172.21.15.7:0/2574153398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c195320 con 0x7fba8c192940 2024-03-18T01:41:46.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.779+0000 7fba91a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c192940 0x7fba8c194de0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fba84010c30 tx=0x7fba8400e430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:46.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.779+0000 7fba89600640 1 -- 172.21.15.7:0/2574153398 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8401a020 con 0x7fba8c192940 2024-03-18T01:41:46.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.779+0000 7fba89600640 1 -- 172.21.15.7:0/2574153398 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba84010e40 con 0x7fba8c192940 2024-03-18T01:41:46.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.780+0000 7fba89600640 1 -- 172.21.15.7:0/2574153398 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84015810 con 0x7fba8c192940 2024-03-18T01:41:46.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.780+0000 7fba93a00640 1 -- 172.21.15.7:0/2574153398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8c195610 con 0x7fba8c192940 2024-03-18T01:41:46.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.780+0000 7fba93a00640 1 -- 172.21.15.7:0/2574153398 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8c195b60 con 0x7fba8c192940 2024-03-18T01:41:46.786 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.786+0000 7fba89600640 1 -- 172.21.15.7:0/2574153398 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fba8401d020 con 0x7fba8c192940 2024-03-18T01:41:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.789+0000 7fba89600640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba740a2880 0x7fba740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.790+0000 7fba89600640 1 -- 172.21.15.7:0/2574153398 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fba840d2e10 con 0x7fba8c192940 2024-03-18T01:41:46.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.790+0000 7fba91000640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba740a2880 0x7fba740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.793+0000 7fba91000640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba740a2880 0x7fba740a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fba8002c7c0 tx=0x7fba80005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:46.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.802+0000 7fba967ca740 1 Processor -- start 2024-03-18T01:41:46.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.804+0000 7fba967ca740 1 -- start start 2024-03-18T01:41:46.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.805+0000 7fba967ca740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53e07440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.805+0000 7fba967ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e0c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.805+0000 7fba967ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 0x555a53e0f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba967ca740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555a53d631f0 con 0x555a53e07020 2024-03-18T01:41:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba967ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555a53d0ba80 con 0x555a53e07a10 2024-03-18T01:41:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba967ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555a53d5f290 con 0x555a53e0cb80 2024-03-18T01:41:46.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e0c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e0c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:42954/0 (socket says 172.21.15.7:42954) 2024-03-18T01:41:46.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba8be00640 1 -- 172.21.15.7:0/2283710150 learned_addr learned my addr 172.21.15.7:0/2283710150 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:46.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba8be00640 1 -- 172.21.15.7:0/2283710150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 msgr2=0x555a53e0f010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:41:46.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba8be00640 1 --2- 172.21.15.7:0/2283710150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 0x555a53e0f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba8be00640 1 -- 172.21.15.7:0/2283710150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 msgr2=0x555a53e07440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:41:46.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba8be00640 1 --2- 172.21.15.7:0/2283710150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53e07440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.806+0000 7fba8be00640 1 -- 172.21.15.7:0/2283710150 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a53b417d0 con 0x555a53e07a10 2024-03-18T01:41:46.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.809+0000 7fba8be00640 1 --2- 172.21.15.7:0/2283710150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e0c510 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba7800bd60 tx=0x7fba780107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ca8f6fd1f5280d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:46.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.810+0000 7fba7e000640 1 -- 172.21.15.7:0/2283710150 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba78011640 con 0x555a53e07a10 2024-03-18T01:41:46.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.810+0000 7fba7e000640 1 -- 172.21.15.7:0/2283710150 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba78011cc0 con 0x555a53e07a10 2024-03-18T01:41:46.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.810+0000 7fba7e000640 1 -- 172.21.15.7:0/2283710150 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7800dba0 con 0x555a53e07a10 2024-03-18T01:41:46.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 -- 172.21.15.7:0/2283710150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 msgr2=0x555a53e0c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 --2- 172.21.15.7:0/2283710150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e0c510 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba7800bd60 tx=0x7fba780107a0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 -- 172.21.15.7:0/2283710150 shutdown_connections 2024-03-18T01:41:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 --2- 172.21.15.7:0/2283710150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 0x555a53e0f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 --2- 172.21.15.7:0/2283710150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e0c510 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 --2- 172.21.15.7:0/2283710150 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53e07440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 -- 172.21.15.7:0/2283710150 >> 172.21.15.7:0/2283710150 conn(0x555a53df9d80 msgr2=0x555a53dfa180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 -- 172.21.15.7:0/2283710150 shutdown_connections 2024-03-18T01:41:46.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.811+0000 7fba967ca740 1 -- 172.21.15.7:0/2283710150 wait complete. 2024-03-18T01:41:46.813 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.812+0000 7fba967ca740 1 -- start start 2024-03-18T01:41:46.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.813+0000 7fba967ca740 1 Processor -- start 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.814+0000 7fba967ca740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53e115f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.814+0000 7fba967ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e11b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.814+0000 7fba967ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e0cb80 0x555a53e18b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.814+0000 7fba967ca740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555a53e0fb30 con 0x555a53e07020 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.814+0000 7fba967ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555a53d1eed0 con 0x555a53e0cb80 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.814+0000 7fba967ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555a53e0fcb0 con 0x555a53e07a10 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.814+0000 7fba967ca740 10 client.0 _ll_register_callbacks cb 0x555a53d5a220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.815+0000 7fba8be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e11b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.815+0000 7fba8be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e11b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59870/0 (socket says 172.21.15.7:59870) 2024-03-18T01:41:46.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.815+0000 7fba8be00640 1 -- 172.21.15.7:0/3009871002 learned_addr learned my addr 172.21.15.7:0/3009871002 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:46.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.815+0000 7fba8be00640 1 -- 172.21.15.7:0/3009871002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e0cb80 msgr2=0x555a53e18b70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:41:46.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.816+0000 7fba91000640 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53e115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.816+0000 7fba8be00640 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e0cb80 0x555a53e18b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.816+0000 7fba8be00640 1 -- 172.21.15.7:0/3009871002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 msgr2=0x555a53e115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:46.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.816+0000 7fba8be00640 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53e115f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.816+0000 7fba8be00640 1 -- 172.21.15.7:0/3009871002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a53a773f0 con 0x555a53e07a10 2024-03-18T01:41:46.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.816+0000 7fba91000640 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53e115f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:41:46.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.816+0000 7fba8be00640 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e11b30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fba7800bd30 tx=0x7fba78011f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=448435b0f5b2650b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:46.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.816+0000 7fba72a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:41:46.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.817+0000 7fba72a00640 1 -- 172.21.15.7:0/3009871002 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7800db30 con 0x555a53e07a10 2024-03-18T01:41:46.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.817+0000 7fba72a00640 1 -- 172.21.15.7:0/3009871002 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fba7800dcd0 con 0x555a53e07a10 2024-03-18T01:41:46.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.817+0000 7fba967ca740 1 -- 172.21.15.7:0/3009871002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba78011cc0 con 0x555a53e07a10 2024-03-18T01:41:46.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.817+0000 7fba72a00640 1 -- 172.21.15.7:0/3009871002 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fba78005040 con 0x555a53e07a10 2024-03-18T01:41:46.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.817+0000 7fba967ca740 10 client.26149 fetch_fsmap learned FSMap version 31 2024-03-18T01:41:46.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.817+0000 7fba967ca740 1 -- 172.21.15.7:0/3009871002 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a53a370f0 con 0x555a53e07a10 2024-03-18T01:41:46.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.818+0000 7fba72a00640 1 -- 172.21.15.7:0/3009871002 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba78011cc0 con 0x555a53e07a10 2024-03-18T01:41:46.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.818+0000 7fba72a00640 1 -- 172.21.15.7:0/3009871002 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fba780046c0 con 0x555a53e07a10 2024-03-18T01:41:46.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.819+0000 7fba967ca740 10 client.26149 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:41:46.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.819+0000 7fba967ca740 10 client.26149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:41:46.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.819+0000 7fba967ca740 20 client.26149 populate_metadata read hostname 'smithi007' 2024-03-18T01:41:46.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.819+0000 7fba967ca740 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x555a53e1d060 0x555a53e1f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.819+0000 7fba967ca740 4 client.26149 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:41:46.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.819+0000 7fba967ca740 1 -- 172.21.15.7:0/3009871002 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a53d76be0 con 0x555a53e1d060 2024-03-18T01:41:46.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.820+0000 7fba91000640 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x555a53e1d060 0x555a53e1f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.821+0000 7fba91000640 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x555a53e1d060 0x555a53e1f480 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e03af196fad0716 server_cookie=817888c39ad5e35c in_seq=0 out_seq=0 2024-03-18T01:41:46.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.821+0000 7fba72a00640 10 client.26149 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:41:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.821+0000 7fba72a00640 1 -- 172.21.15.7:0/3009871002 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555a53d76be0 con 0x555a53e1d060 2024-03-18T01:41:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.822+0000 7fba72a00640 10 client.26149 handle_command_reply: tid=0 2024-03-18T01:41:46.822 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.822+0000 7fba967ca740 1 client.26149 shutdown 2024-03-18T01:41:46.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.823+0000 7fba967ca740 1 -- 172.21.15.7:0/3009871002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 msgr2=0x555a53e11b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.823+0000 7fba967ca740 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e11b30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fba7800bd30 tx=0x7fba78011f90 comp rx=0 tx=0).stop 2024-03-18T01:41:46.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba967ca740 1 -- 172.21.15.7:0/3009871002 shutdown_connections 2024-03-18T01:41:46.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba967ca740 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x555a53e1d060 0x555a53e1f480 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba967ca740 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e0cb80 0x555a53e18b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba967ca740 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e11b30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba967ca740 1 --2- 172.21.15.7:0/3009871002 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53e115f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba72a00640 0 client.26149 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:41:46.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba967ca740 1 -- 172.21.15.7:0/3009871002 >> 172.21.15.7:0/3009871002 conn(0x555a53df9d80 msgr2=0x555a53df8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:46.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba967ca740 1 -- 172.21.15.7:0/3009871002 shutdown_connections 2024-03-18T01:41:46.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.824+0000 7fba967ca740 1 -- 172.21.15.7:0/3009871002 wait complete. 2024-03-18T01:41:46.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.825+0000 7fba967ca740 20 client.26149 trim_cache size 0 max 16384 2024-03-18T01:41:46.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.870+0000 7fba967ca740 1 Processor -- start 2024-03-18T01:41:46.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.870+0000 7fba967ca740 1 -- start start 2024-03-18T01:41:46.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.870+0000 7fba967ca740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53dfa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.870+0000 7fba967ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e18b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.870+0000 7fba967ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 0x555a53e10080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.870+0000 7fba967ca740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c19ff80 con 0x555a53e07020 2024-03-18T01:41:46.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.870+0000 7fba967ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1c3b90 con 0x555a53e07a10 2024-03-18T01:41:46.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.870+0000 7fba967ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1a0650 con 0x555a53e0cb80 2024-03-18T01:41:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.871+0000 7fba91000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53dfa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.871+0000 7fba91000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53dfa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:51548/0 (socket says 172.21.15.7:51548) 2024-03-18T01:41:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.871+0000 7fba91000640 1 -- 172.21.15.7:0/3694717397 learned_addr learned my addr 172.21.15.7:0/3694717397 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:46.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.871+0000 7fba8be00640 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e18b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.871+0000 7fba91a00640 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 0x555a53e10080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.872+0000 7fba91000640 1 -- 172.21.15.7:0/3694717397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 msgr2=0x555a53e10080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.872+0000 7fba91000640 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 0x555a53e10080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.872+0000 7fba91000640 1 -- 172.21.15.7:0/3694717397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 msgr2=0x555a53e18b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.872+0000 7fba91000640 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e18b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.872+0000 7fba91000640 1 -- 172.21.15.7:0/3694717397 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a53df4370 con 0x555a53e07020 2024-03-18T01:41:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.872+0000 7fba91a00640 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 0x555a53e10080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:41:46.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.873+0000 7fba91000640 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53dfa160 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fba80009900 tx=0x7fba8003c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc5e9dd8de2b323 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.874+0000 7fba7d600640 1 -- 172.21.15.7:0/3694717397 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8002caa0 con 0x555a53e07020 2024-03-18T01:41:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.874+0000 7fba7d600640 1 -- 172.21.15.7:0/3694717397 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba80005520 con 0x555a53e07020 2024-03-18T01:41:46.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.874+0000 7fba7d600640 1 -- 172.21.15.7:0/3694717397 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80003850 con 0x555a53e07020 2024-03-18T01:41:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.875+0000 7fba967ca740 1 -- 172.21.15.7:0/3694717397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 msgr2=0x555a53dfa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.875+0000 7fba967ca740 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53dfa160 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fba80009900 tx=0x7fba8003c000 comp rx=0 tx=0).stop 2024-03-18T01:41:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.876+0000 7fba967ca740 1 -- 172.21.15.7:0/3694717397 shutdown_connections 2024-03-18T01:41:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.876+0000 7fba967ca740 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e0cb80 0x555a53e10080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.876+0000 7fba967ca740 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07a10 0x555a53e18b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.876+0000 7fba967ca740 1 --2- 172.21.15.7:0/3694717397 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e07020 0x555a53dfa160 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.876+0000 7fba967ca740 1 -- 172.21.15.7:0/3694717397 >> 172.21.15.7:0/3694717397 conn(0x555a53df9d80 msgr2=0x555a53e08fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:46.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.877+0000 7fba967ca740 1 -- 172.21.15.7:0/3694717397 shutdown_connections 2024-03-18T01:41:46.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.878+0000 7fba967ca740 1 -- 172.21.15.7:0/3694717397 wait complete. 2024-03-18T01:41:46.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.878+0000 7fba967ca740 1 -- start start 2024-03-18T01:41:46.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.879+0000 7fba967ca740 1 Processor -- start 2024-03-18T01:41:46.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba967ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07020 0x555a53e15d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba967ca740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e16280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba967ca740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e0cb80 0x555a53e17290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba967ca740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1a13f0 con 0x555a53e0cb80 2024-03-18T01:41:46.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba967ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c1a0d20 con 0x555a53e07020 2024-03-18T01:41:46.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba967ca740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c19df60 con 0x555a53e07a10 2024-03-18T01:41:46.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba967ca740 10 client.0 _ll_register_callbacks cb 0x7fba7800bbb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:41:46.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba8be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e16280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba8be00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e16280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59902/0 (socket says 172.21.15.7:59902) 2024-03-18T01:41:46.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.881+0000 7fba8be00640 1 -- 172.21.15.7:0/241625478 learned_addr learned my addr 172.21.15.7:0/241625478 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:41:46.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.883+0000 7fba8be00640 1 -- 172.21.15.7:0/241625478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07020 msgr2=0x555a53e15d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:41:46.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.883+0000 7fba8be00640 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07020 0x555a53e15d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.883+0000 7fba8be00640 1 -- 172.21.15.7:0/241625478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e0cb80 msgr2=0x555a53e17290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:41:46.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.883+0000 7fba8be00640 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e0cb80 0x555a53e17290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.883+0000 7fba8be00640 1 -- 172.21.15.7:0/241625478 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a53e17880 con 0x555a53e07a10 2024-03-18T01:41:46.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.884+0000 7fba8be00640 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e16280 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fba78006010 tx=0x7fba7800ba70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=236f8e5c541a6c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:41:46.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.884+0000 7fba72a00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:41:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.884+0000 7fba72a00640 1 -- 172.21.15.7:0/241625478 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba78004e40 con 0x555a53e07a10 2024-03-18T01:41:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.884+0000 7fba72a00640 1 -- 172.21.15.7:0/241625478 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fba7800dd40 con 0x555a53e07a10 2024-03-18T01:41:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.884+0000 7fba967ca740 1 -- 172.21.15.7:0/241625478 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba80005520 con 0x555a53e07a10 2024-03-18T01:41:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.884+0000 7fba72a00640 1 -- 172.21.15.7:0/241625478 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fba780051e0 con 0x555a53e07a10 2024-03-18T01:41:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.885+0000 7fba967ca740 10 client.26152 fetch_fsmap learned FSMap version 31 2024-03-18T01:41:46.885 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.885+0000 7fba967ca740 1 -- 172.21.15.7:0/241625478 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555a53e18b00 con 0x555a53e07a10 2024-03-18T01:41:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.885+0000 7fba72a00640 1 -- 172.21.15.7:0/241625478 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba80005520 con 0x555a53e07a10 2024-03-18T01:41:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.887+0000 7fba72a00640 1 -- 172.21.15.7:0/241625478 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fba780035f0 con 0x555a53e07a10 2024-03-18T01:41:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.887+0000 7fba967ca740 10 client.26152 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:41:46.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.887+0000 7fba967ca740 10 client.26152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:41:46.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.887+0000 7fba967ca740 20 client.26152 populate_metadata read hostname 'smithi007' 2024-03-18T01:41:46.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.887+0000 7fba967ca740 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x555a53e837e0 0x555a53e85c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:41:46.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.887+0000 7fba967ca740 4 client.26152 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:41:46.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.887+0000 7fba967ca740 1 -- 172.21.15.7:0/241625478 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555a53d70730 con 0x555a53e837e0 2024-03-18T01:41:46.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.888+0000 7fba91000640 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x555a53e837e0 0x555a53e85c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:41:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.889+0000 7fba91000640 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x555a53e837e0 0x555a53e85c00 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f1c70ac2be14957 server_cookie=326d46dc7edb509 in_seq=0 out_seq=0 2024-03-18T01:41:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.889+0000 7fba72a00640 10 client.26152 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:41:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.889+0000 7fba72a00640 1 -- 172.21.15.7:0/241625478 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x555a53d70730 con 0x555a53e837e0 2024-03-18T01:41:46.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.889+0000 7fba72a00640 10 client.26152 handle_command_reply: tid=0 2024-03-18T01:41:46.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.890+0000 7fba967ca740 1 client.26152 shutdown 2024-03-18T01:41:46.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.890+0000 7fba967ca740 1 -- 172.21.15.7:0/241625478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 msgr2=0x555a53e16280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.890+0000 7fba967ca740 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e16280 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fba78006010 tx=0x7fba7800ba70 comp rx=0 tx=0).stop 2024-03-18T01:41:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 1 -- 172.21.15.7:0/241625478 shutdown_connections 2024-03-18T01:41:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x555a53e837e0 0x555a53e85c00 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555a53e0cb80 0x555a53e17290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555a53e07a10 0x555a53e16280 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 1 --2- 172.21.15.7:0/241625478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555a53e07020 0x555a53e15d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba72a00640 0 client.26152 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:41:46.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 1 -- 172.21.15.7:0/241625478 >> 172.21.15.7:0/241625478 conn(0x555a53df9d80 msgr2=0x555a53e10080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:46.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 1 -- 172.21.15.7:0/241625478 shutdown_connections 2024-03-18T01:41:46.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 1 -- 172.21.15.7:0/241625478 wait complete. 2024-03-18T01:41:46.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.891+0000 7fba967ca740 20 client.26152 trim_cache size 0 max 16384 2024-03-18T01:41:46.892 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:41:46.892 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (16 inodes in the stack)", 2024-03-18T01:41:46.892 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:41:46.892 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:41:46.893 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:41:46.893 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:41:46.893 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:41:46.893 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:41:46.893 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:41:46.893 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 -- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba740a2880 msgr2=0x7fba740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba740a2880 0x7fba740a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fba8002c7c0 tx=0x7fba80005fd0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 -- 172.21.15.7:0/2574153398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c192940 msgr2=0x7fba8c194de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c192940 0x7fba8c194de0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fba84010c30 tx=0x7fba8400e430 comp rx=0 tx=0).stop 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 -- 172.21.15.7:0/2574153398 shutdown_connections 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba8c192940 0x7fba8c194de0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba8c1022b0 0x7fba8c070900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fba740a2880 0x7fba740a4d40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.894 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 --2- 172.21.15.7:0/2574153398 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fba8c1018f0 0x7fba8c0703c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:41:46.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 -- 172.21.15.7:0/2574153398 >> 172.21.15.7:0/2574153398 conn(0x7fba8c0fbc70 msgr2=0x7fba8c104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:41:46.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 -- 172.21.15.7:0/2574153398 shutdown_connections 2024-03-18T01:41:46.895 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:41:46.893+0000 7fba93a00640 1 -- 172.21.15.7:0/2574153398 wait complete. 2024-03-18T01:41:47.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:47 smithi203 ceph-mon[36345]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 53 MiB/s wr, 7k op/s 2024-03-18T01:41:47.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:47 smithi046 ceph-mon[36108]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 53 MiB/s wr, 7k op/s 2024-03-18T01:41:47.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:47 smithi007 ceph-mon[31996]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 53 MiB/s wr, 7k op/s 2024-03-18T01:41:48.922 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:41:49.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:49 smithi203 ceph-mon[36345]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 MiB/s wr, 8.09k op/s 2024-03-18T01:41:49.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:49 smithi046 ceph-mon[36108]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 MiB/s wr, 8.09k op/s 2024-03-18T01:41:49.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:49 smithi007 ceph-mon[31996]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 MiB/s wr, 8.09k op/s 2024-03-18T01:41:50.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:50 smithi046 ceph-mon[36108]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 MiB/s wr, 8.10k op/s 2024-03-18T01:41:50.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:50 smithi007 ceph-mon[31996]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 MiB/s wr, 8.10k op/s 2024-03-18T01:41:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:50 smithi203 ceph-mon[36345]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 MiB/s wr, 8.10k op/s 2024-03-18T01:41:53.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:52 smithi203 ceph-mon[36345]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 67 MiB/s wr, 7.71k op/s 2024-03-18T01:41:53.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:52 smithi046 ceph-mon[36108]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 67 MiB/s wr, 7.71k op/s 2024-03-18T01:41:53.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:52 smithi007 ceph-mon[31996]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 67 MiB/s wr, 7.71k op/s 2024-03-18T01:41:55.070 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:54 smithi046 ceph-mon[36108]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 MiB/s wr, 7.72k op/s 2024-03-18T01:41:55.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:54 smithi203 ceph-mon[36345]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 MiB/s wr, 7.72k op/s 2024-03-18T01:41:55.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:54 smithi007 ceph-mon[31996]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 MiB/s wr, 7.72k op/s 2024-03-18T01:41:57.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:56 smithi203 ceph-mon[36345]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 65 MiB/s wr, 7.39k op/s 2024-03-18T01:41:57.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:56 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:41:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:56 smithi046 ceph-mon[36108]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 65 MiB/s wr, 7.39k op/s 2024-03-18T01:41:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:56 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:41:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:56 smithi007 ceph-mon[31996]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 65 MiB/s wr, 7.39k op/s 2024-03-18T01:41:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:56 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:41:59.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:41:58 smithi203 ceph-mon[36345]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 71 MiB/s wr, 8.15k op/s 2024-03-18T01:41:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:41:58 smithi046 ceph-mon[36108]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 71 MiB/s wr, 8.15k op/s 2024-03-18T01:41:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:41:58 smithi007 ceph-mon[31996]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 71 MiB/s wr, 8.15k op/s 2024-03-18T01:42:01.125 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:00 smithi007 ceph-mon[31996]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 MiB/s wr, 7.46k op/s 2024-03-18T01:42:01.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:00 smithi203 ceph-mon[36345]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 MiB/s wr, 7.46k op/s 2024-03-18T01:42:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:00 smithi046 ceph-mon[36108]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 MiB/s wr, 7.46k op/s 2024-03-18T01:42:03.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:02 smithi203 ceph-mon[36345]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 68 MiB/s wr, 7.63k op/s 2024-03-18T01:42:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:02 smithi046 ceph-mon[36108]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 68 MiB/s wr, 7.63k op/s 2024-03-18T01:42:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:02 smithi007 ceph-mon[31996]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 68 MiB/s wr, 7.63k op/s 2024-03-18T01:42:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:04 smithi203 ceph-mon[36345]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 MiB/s wr, 7.41k op/s 2024-03-18T01:42:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:42:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:42:05.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:04 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:42:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:04 smithi046 ceph-mon[36108]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 MiB/s wr, 7.41k op/s 2024-03-18T01:42:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:42:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:42:05.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:04 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:42:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:04 smithi007 ceph-mon[31996]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 MiB/s wr, 7.41k op/s 2024-03-18T01:42:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:42:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:42:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:04 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:42:07.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:06 smithi203 ceph-mon[36345]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 65 MiB/s wr, 7.38k op/s 2024-03-18T01:42:07.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:06 smithi046 ceph-mon[36108]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 65 MiB/s wr, 7.38k op/s 2024-03-18T01:42:07.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:06 smithi007 ceph-mon[31996]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 65 MiB/s wr, 7.38k op/s 2024-03-18T01:42:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:08 smithi203 ceph-mon[36345]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 70 MiB/s wr, 7.94k op/s 2024-03-18T01:42:09.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:08 smithi046 ceph-mon[36108]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 70 MiB/s wr, 7.94k op/s 2024-03-18T01:42:09.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:08 smithi007 ceph-mon[31996]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 70 MiB/s wr, 7.94k op/s 2024-03-18T01:42:11.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:10 smithi046 ceph-mon[36108]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 66 MiB/s wr, 7.61k op/s 2024-03-18T01:42:11.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:10 smithi007 ceph-mon[31996]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 66 MiB/s wr, 7.61k op/s 2024-03-18T01:42:11.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:10 smithi203 ceph-mon[36345]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 66 MiB/s wr, 7.61k op/s 2024-03-18T01:42:13.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:12 smithi203 ceph-mon[36345]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 67 MiB/s wr, 8.03k op/s 2024-03-18T01:42:13.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:12 smithi046 ceph-mon[36108]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 67 MiB/s wr, 8.03k op/s 2024-03-18T01:42:13.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:12 smithi007 ceph-mon[31996]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 67 MiB/s wr, 8.03k op/s 2024-03-18T01:42:15.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:14 smithi046 ceph-mon[36108]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 MiB/s wr, 6.80k op/s 2024-03-18T01:42:15.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:14 smithi007 ceph-mon[31996]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 MiB/s wr, 6.80k op/s 2024-03-18T01:42:15.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:14 smithi203 ceph-mon[36345]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 55 MiB/s wr, 6.80k op/s 2024-03-18T01:42:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:16 smithi046 ceph-mon[36108]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 MiB/s wr, 7.15k op/s 2024-03-18T01:42:17.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:16 smithi007 ceph-mon[31996]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 MiB/s wr, 7.15k op/s 2024-03-18T01:42:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:16 smithi203 ceph-mon[36345]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 57 MiB/s wr, 7.15k op/s 2024-03-18T01:42:18.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:42:18.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:42:19.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:18 smithi007 ceph-mon[31996]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 MiB/s wr, 7.48k op/s 2024-03-18T01:42:19.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:18 smithi046 ceph-mon[36108]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 MiB/s wr, 7.48k op/s 2024-03-18T01:42:19.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:18 smithi203 ceph-mon[36345]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 MiB/s wr, 7.48k op/s 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.266+0000 7fc170e00640 1 -- 172.21.15.7:0/2334458872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 msgr2=0x7fc16c101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.266+0000 7fc170e00640 1 --2- 172.21.15.7:0/2334458872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 0x7fc16c101c10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc15c009900 tx=0x7fc15c02bc50 comp rx=0 tx=0).stop 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.266+0000 7fc170e00640 1 -- 172.21.15.7:0/2334458872 shutdown_connections 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.266+0000 7fc170e00640 1 --2- 172.21.15.7:0/2334458872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc16c1033a0 0x7fc16c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.266+0000 7fc170e00640 1 --2- 172.21.15.7:0/2334458872 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c1029e0 0x7fc16c102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.266+0000 7fc170e00640 1 --2- 172.21.15.7:0/2334458872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 0x7fc16c101c10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.266+0000 7fc170e00640 1 -- 172.21.15.7:0/2334458872 >> 172.21.15.7:0/2334458872 conn(0x7fc16c0fcd60 msgr2=0x7fc16c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.267+0000 7fc170e00640 1 -- 172.21.15.7:0/2334458872 shutdown_connections 2024-03-18T01:42:20.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.267+0000 7fc170e00640 1 -- 172.21.15.7:0/2334458872 wait complete. 2024-03-18T01:42:20.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.267+0000 7fc170e00640 1 Processor -- start 2024-03-18T01:42:20.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.268+0000 7fc170e00640 1 -- start start 2024-03-18T01:42:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc170e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 0x7fc16c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc170e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c1029e0 0x7fc16c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc170e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc16c1033a0 0x7fc16c1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc170e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c10d510 con 0x7fc16c1029e0 2024-03-18T01:42:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc170e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c10d390 con 0x7fc16c1017f0 2024-03-18T01:42:20.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc170e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c10d690 con 0x7fc16c1033a0 2024-03-18T01:42:20.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc169e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 0x7fc16c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc169e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 0x7fc16c197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:51244/0 (socket says 172.21.15.7:51244) 2024-03-18T01:42:20.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.270+0000 7fc169e00640 1 -- 172.21.15.7:0/3328898856 learned_addr learned my addr 172.21.15.7:0/3328898856 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:20.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.271+0000 7fc16a800640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc16c1033a0 0x7fc16c1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.271+0000 7fc169e00640 1 -- 172.21.15.7:0/3328898856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc16c1033a0 msgr2=0x7fc16c1a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.271+0000 7fc169e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc16c1033a0 0x7fc16c1a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.271+0000 7fc163e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c1029e0 0x7fc16c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.271+0000 7fc169e00640 1 -- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c1029e0 msgr2=0x7fc16c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.271+0000 7fc169e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c1029e0 0x7fc16c1980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.271+0000 7fc169e00640 1 -- 172.21.15.7:0/3328898856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c19d150 con 0x7fc16c1017f0 2024-03-18T01:42:20.274 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.272+0000 7fc169e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 0x7fc16c197bb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc15c000c00 tx=0x7fc15c005660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.276+0000 7fc162a00640 1 -- 172.21.15.7:0/3328898856 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c041070 con 0x7fc16c1017f0 2024-03-18T01:42:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.276+0000 7fc162a00640 1 -- 172.21.15.7:0/3328898856 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc15c03d430 con 0x7fc16c1017f0 2024-03-18T01:42:20.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.276+0000 7fc162a00640 1 -- 172.21.15.7:0/3328898856 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c034690 con 0x7fc16c1017f0 2024-03-18T01:42:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.277+0000 7fc170e00640 1 -- 172.21.15.7:0/3328898856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc16c19d3e0 con 0x7fc16c1017f0 2024-03-18T01:42:20.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.277+0000 7fc170e00640 1 -- 172.21.15.7:0/3328898856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc16c19d900 con 0x7fc16c1017f0 2024-03-18T01:42:20.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.282+0000 7fc173b64740 1 Processor -- start 2024-03-18T01:42:20.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.282+0000 7fc173b64740 1 -- start start 2024-03-18T01:42:20.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.282+0000 7fc173b64740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 0x559690e20c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.283+0000 7fc173b64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e25d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.283+0000 7fc173b64740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e28800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.283+0000 7fc173b64740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559690d851f0 con 0x559690e20810 2024-03-18T01:42:20.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.283+0000 7fc173b64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559690d2da80 con 0x559690e21200 2024-03-18T01:42:20.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.283+0000 7fc173b64740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559690d81290 con 0x559690e26370 2024-03-18T01:42:20.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.283+0000 7fc163e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e28800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.283+0000 7fc163e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e28800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44956/0 (socket says 172.21.15.7:44956) 2024-03-18T01:42:20.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.283+0000 7fc163e00640 1 -- 172.21.15.7:0/2875868439 learned_addr learned my addr 172.21.15.7:0/2875868439 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:20.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.284+0000 7fc162a00640 1 -- 172.21.15.7:0/3328898856 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fc15c044070 con 0x7fc16c1017f0 2024-03-18T01:42:20.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.284+0000 7fc169e00640 1 --2- 172.21.15.7:0/2875868439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e25d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.285+0000 7fc169e00640 1 -- 172.21.15.7:0/2875868439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 msgr2=0x559690e28800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:20.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.285+0000 7fc169e00640 1 --2- 172.21.15.7:0/2875868439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e28800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.285+0000 7fc169e00640 1 -- 172.21.15.7:0/2875868439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 msgr2=0x559690e20c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:20.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.285+0000 7fc169e00640 1 --2- 172.21.15.7:0/2875868439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 0x559690e20c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.285+0000 7fc169e00640 1 -- 172.21.15.7:0/2875868439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559690b637d0 con 0x559690e21200 2024-03-18T01:42:20.288 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.286+0000 7fc169e00640 1 --2- 172.21.15.7:0/2875868439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e25d00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0ede80 tx=0x7fc15c0f6ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b61a73d42d0e925 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:20.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.287+0000 7fc15b400640 1 -- 172.21.15.7:0/2875868439 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0ff070 con 0x559690e21200 2024-03-18T01:42:20.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.287+0000 7fc15b400640 1 -- 172.21.15.7:0/2875868439 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc15c0f20d0 con 0x559690e21200 2024-03-18T01:42:20.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.287+0000 7fc15b400640 1 -- 172.21.15.7:0/2875868439 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0fb680 con 0x559690e21200 2024-03-18T01:42:20.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.288+0000 7fc173b64740 1 -- 172.21.15.7:0/2875868439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 msgr2=0x559690e25d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.288+0000 7fc173b64740 1 --2- 172.21.15.7:0/2875868439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e25d00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0ede80 tx=0x7fc15c0f6ef0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.289+0000 7fc173b64740 1 -- 172.21.15.7:0/2875868439 shutdown_connections 2024-03-18T01:42:20.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.289+0000 7fc173b64740 1 --2- 172.21.15.7:0/2875868439 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e28800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.289+0000 7fc173b64740 1 --2- 172.21.15.7:0/2875868439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e25d00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.289+0000 7fc173b64740 1 --2- 172.21.15.7:0/2875868439 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 0x559690e20c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.289+0000 7fc173b64740 1 -- 172.21.15.7:0/2875868439 >> 172.21.15.7:0/2875868439 conn(0x559690e1bd80 msgr2=0x559690e1c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:20.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.289+0000 7fc162a00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1500a2950 0x7fc1500a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.289+0000 7fc162a00640 1 -- 172.21.15.7:0/3328898856 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc15c0f07a0 con 0x7fc16c1017f0 2024-03-18T01:42:20.291 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.289+0000 7fc173b64740 1 -- 172.21.15.7:0/2875868439 shutdown_connections 2024-03-18T01:42:20.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.290+0000 7fc173b64740 1 -- 172.21.15.7:0/2875868439 wait complete. 2024-03-18T01:42:20.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.290+0000 7fc173b64740 1 -- start start 2024-03-18T01:42:20.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.292+0000 7fc163e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1500a2950 0x7fc1500a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.293 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.292+0000 7fc173b64740 1 Processor -- start 2024-03-18T01:42:20.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc173b64740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e20810 0x559690e2ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc173b64740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e21200 0x559690e2b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc173b64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e26370 0x559690e32080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.294 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc173b64740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559690e290e0 con 0x559690e21200 2024-03-18T01:42:20.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc173b64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559690d40ed0 con 0x559690e26370 2024-03-18T01:42:20.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc173b64740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559690e29260 con 0x559690e20810 2024-03-18T01:42:20.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc173b64740 10 client.0 _ll_register_callbacks cb 0x559690d7c220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:42:20.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc169e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e20810 0x559690e2ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc169e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e20810 0x559690e2ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44966/0 (socket says 172.21.15.7:44966) 2024-03-18T01:42:20.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc169e00640 1 -- 172.21.15.7:0/802854802 learned_addr learned my addr 172.21.15.7:0/802854802 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:20.295 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.293+0000 7fc163e00640 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e21200 0x559690e2b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc169e00640 1 -- 172.21.15.7:0/802854802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e26370 msgr2=0x559690e32080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc169e00640 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e26370 0x559690e32080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc169e00640 1 -- 172.21.15.7:0/802854802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e21200 msgr2=0x559690e2b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc169e00640 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e21200 0x559690e2b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc169e00640 1 -- 172.21.15.7:0/802854802 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559690a993f0 con 0x559690e20810 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc169e00640 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e20810 0x559690e2ab00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0f62d0 tx=0x7fc15c02bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24bf6ac912fa6218 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc14fe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc14fe00640 1 -- 172.21.15.7:0/802854802 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0ff070 con 0x559690e20810 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.294+0000 7fc173b64740 1 -- 172.21.15.7:0/802854802 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc15c0f20d0 con 0x559690e20810 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.295+0000 7fc14fe00640 1 -- 172.21.15.7:0/802854802 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc15c0edff0 con 0x559690e20810 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.295+0000 7fc14fe00640 1 -- 172.21.15.7:0/802854802 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc15c0b9020 con 0x559690e20810 2024-03-18T01:42:20.296 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.295+0000 7fc173b64740 10 client.26161 fetch_fsmap learned FSMap version 31 2024-03-18T01:42:20.297 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.295+0000 7fc173b64740 1 -- 172.21.15.7:0/802854802 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559690a590f0 con 0x559690e20810 2024-03-18T01:42:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.295+0000 7fc14fe00640 1 -- 172.21.15.7:0/802854802 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc15c0f20d0 con 0x559690e20810 2024-03-18T01:42:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.296+0000 7fc14fe00640 1 -- 172.21.15.7:0/802854802 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fc15c0d5020 con 0x559690e20810 2024-03-18T01:42:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.296+0000 7fc173b64740 10 client.26161 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:42:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.296+0000 7fc173b64740 10 client.26161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:42:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.296+0000 7fc173b64740 20 client.26161 populate_metadata read hostname 'smithi007' 2024-03-18T01:42:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.296+0000 7fc173b64740 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559690e366d0 0x559690e38b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.296+0000 7fc173b64740 4 client.26161 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:42:20.299 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.296+0000 7fc173b64740 1 -- 172.21.15.7:0/802854802 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559690d98be0 con 0x559690e366d0 2024-03-18T01:42:20.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.300+0000 7fc163e00640 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559690e366d0 0x559690e38b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.300+0000 7fc163e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1500a2950 0x7fc1500a4e10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x559690e2bfe0 tx=0x7fc154009b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:20.301 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.300+0000 7fc163e00640 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559690e366d0 0x559690e38b50 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e52a8689014d74c server_cookie=2ce435ccc17fe3f5 in_seq=0 out_seq=0 2024-03-18T01:42:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.300+0000 7fc14fe00640 10 client.26161 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:42:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc14fe00640 1 -- 172.21.15.7:0/802854802 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559690d98be0 con 0x559690e366d0 2024-03-18T01:42:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc14fe00640 10 client.26161 handle_command_reply: tid=0 2024-03-18T01:42:20.303 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 client.26161 shutdown 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 -- 172.21.15.7:0/802854802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e20810 msgr2=0x559690e2ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e20810 0x559690e2ab00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc15c0f62d0 tx=0x7fc15c02bc50 comp rx=0 tx=0).stop 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 -- 172.21.15.7:0/802854802 shutdown_connections 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559690e366d0 0x559690e38b50 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e26370 0x559690e32080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e21200 0x559690e2b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 --2- 172.21.15.7:0/802854802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e20810 0x559690e2ab00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.302+0000 7fc173b64740 1 -- 172.21.15.7:0/802854802 >> 172.21.15.7:0/802854802 conn(0x559690e1bd80 msgr2=0x559690e1aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.303+0000 7fc14fe00640 0 client.26161 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:42:20.304 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.303+0000 7fc173b64740 1 -- 172.21.15.7:0/802854802 shutdown_connections 2024-03-18T01:42:20.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.304+0000 7fc173b64740 1 -- 172.21.15.7:0/802854802 wait complete. 2024-03-18T01:42:20.305 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.304+0000 7fc173b64740 20 client.26161 trim_cache size 0 max 16384 2024-03-18T01:42:20.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.367+0000 7fc173b64740 1 Processor -- start 2024-03-18T01:42:20.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.367+0000 7fc173b64740 1 -- start start 2024-03-18T01:42:20.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.367+0000 7fc173b64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e20810 0x559690e2ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.367+0000 7fc173b64740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e21200 0x559690e32040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.367+0000 7fc173b64740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e26370 0x559690e1a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.367+0000 7fc173b64740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c1b09d0 con 0x559690e26370 2024-03-18T01:42:20.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.368+0000 7fc173b64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c1b25d0 con 0x559690e20810 2024-03-18T01:42:20.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.368+0000 7fc173b64740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c1b10a0 con 0x559690e21200 2024-03-18T01:42:20.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.368+0000 7fc163e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e21200 0x559690e32040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.368+0000 7fc163e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e21200 0x559690e32040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44970/0 (socket says 172.21.15.7:44970) 2024-03-18T01:42:20.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.368+0000 7fc163e00640 1 -- 172.21.15.7:0/3785079731 learned_addr learned my addr 172.21.15.7:0/3785079731 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:20.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc169e00640 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e20810 0x559690e2ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc16a800640 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e26370 0x559690e1a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc163e00640 1 -- 172.21.15.7:0/3785079731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e20810 msgr2=0x559690e2ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc163e00640 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e20810 0x559690e2ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc163e00640 1 -- 172.21.15.7:0/3785079731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e26370 msgr2=0x559690e1a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc163e00640 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e26370 0x559690e1a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc163e00640 1 -- 172.21.15.7:0/3785079731 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559690e1caf0 con 0x559690e21200 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc169e00640 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e20810 0x559690e2ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc16a800640 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e26370 0x559690e1a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.369+0000 7fc163e00640 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e21200 0x559690e32040 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc1540058c0 tx=0x7fc154015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=873589137aa0f05f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.370+0000 7fc15aa00640 1 -- 172.21.15.7:0/3785079731 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154050070 con 0x559690e21200 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.370+0000 7fc15aa00640 1 -- 172.21.15.7:0/3785079731 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc15401a070 con 0x559690e21200 2024-03-18T01:42:20.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.370+0000 7fc15aa00640 1 -- 172.21.15.7:0/3785079731 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154050070 con 0x559690e21200 2024-03-18T01:42:20.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.371+0000 7fc173b64740 1 -- 172.21.15.7:0/3785079731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e21200 msgr2=0x559690e32040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.371+0000 7fc173b64740 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e21200 0x559690e32040 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc1540058c0 tx=0x7fc154015040 comp rx=0 tx=0).stop 2024-03-18T01:42:20.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.372+0000 7fc173b64740 1 -- 172.21.15.7:0/3785079731 shutdown_connections 2024-03-18T01:42:20.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.372+0000 7fc173b64740 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e26370 0x559690e1a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.372+0000 7fc173b64740 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e21200 0x559690e32040 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.372+0000 7fc173b64740 1 --2- 172.21.15.7:0/3785079731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e20810 0x559690e2ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.372+0000 7fc173b64740 1 -- 172.21.15.7:0/3785079731 >> 172.21.15.7:0/3785079731 conn(0x559690e1bd80 msgr2=0x559690e2a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:20.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.372+0000 7fc173b64740 1 -- 172.21.15.7:0/3785079731 shutdown_connections 2024-03-18T01:42:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.372+0000 7fc173b64740 1 -- 172.21.15.7:0/3785079731 wait complete. 2024-03-18T01:42:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.375+0000 7fc173b64740 1 -- start start 2024-03-18T01:42:20.377 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.375+0000 7fc173b64740 1 Processor -- start 2024-03-18T01:42:20.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.377+0000 7fc173b64740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 0x559690e2fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.378 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.377+0000 7fc173b64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e2ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.377+0000 7fc173b64740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e30f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.377+0000 7fc169e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 0x559690e2fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc173b64740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c1b1e40 con 0x559690e20810 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc173b64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c1b1770 con 0x559690e21200 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc173b64740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c1ae9b0 con 0x559690e26370 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc173b64740 10 client.0 _ll_register_callbacks cb 0x7fc154009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc16a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e30f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc16a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e30f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44978/0 (socket says 172.21.15.7:44978) 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc16a800640 1 -- 172.21.15.7:0/2405378132 learned_addr learned my addr 172.21.15.7:0/2405378132 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc163e00640 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e2ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.378+0000 7fc16a800640 1 -- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 msgr2=0x559690e2ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.379+0000 7fc16a800640 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e2ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.379+0000 7fc16a800640 1 -- 172.21.15.7:0/2405378132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 msgr2=0x559690e2fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.379+0000 7fc16a800640 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 0x559690e2fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.379+0000 7fc16a800640 1 -- 172.21.15.7:0/2405378132 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559690e31580 con 0x559690e26370 2024-03-18T01:42:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.379+0000 7fc163e00640 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e2ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:42:20.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.379+0000 7fc169e00640 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 0x559690e2fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:42:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.379+0000 7fc16a800640 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e30f90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc1640099d0 tx=0x7fc164009e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5761b44e41af845 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.379+0000 7fc14fe00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:42:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.380+0000 7fc14fe00640 1 -- 172.21.15.7:0/2405378132 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16401a060 con 0x559690e26370 2024-03-18T01:42:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.380+0000 7fc173b64740 1 -- 172.21.15.7:0/2405378132 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc15401a070 con 0x559690e26370 2024-03-18T01:42:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.380+0000 7fc14fe00640 1 -- 172.21.15.7:0/2405378132 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc164007c40 con 0x559690e26370 2024-03-18T01:42:20.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.381+0000 7fc14fe00640 1 -- 172.21.15.7:0/2405378132 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc164021050 con 0x559690e26370 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.381+0000 7fc173b64740 10 client.26167 fetch_fsmap learned FSMap version 31 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.381+0000 7fc173b64740 1 -- 172.21.15.7:0/2405378132 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559690e9ce80 con 0x559690e26370 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.381+0000 7fc14fe00640 1 -- 172.21.15.7:0/2405378132 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc15401a070 con 0x559690e26370 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.382+0000 7fc14fe00640 1 -- 172.21.15.7:0/2405378132 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fc164013320 con 0x559690e26370 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.382+0000 7fc173b64740 10 client.26167 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.382+0000 7fc173b64740 10 client.26167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.382+0000 7fc173b64740 20 client.26167 populate_metadata read hostname 'smithi007' 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.382+0000 7fc173b64740 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559690e9d920 0x559690e9fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:20.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.382+0000 7fc173b64740 4 client.26167 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:42:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.382+0000 7fc173b64740 1 -- 172.21.15.7:0/2405378132 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559690d92730 con 0x559690e9d920 2024-03-18T01:42:20.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.382+0000 7fc169e00640 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559690e9d920 0x559690e9fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.387+0000 7fc169e00640 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559690e9d920 0x559690e9fd40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b42b88965b7d4501 server_cookie=4969c06d560ab868 in_seq=0 out_seq=0 2024-03-18T01:42:20.388 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.387+0000 7fc14fe00640 10 client.26167 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:42:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.387+0000 7fc14fe00640 1 -- 172.21.15.7:0/2405378132 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x559690d92730 con 0x559690e9d920 2024-03-18T01:42:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.387+0000 7fc14fe00640 10 client.26167 handle_command_reply: tid=0 2024-03-18T01:42:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.387+0000 7fc173b64740 1 client.26167 shutdown 2024-03-18T01:42:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc173b64740 1 -- 172.21.15.7:0/2405378132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 msgr2=0x559690e30f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc173b64740 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e30f90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc1640099d0 tx=0x7fc164009e90 comp rx=0 tx=0).stop 2024-03-18T01:42:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc173b64740 1 -- 172.21.15.7:0/2405378132 shutdown_connections 2024-03-18T01:42:20.389 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc173b64740 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559690e9d920 0x559690e9fd40 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc173b64740 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559690e26370 0x559690e30f90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc173b64740 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559690e21200 0x559690e2ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc14fe00640 0 client.26167 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:42:20.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc173b64740 1 --2- 172.21.15.7:0/2405378132 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559690e20810 0x559690e2fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.390 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.388+0000 7fc173b64740 1 -- 172.21.15.7:0/2405378132 >> 172.21.15.7:0/2405378132 conn(0x559690e1bd80 msgr2=0x559690e1a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:20.394 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.393+0000 7fc173b64740 1 -- 172.21.15.7:0/2405378132 shutdown_connections 2024-03-18T01:42:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.394+0000 7fc173b64740 1 -- 172.21.15.7:0/2405378132 wait complete. 2024-03-18T01:42:20.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.394+0000 7fc173b64740 20 client.26167 trim_cache size 0 max 16384 2024-03-18T01:42:20.396 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:42:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (16 inodes in the stack)", 2024-03-18T01:42:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:42:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:42:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:42:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:42:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:42:20.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:42:20.397 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:42:20.397 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:42:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.395+0000 7fc170e00640 1 -- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1500a2950 msgr2=0x7fc1500a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.395+0000 7fc170e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1500a2950 0x7fc1500a4e10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x559690e2bfe0 tx=0x7fc154009b50 comp rx=0 tx=0).stop 2024-03-18T01:42:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 -- 172.21.15.7:0/3328898856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 msgr2=0x7fc16c197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 0x7fc16c197bb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc15c000c00 tx=0x7fc15c005660 comp rx=0 tx=0).stop 2024-03-18T01:42:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 -- 172.21.15.7:0/3328898856 shutdown_connections 2024-03-18T01:42:20.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc16c1033a0 0x7fc16c1a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc1500a2950 0x7fc1500a4e10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc16c1029e0 0x7fc16c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 --2- 172.21.15.7:0/3328898856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc16c1017f0 0x7fc16c197bb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 -- 172.21.15.7:0/3328898856 >> 172.21.15.7:0/3328898856 conn(0x7fc16c0fcd60 msgr2=0x7fc16c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 -- 172.21.15.7:0/3328898856 shutdown_connections 2024-03-18T01:42:20.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:20.396+0000 7fc170e00640 1 -- 172.21.15.7:0/3328898856 wait complete. 2024-03-18T01:42:20.434 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:42:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:20 smithi046 ceph-mon[36108]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 52 MiB/s wr, 7.07k op/s 2024-03-18T01:42:21.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:20 smithi007 ceph-mon[31996]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 52 MiB/s wr, 7.07k op/s 2024-03-18T01:42:21.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:20 smithi203 ceph-mon[36345]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 52 MiB/s wr, 7.07k op/s 2024-03-18T01:42:23.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:22 smithi046 ceph-mon[36108]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 MiB/s wr, 7.66k op/s 2024-03-18T01:42:23.304 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:22 smithi007 ceph-mon[31996]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 MiB/s wr, 7.66k op/s 2024-03-18T01:42:23.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:22 smithi203 ceph-mon[36345]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 55 MiB/s wr, 7.66k op/s 2024-03-18T01:42:25.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:24 smithi046 ceph-mon[36108]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 MiB/s wr, 6.46k op/s 2024-03-18T01:42:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:24 smithi007 ceph-mon[31996]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 MiB/s wr, 6.46k op/s 2024-03-18T01:42:25.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:24 smithi203 ceph-mon[36345]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 MiB/s wr, 6.46k op/s 2024-03-18T01:42:27.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:26 smithi046 ceph-mon[36108]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 49 MiB/s wr, 6.95k op/s 2024-03-18T01:42:27.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:26 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:27.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:26 smithi007 ceph-mon[31996]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 49 MiB/s wr, 6.95k op/s 2024-03-18T01:42:27.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:27 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:27.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:27 smithi203 ceph-mon[36345]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 49 MiB/s wr, 6.95k op/s 2024-03-18T01:42:27.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:27 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:29.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:29 smithi046 ceph-mon[36108]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 MiB/s wr, 6.86k op/s 2024-03-18T01:42:29.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:29 smithi007 ceph-mon[31996]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 MiB/s wr, 6.86k op/s 2024-03-18T01:42:29.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:29 smithi203 ceph-mon[36345]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 MiB/s wr, 6.86k op/s 2024-03-18T01:42:31.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:31 smithi007 ceph-mon[31996]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 MiB/s wr, 6.15k op/s 2024-03-18T01:42:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:31 smithi203 ceph-mon[36345]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 MiB/s wr, 6.15k op/s 2024-03-18T01:42:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:31 smithi046 ceph-mon[36108]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 43 MiB/s wr, 6.15k op/s 2024-03-18T01:42:32.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:32 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:32.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:32 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:32.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:32 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:33.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:33 smithi203 ceph-mon[36345]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 49 MiB/s wr, 7.18k op/s 2024-03-18T01:42:33.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:33 smithi046 ceph-mon[36108]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 49 MiB/s wr, 7.18k op/s 2024-03-18T01:42:33.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:33 smithi007 ceph-mon[31996]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 49 MiB/s wr, 7.18k op/s 2024-03-18T01:42:35.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:35 smithi203 ceph-mon[36345]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 40 MiB/s wr, 5.91k op/s 2024-03-18T01:42:35.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:35 smithi046 ceph-mon[36108]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 40 MiB/s wr, 5.91k op/s 2024-03-18T01:42:35.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:35 smithi007 ceph-mon[31996]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 40 MiB/s wr, 5.91k op/s 2024-03-18T01:42:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:37 smithi203 ceph-mon[36345]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 45 MiB/s wr, 6.71k op/s 2024-03-18T01:42:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:37 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:37.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:37 smithi046 ceph-mon[36108]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 45 MiB/s wr, 6.71k op/s 2024-03-18T01:42:37.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:37 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:37.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:37 smithi007 ceph-mon[31996]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 45 MiB/s wr, 6.71k op/s 2024-03-18T01:42:37.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:37 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:39 smithi203 ceph-mon[36345]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 MiB/s wr, 6.59k op/s 2024-03-18T01:42:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:39 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:39 smithi046 ceph-mon[36108]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 MiB/s wr, 6.59k op/s 2024-03-18T01:42:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:39 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:39 smithi007 ceph-mon[31996]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 43 MiB/s wr, 6.59k op/s 2024-03-18T01:42:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:39 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:42:41.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:41 smithi203 ceph-mon[36345]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 5.96k op/s 2024-03-18T01:42:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:41 smithi046 ceph-mon[36108]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 5.96k op/s 2024-03-18T01:42:41.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:41 smithi007 ceph-mon[31996]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 39 MiB/s wr, 5.96k op/s 2024-03-18T01:42:43.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:43 smithi203 ceph-mon[36345]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 MiB/s wr, 7.46k op/s 2024-03-18T01:42:43.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:43 smithi046 ceph-mon[36108]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 MiB/s wr, 7.46k op/s 2024-03-18T01:42:43.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:43 smithi007 ceph-mon[31996]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 49 MiB/s wr, 7.46k op/s 2024-03-18T01:42:45.393 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:45 smithi203 ceph-mon[36345]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 5.74k op/s 2024-03-18T01:42:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:45 smithi046 ceph-mon[36108]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 5.74k op/s 2024-03-18T01:42:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:45 smithi007 ceph-mon[31996]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 MiB/s wr, 5.74k op/s 2024-03-18T01:42:47.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:47 smithi203 ceph-mon[36345]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 MiB/s wr, 5.61k op/s 2024-03-18T01:42:47.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:47 smithi046 ceph-mon[36108]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 MiB/s wr, 5.61k op/s 2024-03-18T01:42:47.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:47 smithi007 ceph-mon[31996]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 MiB/s wr, 5.61k op/s 2024-03-18T01:42:48.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:48 smithi046 ceph-mon[36108]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 MiB/s wr, 6.40k op/s 2024-03-18T01:42:48.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:48 smithi007 ceph-mon[31996]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 MiB/s wr, 6.40k op/s 2024-03-18T01:42:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:48 smithi203 ceph-mon[36345]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 MiB/s wr, 6.40k op/s 2024-03-18T01:42:50.434 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:42:50.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:42:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:50 smithi046 ceph-mon[36108]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 38 MiB/s wr, 5.83k op/s 2024-03-18T01:42:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:50 smithi007 ceph-mon[31996]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 38 MiB/s wr, 5.83k op/s 2024-03-18T01:42:50.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:50 smithi203 ceph-mon[36345]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 38 MiB/s wr, 5.83k op/s 2024-03-18T01:42:51.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.866+0000 7f5766e00640 1 -- 172.21.15.7:0/1620612350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758095d70 msgr2=0x7f5758096190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:51.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.866+0000 7f5766e00640 1 --2- 172.21.15.7:0/1620612350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758095d70 0x7f5758096190 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5754009900 tx=0x7f575402bc30 comp rx=0 tx=0).stop 2024-03-18T01:42:51.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.869+0000 7f5766e00640 1 -- 172.21.15.7:0/1620612350 shutdown_connections 2024-03-18T01:42:51.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.869+0000 7f5766e00640 1 --2- 172.21.15.7:0/1620612350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5758097920 0x7f575809e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.869+0000 7f5766e00640 1 --2- 172.21.15.7:0/1620612350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5758096f60 0x7f57580973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.869+0000 7f5766e00640 1 --2- 172.21.15.7:0/1620612350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758095d70 0x7f5758096190 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.869+0000 7f5766e00640 1 -- 172.21.15.7:0/1620612350 >> 172.21.15.7:0/1620612350 conn(0x7f57580912e0 msgr2=0x7f5758093720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:51.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.869+0000 7f5766e00640 1 -- 172.21.15.7:0/1620612350 shutdown_connections 2024-03-18T01:42:51.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.869+0000 7f5766e00640 1 -- 172.21.15.7:0/1620612350 wait complete. 2024-03-18T01:42:51.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.871+0000 7f5766e00640 1 Processor -- start 2024-03-18T01:42:51.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.878+0000 7f5766e00640 1 -- start start 2024-03-18T01:42:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.878+0000 7f5766e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5758095d70 0x7f575812e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.878+0000 7f5766e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5758096f60 0x7f575812e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.878+0000 7f5766e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758097920 0x7f5758135830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.879+0000 7f5766400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758097920 0x7f5758135830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.879+0000 7f5765000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5758096f60 0x7f575812e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.880+0000 7f5766400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758097920 0x7f5758135830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:48568/0 (socket says 172.21.15.7:48568) 2024-03-18T01:42:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.880+0000 7f5766400640 1 -- 172.21.15.7:0/2784267195 learned_addr learned my addr 172.21.15.7:0/2784267195 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.880+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f57580a1a30 con 0x7f5758095d70 2024-03-18T01:42:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.880+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f57580a18b0 con 0x7f5758097920 2024-03-18T01:42:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.880+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f57580a1bb0 con 0x7f5758096f60 2024-03-18T01:42:51.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.880+0000 7f5765a00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5758095d70 0x7f575812e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.881+0000 7f5765000640 1 -- 172.21.15.7:0/2784267195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758097920 msgr2=0x7f5758135830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.882+0000 7f5765000640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758097920 0x7f5758135830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.882+0000 7f5765000640 1 -- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5758095d70 msgr2=0x7f575812e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.882+0000 7f5765000640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5758095d70 0x7f575812e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.882+0000 7f5765000640 1 -- 172.21.15.7:0/2784267195 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5758135ea0 con 0x7f5758096f60 2024-03-18T01:42:51.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.887+0000 7f5765000640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5758096f60 0x7f575812e7c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f574c00e9f0 tx=0x7f574c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.888+0000 7f575e000640 1 -- 172.21.15.7:0/2784267195 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c00ccc0 con 0x7f5758096f60 2024-03-18T01:42:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.888+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5758136190 con 0x7f5758096f60 2024-03-18T01:42:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.889+0000 7f575e000640 1 -- 172.21.15.7:0/2784267195 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f574c005020 con 0x7f5758096f60 2024-03-18T01:42:51.889 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.889+0000 7f575e000640 1 -- 172.21.15.7:0/2784267195 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574c004b20 con 0x7f5758096f60 2024-03-18T01:42:51.900 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.890+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57581366e0 con 0x7f5758096f60 2024-03-18T01:42:51.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.916+0000 7f575e000640 1 -- 172.21.15.7:0/2784267195 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f574c004cc0 con 0x7f5758096f60 2024-03-18T01:42:51.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.917+0000 7f576b07e740 1 Processor -- start 2024-03-18T01:42:51.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.918+0000 7f576b07e740 1 -- start start 2024-03-18T01:42:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.920+0000 7f576b07e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf19c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.920+0000 7f576b07e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19cc10 0x5555bf1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.920+0000 7f576b07e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf1a1d80 0x5555bf1a4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.920+0000 7f576b07e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5555bf0f81f0 con 0x5555bf19cc10 2024-03-18T01:42:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.920+0000 7f576b07e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5555bf0a0a80 con 0x5555bf19c220 2024-03-18T01:42:51.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.920+0000 7f576b07e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5555bf0f4290 con 0x5555bf1a1d80 2024-03-18T01:42:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.920+0000 7f5765a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19cc10 0x5555bf1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.921+0000 7f5765a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19cc10 0x5555bf1a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:56704/0 (socket says 172.21.15.7:56704) 2024-03-18T01:42:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.921+0000 7f5766400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.921+0000 7f5766400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf19c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:48582/0 (socket says 172.21.15.7:48582) 2024-03-18T01:42:51.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.921+0000 7f5766400640 1 -- 172.21.15.7:0/279371587 learned_addr learned my addr 172.21.15.7:0/279371587 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.921+0000 7f5765000640 1 --2- 172.21.15.7:0/279371587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf1a1d80 0x5555bf1a4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.922+0000 7f5766400640 1 -- 172.21.15.7:0/279371587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf1a1d80 msgr2=0x5555bf1a4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.922+0000 7f5766400640 1 --2- 172.21.15.7:0/279371587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf1a1d80 0x5555bf1a4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.922+0000 7f5766400640 1 -- 172.21.15.7:0/279371587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19cc10 msgr2=0x5555bf1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.922+0000 7f5766400640 1 --2- 172.21.15.7:0/279371587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19cc10 0x5555bf1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.922+0000 7f5766400640 1 -- 172.21.15.7:0/279371587 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555beed67d0 con 0x5555bf19c220 2024-03-18T01:42:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.923+0000 7f5766400640 1 --2- 172.21.15.7:0/279371587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf19c640 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5760064cd0 tx=0x7f5760069420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d422bb6717fb7fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.923+0000 7f575e000640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57480a2880 0x7f57480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.923+0000 7f5752a00640 1 -- 172.21.15.7:0/279371587 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760086070 con 0x5555bf19c220 2024-03-18T01:42:51.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.925+0000 7f575e000640 1 -- 172.21.15.7:0/2784267195 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f574c0041e0 con 0x7f5758096f60 2024-03-18T01:42:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.925+0000 7f5752a00640 1 -- 172.21.15.7:0/279371587 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5760081950 con 0x5555bf19c220 2024-03-18T01:42:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.926+0000 7f5765a00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57480a2880 0x7f57480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.926+0000 7f576b07e740 1 -- 172.21.15.7:0/279371587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 msgr2=0x5555bf19c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:51.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.926+0000 7f576b07e740 1 --2- 172.21.15.7:0/279371587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf19c640 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5760064cd0 tx=0x7f5760069420 comp rx=0 tx=0).stop 2024-03-18T01:42:51.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.928+0000 7f576b07e740 1 -- 172.21.15.7:0/279371587 shutdown_connections 2024-03-18T01:42:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.928+0000 7f576b07e740 1 --2- 172.21.15.7:0/279371587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf1a1d80 0x5555bf1a4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.928+0000 7f576b07e740 1 --2- 172.21.15.7:0/279371587 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19cc10 0x5555bf1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.928+0000 7f576b07e740 1 --2- 172.21.15.7:0/279371587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf19c640 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.928+0000 7f576b07e740 1 -- 172.21.15.7:0/279371587 >> 172.21.15.7:0/279371587 conn(0x5555bf197790 msgr2=0x5555bf197b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.929+0000 7f5765a00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57480a2880 0x7f57480a4d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f575402c170 tx=0x7f575402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.929+0000 7f576b07e740 1 -- 172.21.15.7:0/279371587 shutdown_connections 2024-03-18T01:42:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.929+0000 7f576b07e740 1 -- 172.21.15.7:0/279371587 wait complete. 2024-03-18T01:42:51.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.930+0000 7f576b07e740 1 -- start start 2024-03-18T01:42:51.932 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.930+0000 7f576b07e740 1 Processor -- start 2024-03-18T01:42:51.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.933+0000 7f576b07e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf1a6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.934+0000 7f576b07e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 0x5555bf1a6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.934+0000 7f576b07e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 0x5555bf1adba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.934+0000 7f5765a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf1a6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.934+0000 7f5765a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf1a6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:48596/0 (socket says 172.21.15.7:48596) 2024-03-18T01:42:51.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.935+0000 7f576b07e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5555bf1a4d30 con 0x5555bf1a1d80 2024-03-18T01:42:51.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.935+0000 7f576b07e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5555bf0b3ed0 con 0x5555bf19c220 2024-03-18T01:42:51.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.935+0000 7f576b07e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5555bf1a4eb0 con 0x5555bf19cc10 2024-03-18T01:42:51.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.935+0000 7f576b07e740 10 client.0 _ll_register_callbacks cb 0x5555bf0ef220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:42:51.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.936+0000 7f5765a00640 1 -- 172.21.15.7:0/920837233 learned_addr learned my addr 172.21.15.7:0/920837233 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.937+0000 7f5765a00640 1 -- 172.21.15.7:0/920837233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 msgr2=0x5555bf1a6b60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:42:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.937+0000 7f5765a00640 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 0x5555bf1a6b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.937+0000 7f5765a00640 1 -- 172.21.15.7:0/920837233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 msgr2=0x5555bf1adba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.937+0000 7f5765a00640 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 0x5555bf1adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.937+0000 7f5765a00640 1 -- 172.21.15.7:0/920837233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555bee0c3f0 con 0x5555bf19c220 2024-03-18T01:42:51.938 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.938+0000 7f5747400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:42:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.938+0000 7f5765a00640 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf1a6620 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f57540096c0 tx=0x7f5754009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9521340c193d93fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:51.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.939+0000 7f5747400640 1 -- 172.21.15.7:0/920837233 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754045070 con 0x5555bf19c220 2024-03-18T01:42:51.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.940+0000 7f5747400640 1 -- 172.21.15.7:0/920837233 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5754003770 con 0x5555bf19c220 2024-03-18T01:42:51.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.940+0000 7f576b07e740 1 -- 172.21.15.7:0/920837233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5760081950 con 0x5555bf19c220 2024-03-18T01:42:51.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.942+0000 7f5747400640 1 -- 172.21.15.7:0/920837233 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5754003970 con 0x5555bf19c220 2024-03-18T01:42:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.942+0000 7f576b07e740 10 client.26204 fetch_fsmap learned FSMap version 35 2024-03-18T01:42:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.943+0000 7f576b07e740 1 -- 172.21.15.7:0/920837233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5555bedcc0f0 con 0x5555bf19c220 2024-03-18T01:42:51.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.943+0000 7f5747400640 1 -- 172.21.15.7:0/920837233 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5760081950 con 0x5555bf19c220 2024-03-18T01:42:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.945+0000 7f5747400640 1 -- 172.21.15.7:0/920837233 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f5754034440 con 0x5555bf19c220 2024-03-18T01:42:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.945+0000 7f576b07e740 10 client.26204 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T01:42:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.945+0000 7f576b07e740 10 client.26204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:42:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.945+0000 7f576b07e740 20 client.26204 populate_metadata read hostname 'smithi007' 2024-03-18T01:42:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.946+0000 7f576b07e740 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5555bf1b24a0 0x5555bf1b4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.946+0000 7f576b07e740 4 client.26204 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:42:51.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.946+0000 7f576b07e740 1 -- 172.21.15.7:0/920837233 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555bf10bbe0 con 0x5555bf1b24a0 2024-03-18T01:42:51.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.947+0000 7f5765000640 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5555bf1b24a0 0x5555bf1b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:51.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.949+0000 7f5747400640 10 client.26204 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:42:51.949 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.949+0000 7f5765000640 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5555bf1b24a0 0x5555bf1b4940 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b9a4d467027d508 server_cookie=eec05ab6f209a4ef in_seq=0 out_seq=0 2024-03-18T01:42:51.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.951+0000 7f5747400640 1 -- 172.21.15.7:0/920837233 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555bf10bbe0 con 0x5555bf1b24a0 2024-03-18T01:42:51.951 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.951+0000 7f5747400640 10 client.26204 handle_command_reply: tid=0 2024-03-18T01:42:51.952 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.951+0000 7f576b07e740 1 client.26204 shutdown 2024-03-18T01:42:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.954+0000 7f576b07e740 1 -- 172.21.15.7:0/920837233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 msgr2=0x5555bf1a6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:51.954 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.954+0000 7f576b07e740 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf1a6620 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f57540096c0 tx=0x7f5754009260 comp rx=0 tx=0).stop 2024-03-18T01:42:51.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.955+0000 7f576b07e740 1 -- 172.21.15.7:0/920837233 shutdown_connections 2024-03-18T01:42:51.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.956+0000 7f576b07e740 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5555bf1b24a0 0x5555bf1b4940 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.956+0000 7f576b07e740 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 0x5555bf1adba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.956+0000 7f576b07e740 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 0x5555bf1a6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.956 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.956+0000 7f576b07e740 1 --2- 172.21.15.7:0/920837233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19c220 0x5555bf1a6620 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:51.957 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.957+0000 7f576b07e740 1 -- 172.21.15.7:0/920837233 >> 172.21.15.7:0/920837233 conn(0x5555bf197790 msgr2=0x5555bf18e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:51.958 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.958+0000 7f5747400640 0 client.26204 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:42:51.959 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.958+0000 7f576b07e740 1 -- 172.21.15.7:0/920837233 shutdown_connections 2024-03-18T01:42:51.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.963+0000 7f576b07e740 1 -- 172.21.15.7:0/920837233 wait complete. 2024-03-18T01:42:51.964 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:51.963+0000 7f576b07e740 20 client.26204 trim_cache size 0 max 16384 2024-03-18T01:42:52.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.006+0000 7f576b07e740 1 Processor -- start 2024-03-18T01:42:52.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.006+0000 7f576b07e740 1 -- start start 2024-03-18T01:42:52.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.006+0000 7f576b07e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19c220 0x5555bf1a6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.007+0000 7f576b07e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19cc10 0x5555bf1adb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.007+0000 7f576b07e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 0x5555bf1a6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.007+0000 7f5765a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19c220 0x5555bf1a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.007+0000 7f5765a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19c220 0x5555bf1a6230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53626/0 (socket says 172.21.15.7:53626) 2024-03-18T01:42:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.007+0000 7f5765a00640 1 -- 172.21.15.7:0/659875983 learned_addr learned my addr 172.21.15.7:0/659875983 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:52.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.007+0000 7f5765000640 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19cc10 0x5555bf1adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.008+0000 7f576b07e740 1 -- 172.21.15.7:0/659875983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f575813c6b0 con 0x5555bf1a1d80 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.008+0000 7f576b07e740 1 -- 172.21.15.7:0/659875983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f575813e220 con 0x5555bf19cc10 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.008+0000 7f576b07e740 1 -- 172.21.15.7:0/659875983 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f575813cdb0 con 0x5555bf19c220 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.008+0000 7f5766400640 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 0x5555bf1a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.009+0000 7f5766400640 1 -- 172.21.15.7:0/659875983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19c220 msgr2=0x5555bf1a6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.009+0000 7f5766400640 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19c220 0x5555bf1a6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.009+0000 7f5766400640 1 -- 172.21.15.7:0/659875983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19cc10 msgr2=0x5555bf1adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.009+0000 7f5766400640 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19cc10 0x5555bf1adb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.009+0000 7f5766400640 1 -- 172.21.15.7:0/659875983 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555bf1981e0 con 0x5555bf1a1d80 2024-03-18T01:42:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.009+0000 7f5766400640 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 0x5555bf1a6770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f57600502f0 tx=0x7f57600676f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0fc4d81a4b1310 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:52.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.010+0000 7f5752000640 1 -- 172.21.15.7:0/659875983 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760086040 con 0x5555bf1a1d80 2024-03-18T01:42:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.010+0000 7f5752000640 1 -- 172.21.15.7:0/659875983 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f576004fe10 con 0x5555bf1a1d80 2024-03-18T01:42:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.011+0000 7f5765a00640 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19c220 0x5555bf1a6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:42:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.011+0000 7f5765000640 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19cc10 0x5555bf1adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:42:52.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.011+0000 7f5752000640 1 -- 172.21.15.7:0/659875983 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760080d50 con 0x5555bf1a1d80 2024-03-18T01:42:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.014+0000 7f576b07e740 1 -- 172.21.15.7:0/659875983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 msgr2=0x5555bf1a6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:52.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.014+0000 7f576b07e740 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 0x5555bf1a6770 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f57600502f0 tx=0x7f57600676f0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.017+0000 7f576b07e740 1 -- 172.21.15.7:0/659875983 shutdown_connections 2024-03-18T01:42:52.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.017+0000 7f576b07e740 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf1a1d80 0x5555bf1a6770 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.017+0000 7f576b07e740 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf19cc10 0x5555bf1adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.017+0000 7f576b07e740 1 --2- 172.21.15.7:0/659875983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19c220 0x5555bf1a6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.017+0000 7f576b07e740 1 -- 172.21.15.7:0/659875983 >> 172.21.15.7:0/659875983 conn(0x5555bf197790 msgr2=0x5555bf189fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.018+0000 7f576b07e740 1 -- 172.21.15.7:0/659875983 shutdown_connections 2024-03-18T01:42:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.018+0000 7f576b07e740 1 -- 172.21.15.7:0/659875983 wait complete. 2024-03-18T01:42:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.018+0000 7f576b07e740 1 -- start start 2024-03-18T01:42:52.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.019+0000 7f576b07e740 1 Processor -- start 2024-03-18T01:42:52.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.023+0000 7f576b07e740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19c220 0x5555bf1aaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:52.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.024+0000 7f576b07e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 0x5555bf1ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:52.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.024+0000 7f576b07e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf1a1d80 0x5555bf216830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.024+0000 7f5765000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 0x5555bf1ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.024+0000 7f5765000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 0x5555bf1ab030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53638/0 (socket says 172.21.15.7:53638) 2024-03-18T01:42:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.024+0000 7f5765000640 1 -- 172.21.15.7:0/1368183167 learned_addr learned my addr 172.21.15.7:0/1368183167 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:42:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.024+0000 7f576b07e740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f575813db50 con 0x5555bf19c220 2024-03-18T01:42:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.025+0000 7f5766400640 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf1a1d80 0x5555bf216830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.025+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f575813d480 con 0x5555bf1a1d80 2024-03-18T01:42:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.025+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f575813a6b0 con 0x5555bf19cc10 2024-03-18T01:42:52.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.025+0000 7f576b07e740 10 client.0 _ll_register_callbacks cb 0x7f575402c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:42:52.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.025+0000 7f5765a00640 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19c220 0x5555bf1aaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:52.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.026+0000 7f5766400640 1 -- 172.21.15.7:0/1368183167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 msgr2=0x5555bf1ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:52.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.026+0000 7f5766400640 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 0x5555bf1ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.026+0000 7f5766400640 1 -- 172.21.15.7:0/1368183167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19c220 msgr2=0x5555bf1aaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:42:52.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.026+0000 7f5766400640 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19c220 0x5555bf1aaaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.026+0000 7f5766400640 1 -- 172.21.15.7:0/1368183167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555bf216da0 con 0x5555bf1a1d80 2024-03-18T01:42:52.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.026+0000 7f5747400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:42:52.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.027+0000 7f5766400640 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf1a1d80 0x5555bf216830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x5555bf18e9c0 tx=0x7f57600671d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0cb2169ecd45db6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:42:52.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.027+0000 7f5747400640 1 -- 172.21.15.7:0/1368183167 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5760086070 con 0x5555bf1a1d80 2024-03-18T01:42:52.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.028+0000 7f5747400640 1 -- 172.21.15.7:0/1368183167 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5760080e60 con 0x5555bf1a1d80 2024-03-18T01:42:52.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.028+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f576004fe10 con 0x5555bf1a1d80 2024-03-18T01:42:52.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.028+0000 7f5747400640 1 -- 172.21.15.7:0/1368183167 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f576007f880 con 0x5555bf1a1d80 2024-03-18T01:42:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.029+0000 7f576b07e740 10 client.26207 fetch_fsmap learned FSMap version 35 2024-03-18T01:42:52.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.029+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5555bf218020 con 0x5555bf1a1d80 2024-03-18T01:42:52.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.030+0000 7f5747400640 1 -- 172.21.15.7:0/1368183167 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f576004fe10 con 0x5555bf1a1d80 2024-03-18T01:42:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.030+0000 7f5747400640 1 -- 172.21.15.7:0/1368183167 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f5760081b90 con 0x5555bf1a1d80 2024-03-18T01:42:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.030+0000 7f576b07e740 10 client.26207 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T01:42:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.030+0000 7f576b07e740 10 client.26207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:42:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.030+0000 7f576b07e740 20 client.26207 populate_metadata read hostname 'smithi007' 2024-03-18T01:42:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.030+0000 7f576b07e740 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5555bf218630 0x5555bf21aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:42:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.031+0000 7f576b07e740 4 client.26207 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:42:52.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.031+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5555bf105730 con 0x5555bf218630 2024-03-18T01:42:52.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.031+0000 7f5765a00640 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5555bf218630 0x5555bf21aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:42:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.032+0000 7f5765a00640 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5555bf218630 0x5555bf21aa20 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b7e9485b899ebc7 server_cookie=5f29ed3d7686747d in_seq=0 out_seq=0 2024-03-18T01:42:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.032+0000 7f5747400640 10 client.26207 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:42:52.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.033+0000 7f5747400640 1 -- 172.21.15.7:0/1368183167 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x5555bf105730 con 0x5555bf218630 2024-03-18T01:42:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.033+0000 7f5747400640 10 client.26207 handle_command_reply: tid=0 2024-03-18T01:42:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.033+0000 7f576b07e740 1 client.26207 shutdown 2024-03-18T01:42:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.034+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf1a1d80 msgr2=0x5555bf216830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:52.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.034+0000 7f576b07e740 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf1a1d80 0x5555bf216830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x5555bf18e9c0 tx=0x7f57600671d0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 shutdown_connections 2024-03-18T01:42:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f576b07e740 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5555bf218630 0x5555bf21aa20 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f576b07e740 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555bf1a1d80 0x5555bf216830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f576b07e740 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5555bf19cc10 0x5555bf1ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f576b07e740 1 --2- 172.21.15.7:0/1368183167 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5555bf19c220 0x5555bf1aaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 >> 172.21.15.7:0/1368183167 conn(0x5555bf197790 msgr2=0x5555bf1a61b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f5747400640 0 client.26207 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:42:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 shutdown_connections 2024-03-18T01:42:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.036+0000 7f576b07e740 1 -- 172.21.15.7:0/1368183167 wait complete. 2024-03-18T01:42:52.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.037+0000 7f576b07e740 20 client.26207 trim_cache size 0 max 16384 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (16 inodes in the stack)", 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:42:52.038 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:42:52.039 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:42:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57480a2880 msgr2=0x7f57480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57480a2880 0x7f57480a4d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f575402c170 tx=0x7f575402df90 comp rx=0 tx=0).stop 2024-03-18T01:42:52.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5758096f60 msgr2=0x7f575812e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5758096f60 0x7f575812e7c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f574c00e9f0 tx=0x7f574c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 shutdown_connections 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5758097920 0x7f5758135830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5758096f60 0x7f575812e7c0 secure :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f574c00e9f0 tx=0x7f574c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f57480a2880 0x7f57480a4d40 secure :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f575402c170 tx=0x7f575402df90 comp rx=0 tx=0).stop 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 --2- 172.21.15.7:0/2784267195 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5758095d70 0x7f575812e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.038+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 >> 172.21.15.7:0/2784267195 conn(0x7f57580912e0 msgr2=0x7f5758007b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.039+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 shutdown_connections 2024-03-18T01:42:52.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:42:52.039+0000 7f5766e00640 1 -- 172.21.15.7:0/2784267195 wait complete. 2024-03-18T01:42:52.079 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:42:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:52 smithi046 ceph-mon[36108]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 49 MiB/s wr, 7.28k op/s 2024-03-18T01:42:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:52 smithi007 ceph-mon[31996]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 49 MiB/s wr, 7.28k op/s 2024-03-18T01:42:52.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:52 smithi203 ceph-mon[36345]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 49 MiB/s wr, 7.28k op/s 2024-03-18T01:42:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:54 smithi046 ceph-mon[36108]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 39 MiB/s wr, 5.69k op/s 2024-03-18T01:42:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:54 smithi007 ceph-mon[31996]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 39 MiB/s wr, 5.69k op/s 2024-03-18T01:42:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:54 smithi203 ceph-mon[36345]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 39 MiB/s wr, 5.69k op/s 2024-03-18T01:42:56.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:56 smithi046 ceph-mon[36108]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 43 MiB/s wr, 6.09k op/s 2024-03-18T01:42:56.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:56 smithi007 ceph-mon[31996]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 43 MiB/s wr, 6.09k op/s 2024-03-18T01:42:56.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:56 smithi203 ceph-mon[36345]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 43 MiB/s wr, 6.09k op/s 2024-03-18T01:42:58.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:42:58 smithi046 ceph-mon[36108]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 50 MiB/s wr, 7.16k op/s 2024-03-18T01:42:58.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:42:58 smithi007 ceph-mon[31996]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 50 MiB/s wr, 7.16k op/s 2024-03-18T01:42:58.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:42:58 smithi203 ceph-mon[36345]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 50 MiB/s wr, 7.16k op/s 2024-03-18T01:43:00.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:00 smithi046 ceph-mon[36108]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 40 MiB/s wr, 5.73k op/s 2024-03-18T01:43:00.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:00 smithi007 ceph-mon[31996]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 40 MiB/s wr, 5.73k op/s 2024-03-18T01:43:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:00 smithi203 ceph-mon[36345]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 40 MiB/s wr, 5.73k op/s 2024-03-18T01:43:02.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:02 smithi046 ceph-mon[36108]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 MiB/s wr, 6.64k op/s 2024-03-18T01:43:02.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:02 smithi007 ceph-mon[31996]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 MiB/s wr, 6.64k op/s 2024-03-18T01:43:02.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:02 smithi203 ceph-mon[36345]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 MiB/s wr, 6.64k op/s 2024-03-18T01:43:04.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:04 smithi007 ceph-mon[31996]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 39 MiB/s wr, 5.78k op/s 2024-03-18T01:43:04.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:04 smithi203 ceph-mon[36345]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 39 MiB/s wr, 5.78k op/s 2024-03-18T01:43:05.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:04 smithi046 ceph-mon[36108]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 39 MiB/s wr, 5.78k op/s 2024-03-18T01:43:05.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:05 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:43:05.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:05 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:43:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:05 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:43:06.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:06 smithi007 ceph-mon[31996]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 MiB/s wr, 6.03k op/s 2024-03-18T01:43:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:06 smithi203 ceph-mon[36345]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 MiB/s wr, 6.03k op/s 2024-03-18T01:43:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:06 smithi046 ceph-mon[36108]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 MiB/s wr, 6.03k op/s 2024-03-18T01:43:07.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:43:07.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:43:07.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:43:07.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:43:07.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:43:07.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:43:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:43:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:43:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:43:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:08 smithi046 ceph-mon[36108]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 44 MiB/s wr, 6.86k op/s 2024-03-18T01:43:09.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:08 smithi007 ceph-mon[31996]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 44 MiB/s wr, 6.86k op/s 2024-03-18T01:43:09.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:08 smithi203 ceph-mon[36345]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 44 MiB/s wr, 6.86k op/s 2024-03-18T01:43:11.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:10 smithi046 ceph-mon[36108]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 39 MiB/s wr, 6.02k op/s 2024-03-18T01:43:11.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:10 smithi007 ceph-mon[31996]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 39 MiB/s wr, 6.02k op/s 2024-03-18T01:43:11.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:10 smithi203 ceph-mon[36345]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 39 MiB/s wr, 6.02k op/s 2024-03-18T01:43:13.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:12 smithi046 ceph-mon[36108]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 MiB/s wr, 6.19k op/s 2024-03-18T01:43:13.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:12 smithi007 ceph-mon[31996]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 MiB/s wr, 6.19k op/s 2024-03-18T01:43:13.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:12 smithi203 ceph-mon[36345]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 MiB/s wr, 6.19k op/s 2024-03-18T01:43:15.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:14 smithi046 ceph-mon[36108]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 40 MiB/s wr, 6.02k op/s 2024-03-18T01:43:15.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:14 smithi007 ceph-mon[31996]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 40 MiB/s wr, 6.02k op/s 2024-03-18T01:43:15.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:14 smithi203 ceph-mon[36345]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 40 MiB/s wr, 6.02k op/s 2024-03-18T01:43:17.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:16 smithi007 ceph-mon[31996]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 39 MiB/s wr, 5.85k op/s 2024-03-18T01:43:17.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:16 smithi203 ceph-mon[36345]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 39 MiB/s wr, 5.85k op/s 2024-03-18T01:43:17.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:16 smithi046 ceph-mon[36108]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 39 MiB/s wr, 5.85k op/s 2024-03-18T01:43:19.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:18 smithi007 ceph-mon[31996]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 42 MiB/s wr, 6.29k op/s 2024-03-18T01:43:19.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:18 smithi203 ceph-mon[36345]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 42 MiB/s wr, 6.29k op/s 2024-03-18T01:43:19.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:18 smithi046 ceph-mon[36108]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 42 MiB/s wr, 6.29k op/s 2024-03-18T01:43:21.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:20 smithi007 ceph-mon[31996]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 38 MiB/s wr, 5.69k op/s 2024-03-18T01:43:21.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:20 smithi203 ceph-mon[36345]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 38 MiB/s wr, 5.69k op/s 2024-03-18T01:43:21.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:20 smithi046 ceph-mon[36108]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 38 MiB/s wr, 5.69k op/s 2024-03-18T01:43:22.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:43:22.081 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:43:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.001+0000 7f1bbd600640 1 -- 172.21.15.7:0/1386453665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 msgr2=0x7f1bb8106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.002 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.001+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1386453665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 0x7f1bb8106ec0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009900 tx=0x7f1ba802bc30 comp rx=0 tx=0).stop 2024-03-18T01:43:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.002+0000 7f1bbd600640 1 -- 172.21.15.7:0/1386453665 shutdown_connections 2024-03-18T01:43:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.002+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1386453665 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8107450 0x7f1bb8109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.003 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.002+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1386453665 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 0x7f1bb8106ec0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.002+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1386453665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bb80ff560 0x7f1bb80ff980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.002+0000 7f1bbd600640 1 -- 172.21.15.7:0/1386453665 >> 172.21.15.7:0/1386453665 conn(0x7f1bb80faf90 msgr2=0x7f1bb80fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.002+0000 7f1bbd600640 1 -- 172.21.15.7:0/1386453665 shutdown_connections 2024-03-18T01:43:23.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.002+0000 7f1bbd600640 1 -- 172.21.15.7:0/1386453665 wait complete. 2024-03-18T01:43:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.005+0000 7f1bbd600640 1 Processor -- start 2024-03-18T01:43:23.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.005+0000 7f1bbd600640 1 -- start start 2024-03-18T01:43:23.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.006+0000 7f1bbd600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bb80ff560 0x7f1bb8072d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.006+0000 7f1bbd600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 0x7f1bb8073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.006+0000 7f1bbd600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8107450 0x7f1bb8071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.006+0000 7f1bbd600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb806bc50 con 0x7f1bb8107450 2024-03-18T01:43:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.006+0000 7f1bbd600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb806bad0 con 0x7f1bb80ff560 2024-03-18T01:43:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.006+0000 7f1bbd600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb806bdd0 con 0x7f1bb80ffec0 2024-03-18T01:43:23.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.007+0000 7f1bb6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bb80ff560 0x7f1bb8072d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.007+0000 7f1bb6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bb80ff560 0x7f1bb8072d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46192/0 (socket says 172.21.15.7:46192) 2024-03-18T01:43:23.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.007+0000 7f1bb6a00640 1 -- 172.21.15.7:0/1443828747 learned_addr learned my addr 172.21.15.7:0/1443828747 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:23.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.007+0000 7f1bb7400640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8107450 0x7f1bb8071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.007+0000 7f1bb6000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 0x7f1bb8073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.009+0000 7f1bb6000640 1 -- 172.21.15.7:0/1443828747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bb80ff560 msgr2=0x7f1bb8072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.009+0000 7f1bb6000640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bb80ff560 0x7f1bb8072d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.009+0000 7f1bb6000640 1 -- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8107450 msgr2=0x7f1bb8071370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.009+0000 7f1bb6000640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8107450 0x7f1bb8071370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.009+0000 7f1bb6000640 1 -- 172.21.15.7:0/1443828747 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb8071af0 con 0x7f1bb80ffec0 2024-03-18T01:43:23.011 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.009+0000 7f1bb7400640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8107450 0x7f1bb8071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:43:23.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.010+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bb80ff560 0x7f1bb8072d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:43:23.012 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.011+0000 7f1bb6000640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 0x7f1bb8073270 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009230 tx=0x7f1ba8004e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:23.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.011+0000 7f1baf400640 1 -- 172.21.15.7:0/1443828747 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8043020 con 0x7f1bb80ffec0 2024-03-18T01:43:23.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.013+0000 7f1baf400640 1 -- 172.21.15.7:0/1443828747 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba80054e0 con 0x7f1bb80ffec0 2024-03-18T01:43:23.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.013+0000 7f1baf400640 1 -- 172.21.15.7:0/1443828747 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8003bc0 con 0x7f1bb80ffec0 2024-03-18T01:43:23.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.014+0000 7f1bbd600640 1 -- 172.21.15.7:0/1443828747 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb8071d80 con 0x7f1bb80ffec0 2024-03-18T01:43:23.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.014+0000 7f1bbd600640 1 -- 172.21.15.7:0/1443828747 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb8100c90 con 0x7f1bb80ffec0 2024-03-18T01:43:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.024+0000 7f1bc0230740 1 Processor -- start 2024-03-18T01:43:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.024+0000 7f1baf400640 1 -- 172.21.15.7:0/1443828747 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f1ba8046020 con 0x7f1bb80ffec0 2024-03-18T01:43:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.025+0000 7f1bc0230740 1 -- start start 2024-03-18T01:43:23.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.025+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfd810 0x5608edbfdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.025+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc02d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.025+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc05800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.027 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.025+0000 7f1bc0230740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5608edb621f0 con 0x5608edbfd810 2024-03-18T01:43:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.025+0000 7f1bc0230740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5608edb0aa80 con 0x5608edc03370 2024-03-18T01:43:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.025+0000 7f1bc0230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5608edb5e290 con 0x5608edbfe200 2024-03-18T01:43:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.026+0000 7f1bb6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc05800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.026+0000 7f1bb6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc05800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46204/0 (socket says 172.21.15.7:46204) 2024-03-18T01:43:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.026+0000 7f1bb6000640 1 -- 172.21.15.7:0/2477366317 learned_addr learned my addr 172.21.15.7:0/2477366317 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:23.028 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.027+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc02d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.027+0000 7f1bb6000640 1 -- 172.21.15.7:0/2477366317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 msgr2=0x5608edc02d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.027+0000 7f1bb6000640 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc02d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.027+0000 7f1bb6000640 1 -- 172.21.15.7:0/2477366317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfd810 msgr2=0x5608edbfdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.027+0000 7f1bb6000640 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfd810 0x5608edbfdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.027+0000 7f1bb6000640 1 -- 172.21.15.7:0/2477366317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608ed9407d0 con 0x5608edc03370 2024-03-18T01:43:23.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.028+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc02d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:43:23.030 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.028+0000 7f1bb6000640 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc05800 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ba80f6910 tx=0x7f1ba80f6a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90915b9f2be938cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.030+0000 7f1ba3400640 1 -- 172.21.15.7:0/2477366317 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80f87d0 con 0x5608edc03370 2024-03-18T01:43:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.030+0000 7f1ba3400640 1 -- 172.21.15.7:0/2477366317 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba8102070 con 0x5608edc03370 2024-03-18T01:43:23.031 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.030+0000 7f1ba3400640 1 -- 172.21.15.7:0/2477366317 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80fdde0 con 0x5608edc03370 2024-03-18T01:43:23.032 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.031+0000 7f1baf400640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b9c0a28d0 0x7f1b9c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.031+0000 7f1bc0230740 1 -- 172.21.15.7:0/2477366317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 msgr2=0x5608edc05800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.031+0000 7f1bc0230740 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc05800 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ba80f6910 tx=0x7f1ba80f6a00 comp rx=0 tx=0).stop 2024-03-18T01:43:23.033 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.032+0000 7f1baf400640 1 -- 172.21.15.7:0/1443828747 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1ba80f0910 con 0x7f1bb80ffec0 2024-03-18T01:43:23.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.033+0000 7f1bb7400640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b9c0a28d0 0x7f1b9c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.033+0000 7f1bc0230740 1 -- 172.21.15.7:0/2477366317 shutdown_connections 2024-03-18T01:43:23.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.033+0000 7f1bc0230740 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc05800 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.033+0000 7f1bc0230740 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc02d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.033+0000 7f1bc0230740 1 --2- 172.21.15.7:0/2477366317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfd810 0x5608edbfdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.034 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.033+0000 7f1bc0230740 1 -- 172.21.15.7:0/2477366317 >> 172.21.15.7:0/2477366317 conn(0x5608edbf8d80 msgr2=0x5608edbf9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:23.035 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.034+0000 7f1bb7400640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b9c0a28d0 0x7f1b9c0a4d90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0007310 tx=0x7f1bb00073e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.035+0000 7f1bc0230740 1 -- 172.21.15.7:0/2477366317 shutdown_connections 2024-03-18T01:43:23.036 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.035+0000 7f1bc0230740 1 -- 172.21.15.7:0/2477366317 wait complete. 2024-03-18T01:43:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.036+0000 7f1bc0230740 1 -- start start 2024-03-18T01:43:23.037 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.036+0000 7f1bc0230740 1 Processor -- start 2024-03-18T01:43:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.037+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc07960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.037+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfe200 0x5608edc07ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.038 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.037+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.038+0000 7f1bc0230740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5608edb0aa80 con 0x5608edbfe200 2024-03-18T01:43:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.038+0000 7f1bb7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.038+0000 7f1bb7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46206/0 (socket says 172.21.15.7:46206) 2024-03-18T01:43:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.038+0000 7f1bb7400640 1 -- 172.21.15.7:0/4000054584 learned_addr learned my addr 172.21.15.7:0/4000054584 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:23.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.038+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5608edb1ded0 con 0x5608edc03370 2024-03-18T01:43:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.038+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5608edc060e0 con 0x5608edbfd810 2024-03-18T01:43:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.039+0000 7f1bc0230740 10 client.0 _ll_register_callbacks cb 0x5608edb59220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:43:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.039+0000 7f1bb6000640 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfe200 0x5608edc07ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.039+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc07960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.039+0000 7f1bb7400640 1 -- 172.21.15.7:0/4000054584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 msgr2=0x5608edc07960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.039+0000 7f1bb7400640 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc07960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.039+0000 7f1bb7400640 1 -- 172.21.15.7:0/4000054584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfe200 msgr2=0x5608edc07ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.039+0000 7f1bb7400640 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfe200 0x5608edc07ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.039+0000 7f1bb7400640 1 -- 172.21.15.7:0/4000054584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608ed8763f0 con 0x5608edc03370 2024-03-18T01:43:23.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.040+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc07960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:43:23.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.043+0000 7f1bb7400640 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0eee0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0010e00 tx=0x7f1bb0004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c057c45c82ed12a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:23.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.043+0000 7f1b97e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:43:23.045 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.044+0000 7f1b97e00640 1 -- 172.21.15.7:0/4000054584 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0021070 con 0x5608edc03370 2024-03-18T01:43:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.044+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ba8102070 con 0x5608edc03370 2024-03-18T01:43:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.044+0000 7f1b97e00640 1 -- 172.21.15.7:0/4000054584 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1bb0004b20 con 0x5608edc03370 2024-03-18T01:43:23.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.045+0000 7f1b97e00640 1 -- 172.21.15.7:0/4000054584 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1bb002c050 con 0x5608edc03370 2024-03-18T01:43:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.045+0000 7f1bc0230740 10 client.26219 fetch_fsmap learned FSMap version 35 2024-03-18T01:43:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.045+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5608ed8360f0 con 0x5608edc03370 2024-03-18T01:43:23.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.046+0000 7f1b97e00640 1 -- 172.21.15.7:0/4000054584 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba8102070 con 0x5608edc03370 2024-03-18T01:43:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.047+0000 7f1b97e00640 1 -- 172.21.15.7:0/4000054584 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f1bb001c4d0 con 0x5608edc03370 2024-03-18T01:43:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.047+0000 7f1bc0230740 10 client.26219 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T01:43:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.047+0000 7f1bc0230740 10 client.26219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:43:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.047+0000 7f1bc0230740 20 client.26219 populate_metadata read hostname 'smithi007' 2024-03-18T01:43:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.047+0000 7f1bc0230740 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5608edc13790 0x5608edc15bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.047+0000 7f1bc0230740 4 client.26219 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:43:23.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.047+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608edb75be0 con 0x5608edc13790 2024-03-18T01:43:23.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.048+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5608edc13790 0x5608edc15bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.050+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5608edc13790 0x5608edc15bf0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f478c2ee76ac3158 server_cookie=c63d45302820ebb5 in_seq=0 out_seq=0 2024-03-18T01:43:23.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.050+0000 7f1b97e00640 10 client.26219 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:43:23.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:22 smithi007 ceph-mon[31996]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 39 MiB/s wr, 5.90k op/s 2024-03-18T01:43:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.051+0000 7f1b97e00640 1 -- 172.21.15.7:0/4000054584 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5608edb75be0 con 0x5608edc13790 2024-03-18T01:43:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.051+0000 7f1b97e00640 10 client.26219 handle_command_reply: tid=0 2024-03-18T01:43:23.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.051+0000 7f1bc0230740 1 client.26219 shutdown 2024-03-18T01:43:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 msgr2=0x5608edc0eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1bc0230740 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0eee0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0010e00 tx=0x7f1bb0004ab0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 shutdown_connections 2024-03-18T01:43:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1bc0230740 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5608edc13790 0x5608edc15bf0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1bc0230740 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0eee0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1bc0230740 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edbfe200 0x5608edc07ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.054 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1bc0230740 1 --2- 172.21.15.7:0/4000054584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc07960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 >> 172.21.15.7:0/4000054584 conn(0x5608edbf8d80 msgr2=0x5608edbf7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:23.055 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.053+0000 7f1b97e00640 0 client.26219 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:43:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.055+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 shutdown_connections 2024-03-18T01:43:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.055+0000 7f1bc0230740 1 -- 172.21.15.7:0/4000054584 wait complete. 2024-03-18T01:43:23.056 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.055+0000 7f1bc0230740 20 client.26219 trim_cache size 0 max 16384 2024-03-18T01:43:23.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.096+0000 7f1bc0230740 1 Processor -- start 2024-03-18T01:43:23.097 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.096+0000 7f1bc0230740 1 -- start start 2024-03-18T01:43:23.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.097+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc063d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.097+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edbfe200 0x5608edc0eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.097+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc03370 0x5608edc06910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.097+0000 7f1bc0230740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81b4d20 con 0x5608edc03370 2024-03-18T01:43:23.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.097+0000 7f1bc0230740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81b6890 con 0x5608edbfe200 2024-03-18T01:43:23.098 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.097+0000 7f1bc0230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81b5420 con 0x5608edbfd810 2024-03-18T01:43:23.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.097+0000 7f1bb6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb7400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc03370 0x5608edc06910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34512/0 (socket says 172.21.15.7:34512) 2024-03-18T01:43:23.099 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb6a00640 1 -- 172.21.15.7:0/3165871160 learned_addr learned my addr 172.21.15.7:0/3165871160 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb6000640 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edbfe200 0x5608edc0eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb6a00640 1 -- 172.21.15.7:0/3165871160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edbfe200 msgr2=0x5608edc0eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edbfe200 0x5608edc0eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb6a00640 1 -- 172.21.15.7:0/3165871160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc03370 msgr2=0x5608edc06910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc03370 0x5608edc06910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb6a00640 1 -- 172.21.15.7:0/3165871160 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608edbf3370 con 0x5608edbfd810 2024-03-18T01:43:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.098+0000 7f1bb7400640 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc03370 0x5608edc06910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:43:23.100 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.099+0000 7f1bb6000640 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edbfe200 0x5608edc0eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:43:23.102 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.101+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc063d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x5608edbfe600 tx=0x7f1ba4015b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b9b98c5fa29d5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:23.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.102+0000 7f1ba2a00640 1 -- 172.21.15.7:0/3165871160 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba401eb90 con 0x5608edbfd810 2024-03-18T01:43:23.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.102+0000 7f1ba2a00640 1 -- 172.21.15.7:0/3165871160 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba401ed30 con 0x5608edbfd810 2024-03-18T01:43:23.103 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.102+0000 7f1ba2a00640 1 -- 172.21.15.7:0/3165871160 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4012be0 con 0x5608edbfd810 2024-03-18T01:43:23.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 -- 172.21.15.7:0/3165871160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 msgr2=0x5608edc063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc063d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x5608edbfe600 tx=0x7f1ba4015b70 comp rx=0 tx=0).stop 2024-03-18T01:43:23.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 -- 172.21.15.7:0/3165871160 shutdown_connections 2024-03-18T01:43:23.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc03370 0x5608edc06910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edbfe200 0x5608edc0eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.104 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 --2- 172.21.15.7:0/3165871160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfd810 0x5608edc063d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 -- 172.21.15.7:0/3165871160 >> 172.21.15.7:0/3165871160 conn(0x5608edbf8d80 msgr2=0x5608edc07960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:23.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 -- 172.21.15.7:0/3165871160 shutdown_connections 2024-03-18T01:43:23.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.103+0000 7f1bc0230740 1 -- 172.21.15.7:0/3165871160 wait complete. 2024-03-18T01:43:23.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.104+0000 7f1bc0230740 1 -- start start 2024-03-18T01:43:23.105 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.104+0000 7f1bc0230740 1 Processor -- start 2024-03-18T01:43:23.108 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.107+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc0c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bc0230740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc0db00 0x5608edc0df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bc0230740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81b61c0 con 0x5608edc0db00 2024-03-18T01:43:23.109 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bc0230740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81b5af0 con 0x5608edc03370 2024-03-18T01:43:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bc0230740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb81b2d00 con 0x5608edbfe200 2024-03-18T01:43:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bc0230740 10 client.0 _ll_register_callbacks cb 0x7f1bb0014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:43:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bb6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc0c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bb6a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc0c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:34522/0 (socket says 172.21.15.7:34522) 2024-03-18T01:43:23.110 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.108+0000 7f1bb6a00640 1 -- 172.21.15.7:0/695162725 learned_addr learned my addr 172.21.15.7:0/695162725 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:23.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.109+0000 7f1bb6a00640 1 -- 172.21.15.7:0/695162725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 msgr2=0x5608edc0cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:23.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.109+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.109+0000 7f1bb6a00640 1 -- 172.21.15.7:0/695162725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc0db00 msgr2=0x5608edc0df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:23.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.109+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc0db00 0x5608edc0df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.111 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.109+0000 7f1bb6a00640 1 -- 172.21.15.7:0/695162725 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608edc0e4c0 con 0x5608edbfe200 2024-03-18T01:43:23.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.110+0000 7f1bb6a00640 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc0c600 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4017010 tx=0x7f1ba4004590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cef577bbef3d351f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:23.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.110+0000 7f1b97e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:43:23.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.110+0000 7f1b97e00640 1 -- 172.21.15.7:0/695162725 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba402d070 con 0x5608edbfe200 2024-03-18T01:43:23.112 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.110+0000 7f1bc0230740 1 -- 172.21.15.7:0/695162725 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ba401ed30 con 0x5608edbfe200 2024-03-18T01:43:23.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.110+0000 7f1b97e00640 1 -- 172.21.15.7:0/695162725 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1ba401e670 con 0x5608edbfe200 2024-03-18T01:43:23.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.112+0000 7f1b97e00640 1 -- 172.21.15.7:0/695162725 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1ba401eb90 con 0x5608edbfe200 2024-03-18T01:43:23.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.112+0000 7f1bc0230740 10 client.26194 fetch_fsmap learned FSMap version 35 2024-03-18T01:43:23.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.112+0000 7f1bc0230740 1 -- 172.21.15.7:0/695162725 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5608edc82670 con 0x5608edbfe200 2024-03-18T01:43:23.113 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.112+0000 7f1b97e00640 1 -- 172.21.15.7:0/695162725 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba401ed30 con 0x5608edbfe200 2024-03-18T01:43:23.114 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.113+0000 7f1b97e00640 1 -- 172.21.15.7:0/695162725 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 35) ==== 8366+0+0 (secure 0 0 0) 0x7f1ba4012bc0 con 0x5608edbfe200 2024-03-18T01:43:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.113+0000 7f1bc0230740 10 client.26194 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T01:43:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.113+0000 7f1bc0230740 10 client.26194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:43:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.113+0000 7f1bc0230740 20 client.26194 populate_metadata read hostname 'smithi007' 2024-03-18T01:43:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.113+0000 7f1bc0230740 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5608edc82e70 0x5608edc852b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.113+0000 7f1bc0230740 4 client.26194 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:43:23.115 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.113+0000 7f1bc0230740 1 -- 172.21.15.7:0/695162725 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5608edb6f730 con 0x5608edc82e70 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.114+0000 7f1bb6000640 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5608edc82e70 0x5608edc852b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.115+0000 7f1b97e00640 10 client.26194 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.115+0000 7f1bb6000640 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5608edc82e70 0x5608edc852b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=450e81e762ac0f34 server_cookie=231cb12054ef3a90 in_seq=0 out_seq=0 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1b97e00640 1 -- 172.21.15.7:0/695162725 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x5608edb6f730 con 0x5608edc82e70 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1b97e00640 10 client.26194 handle_command_reply: tid=0 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 client.26194 shutdown 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 -- 172.21.15.7:0/695162725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 msgr2=0x5608edc0c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc0c600 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4017010 tx=0x7f1ba4004590 comp rx=0 tx=0).stop 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 -- 172.21.15.7:0/695162725 shutdown_connections 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5608edc82e70 0x5608edc852b0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5608edc0db00 0x5608edc0df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5608edc03370 0x5608edc0cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 --2- 172.21.15.7:0/695162725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5608edbfe200 0x5608edc0c600 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1bc0230740 1 -- 172.21.15.7:0/695162725 >> 172.21.15.7:0/695162725 conn(0x5608edbf8d80 msgr2=0x5608edc07830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.116+0000 7f1b97e00640 0 client.26194 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.117+0000 7f1bc0230740 1 -- 172.21.15.7:0/695162725 shutdown_connections 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.117+0000 7f1bc0230740 1 -- 172.21.15.7:0/695162725 wait complete. 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.117+0000 7f1bc0230740 20 client.26194 trim_cache size 0 max 16384 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:43:23.121 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (16 inodes in the stack)", 2024-03-18T01:43:23.122 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:43:23.122 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:43:23.122 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:43:23.122 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:43:23.122 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:43:23.122 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:43:23.122 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:43:23.122 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 -- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b9c0a28d0 msgr2=0x7f1b9c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b9c0a28d0 0x7f1b9c0a4d90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0007310 tx=0x7f1bb00073e0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 -- 172.21.15.7:0/1443828747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 msgr2=0x7f1bb8073270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 0x7f1bb8073270 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8009230 tx=0x7f1ba8004e30 comp rx=0 tx=0).stop 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 -- 172.21.15.7:0/1443828747 shutdown_connections 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1b9c0a28d0 0x7f1b9c0a4d90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1bb8107450 0x7f1bb8071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bb80ffec0 0x7f1bb8073270 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 --2- 172.21.15.7:0/1443828747 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1bb80ff560 0x7f1bb8072d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 -- 172.21.15.7:0/1443828747 >> 172.21.15.7:0/1443828747 conn(0x7f1bb80faf90 msgr2=0x7f1bb8102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 -- 172.21.15.7:0/1443828747 shutdown_connections 2024-03-18T01:43:23.123 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:23.120+0000 7f1bbd600640 1 -- 172.21.15.7:0/1443828747 wait complete. 2024-03-18T01:43:23.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:22 smithi203 ceph-mon[36345]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 39 MiB/s wr, 5.90k op/s 2024-03-18T01:43:23.159 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:43:23.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:22 smithi046 ceph-mon[36108]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 39 MiB/s wr, 5.90k op/s 2024-03-18T01:43:25.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:24 smithi203 ceph-mon[36345]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 MiB/s wr, 6.42k op/s 2024-03-18T01:43:25.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:24 smithi046 ceph-mon[36108]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 MiB/s wr, 6.42k op/s 2024-03-18T01:43:25.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:24 smithi007 ceph-mon[31996]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 MiB/s wr, 6.42k op/s 2024-03-18T01:43:27.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:26 smithi203 ceph-mon[36345]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 43 MiB/s wr, 6.36k op/s 2024-03-18T01:43:27.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:26 smithi046 ceph-mon[36108]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 43 MiB/s wr, 6.36k op/s 2024-03-18T01:43:27.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:26 smithi007 ceph-mon[31996]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 43 MiB/s wr, 6.36k op/s 2024-03-18T01:43:28.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:27 smithi203 ceph-mon[36345]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:27 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:28.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:27 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:27 smithi046 ceph-mon[36108]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:27 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:28.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:27 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:28.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:27 smithi007 ceph-mon[31996]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:28.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:27 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:28.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:27 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:28 smithi203 ceph-mon[36345]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 43 MiB/s wr, 6.37k op/s 2024-03-18T01:43:29.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:28 smithi046 ceph-mon[36108]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 43 MiB/s wr, 6.37k op/s 2024-03-18T01:43:29.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:28 smithi007 ceph-mon[31996]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 43 MiB/s wr, 6.37k op/s 2024-03-18T01:43:30.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:29 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:30.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:29 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:30.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:29 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:30.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:29 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:30.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:29 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:30.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:29 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:31.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:30 smithi046 ceph-mon[36108]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 41 MiB/s wr, 5.95k op/s 2024-03-18T01:43:31.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:30 smithi046 ceph-mon[36108]: Health check failed: 0 slow ops, oldest one blocked for 181 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:31.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:30 smithi007 ceph-mon[31996]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 41 MiB/s wr, 5.95k op/s 2024-03-18T01:43:31.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:30 smithi007 ceph-mon[31996]: Health check failed: 0 slow ops, oldest one blocked for 181 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:30 smithi203 ceph-mon[36345]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 41 MiB/s wr, 5.95k op/s 2024-03-18T01:43:31.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:30 smithi203 ceph-mon[36345]: Health check failed: 0 slow ops, oldest one blocked for 181 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:33.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:32 smithi046 ceph-mon[36108]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 MiB/s wr, 6.33k op/s 2024-03-18T01:43:33.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:32 smithi007 ceph-mon[31996]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 MiB/s wr, 6.33k op/s 2024-03-18T01:43:33.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:32 smithi203 ceph-mon[36345]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 MiB/s wr, 6.33k op/s 2024-03-18T01:43:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:34.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:33 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:34.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:34.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:33 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:34.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:34.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:33 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:35.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:34 smithi046 ceph-mon[36108]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 6.26k op/s 2024-03-18T01:43:35.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:34 smithi007 ceph-mon[31996]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 6.26k op/s 2024-03-18T01:43:35.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:34 smithi203 ceph-mon[36345]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 6.26k op/s 2024-03-18T01:43:36.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:36 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:36.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:36 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:36.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:36 smithi203 ceph-mon[36345]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:36.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:36 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:36.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:36 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:36.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:36 smithi046 ceph-mon[36108]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:36 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:36 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:36.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:36 smithi007 ceph-mon[31996]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:37 smithi203 ceph-mon[36345]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 46 MiB/s wr, 6.44k op/s 2024-03-18T01:43:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:37 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 186 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:37.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:37 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:37.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:37 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:37.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:37 smithi046 ceph-mon[36108]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 46 MiB/s wr, 6.44k op/s 2024-03-18T01:43:37.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:37 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 186 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:37.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:37 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:37.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:37 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:37.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:37 smithi007 ceph-mon[31996]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 46 MiB/s wr, 6.44k op/s 2024-03-18T01:43:37.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:37 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 186 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:37.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:37 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:37.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:37 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:38.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:38 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:38.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:38 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:38.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:38 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:38.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:38 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:38 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:38.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:38 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:39.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:39 smithi203 ceph-mon[36345]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 45 MiB/s wr, 6.32k op/s 2024-03-18T01:43:39.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:39 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:39.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:39 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:39 smithi046 ceph-mon[36108]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 45 MiB/s wr, 6.32k op/s 2024-03-18T01:43:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:39 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:39 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:39 smithi007 ceph-mon[31996]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 45 MiB/s wr, 6.32k op/s 2024-03-18T01:43:39.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:39 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:39.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:39 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:40 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:40 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:40 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:40.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:40 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:40.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:40 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:40.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:40 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:41 smithi046 ceph-mon[36108]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 45 MiB/s wr, 6.37k op/s 2024-03-18T01:43:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:41 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:41 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:41 smithi046 ceph-mon[36108]: Health check update: 5 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:41.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:41 smithi007 ceph-mon[31996]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 45 MiB/s wr, 6.37k op/s 2024-03-18T01:43:41.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:41 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:41.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:41 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:41.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:41 smithi007 ceph-mon[31996]: Health check update: 5 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:41.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:41 smithi203 ceph-mon[36345]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 45 MiB/s wr, 6.37k op/s 2024-03-18T01:43:41.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:41 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:41.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:41 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:41.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:41 smithi203 ceph-mon[36345]: Health check update: 5 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:42 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:42 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:42 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:42.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:42 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:42 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:42 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:42 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:42.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:42 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:42.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:42 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:42.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:42 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:42.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:42 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:42.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:42 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:43.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:43 smithi046 ceph-mon[36108]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 6.42k op/s 2024-03-18T01:43:43.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:43 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:43.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:43 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:43.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:43 smithi007 ceph-mon[31996]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 6.42k op/s 2024-03-18T01:43:43.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:43 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:43.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:43 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:43.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:43 smithi203 ceph-mon[36345]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 MiB/s wr, 6.42k op/s 2024-03-18T01:43:43.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:43 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:43.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:43 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:44 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:44 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:44.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:44 smithi046 ceph-mon[36108]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 43 MiB/s wr, 6.26k op/s 2024-03-18T01:43:44.541 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:44 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:44.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:44 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:44.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:44 smithi203 ceph-mon[36345]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 43 MiB/s wr, 6.26k op/s 2024-03-18T01:43:44.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:44 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:44.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:44 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:44.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:44 smithi007 ceph-mon[31996]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 43 MiB/s wr, 6.26k op/s 2024-03-18T01:43:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:45 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:45 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:45.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:45 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:45.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:45 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:45 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:45 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:46.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:46 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:46.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:46 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:46.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:46 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:46.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:46 smithi007 ceph-mon[31996]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 45 MiB/s wr, 6.51k op/s 2024-03-18T01:43:46.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:46 smithi007 ceph-mon[31996]: Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:46.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:46 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 191 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:46.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:46 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:46.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:46 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:46.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:46 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:46.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:46 smithi203 ceph-mon[36345]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 45 MiB/s wr, 6.51k op/s 2024-03-18T01:43:46.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:46 smithi203 ceph-mon[36345]: Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:46.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:46 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 191 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:46 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:46 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:46 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:46.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:46 smithi046 ceph-mon[36108]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 45 MiB/s wr, 6.51k op/s 2024-03-18T01:43:46.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:46 smithi046 ceph-mon[36108]: Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:46.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:46 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 191 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:48 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:48 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:48.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:48 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:48.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:48 smithi203 ceph-mon[36345]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 41 MiB/s wr, 6.01k op/s 2024-03-18T01:43:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:48 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:48 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:48 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:48.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:48 smithi046 ceph-mon[36108]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 41 MiB/s wr, 6.01k op/s 2024-03-18T01:43:48.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:48 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:48.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:48 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:48.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:48 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:48.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:48 smithi007 ceph-mon[31996]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 41 MiB/s wr, 6.01k op/s 2024-03-18T01:43:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:50 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:50.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:50 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:50.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:50 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:50.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:50 smithi203 ceph-mon[36345]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 6.00k op/s 2024-03-18T01:43:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:50 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:50 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:50 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:50.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:50 smithi046 ceph-mon[36108]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 6.00k op/s 2024-03-18T01:43:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:50 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:50 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:50 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:50.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:50 smithi007 ceph-mon[31996]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 6.00k op/s 2024-03-18T01:43:51.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:51 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:51.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:51 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:51.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:51 smithi203 ceph-mon[36345]: Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:51.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:51 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 196 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:51.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:51 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:51.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:51 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:51.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:51 smithi046 ceph-mon[36108]: Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:51.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:51 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 196 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:51.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:51 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:51.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:51 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:51.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:51 smithi007 ceph-mon[31996]: Health check update: 7 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:43:51.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:51 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 196 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:52 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:52 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:52.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:52 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:52.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:52 smithi046 ceph-mon[36108]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 43 MiB/s wr, 6.24k op/s 2024-03-18T01:43:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:52 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:52 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:52 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:52.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:52 smithi007 ceph-mon[31996]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 43 MiB/s wr, 6.24k op/s 2024-03-18T01:43:52.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:52 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:52.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:52 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:52.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:52 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:52.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:52 smithi203 ceph-mon[36345]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 43 MiB/s wr, 6.24k op/s 2024-03-18T01:43:53.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:43:53.160 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:43:53.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:53 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:53.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:53 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:53.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:53 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:53.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:53 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:53.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.853+0000 7f62c8c00640 1 -- 172.21.15.7:0/2547565581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c410ddc0 msgr2=0x7f62c41101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:53.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.853+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2547565581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c410ddc0 0x7f62c41101b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f62bc01ace0 tx=0x7f62bc03d110 comp rx=0 tx=0).stop 2024-03-18T01:43:53.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.854+0000 7f62c8c00640 1 -- 172.21.15.7:0/2547565581 shutdown_connections 2024-03-18T01:43:53.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.854+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2547565581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c410ddc0 0x7f62c41101b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.854+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2547565581 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62c4074240 0x7f62c40746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.854+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2547565581 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c4075820 0x7f62c4073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.854+0000 7f62c8c00640 1 -- 172.21.15.7:0/2547565581 >> 172.21.15.7:0/2547565581 conn(0x7f62c40fcd60 msgr2=0x7f62c40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:53.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.854+0000 7f62c8c00640 1 -- 172.21.15.7:0/2547565581 shutdown_connections 2024-03-18T01:43:53.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.854+0000 7f62c8c00640 1 -- 172.21.15.7:0/2547565581 wait complete. 2024-03-18T01:43:53.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.855+0000 7f62c8c00640 1 Processor -- start 2024-03-18T01:43:53.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.856+0000 7f62c8c00640 1 -- start start 2024-03-18T01:43:53.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.856+0000 7f62c8c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c4074240 0x7f62c4195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.856+0000 7f62c8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62c4075820 0x7f62c41962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.856+0000 7f62c8c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c410ddc0 0x7f62c419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.856+0000 7f62c8c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41081e0 con 0x7f62c410ddc0 2024-03-18T01:43:53.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.856+0000 7f62c8c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62c4108060 con 0x7f62c4074240 2024-03-18T01:43:53.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.856+0000 7f62c8c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62c4108360 con 0x7f62c4075820 2024-03-18T01:43:53.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.857+0000 7f62c1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62c4075820 0x7f62c41962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.857+0000 7f62c1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62c4075820 0x7f62c41962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38982/0 (socket says 172.21.15.7:38982) 2024-03-18T01:43:53.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.857+0000 7f62c1400640 1 -- 172.21.15.7:0/2958515321 learned_addr learned my addr 172.21.15.7:0/2958515321 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:53.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.857+0000 7f62c1e00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c4074240 0x7f62c4195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.858+0000 7f62c1e00640 1 -- 172.21.15.7:0/2958515321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62c4075820 msgr2=0x7f62c41962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:53.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.858+0000 7f62c1e00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62c4075820 0x7f62c41962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.858+0000 7f62c1e00640 1 -- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c410ddc0 msgr2=0x7f62c419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:53.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.858+0000 7f62c2800640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c410ddc0 0x7f62c419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.858+0000 7f62c1e00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c410ddc0 0x7f62c419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.858+0000 7f62c1e00640 1 -- 172.21.15.7:0/2958515321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c419d510 con 0x7f62c4074240 2024-03-18T01:43:53.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.858+0000 7f62c2800640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c410ddc0 0x7f62c419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:43:53.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.859+0000 7f62c1e00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c4074240 0x7f62c4195d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00d720 tx=0x7f62ac00dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:53.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.859+0000 7f62baa00640 1 -- 172.21.15.7:0/2958515321 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac017070 con 0x7f62c4074240 2024-03-18T01:43:53.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.859+0000 7f62baa00640 1 -- 172.21.15.7:0/2958515321 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62ac00de30 con 0x7f62c4074240 2024-03-18T01:43:53.863 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.859+0000 7f62baa00640 1 -- 172.21.15.7:0/2958515321 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac0047a0 con 0x7f62c4074240 2024-03-18T01:43:53.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.859+0000 7f62c8c00640 1 -- 172.21.15.7:0/2958515321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c419d800 con 0x7f62c4074240 2024-03-18T01:43:53.864 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.860+0000 7f62c8c00640 1 -- 172.21.15.7:0/2958515321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c419dc40 con 0x7f62c4074240 2024-03-18T01:43:53.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.865+0000 7f62cb98a740 1 Processor -- start 2024-03-18T01:43:53.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.865+0000 7f62baa00640 1 -- 172.21.15.7:0/2958515321 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f62ac004940 con 0x7f62c4074240 2024-03-18T01:43:53.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.865+0000 7f62cb98a740 1 -- start start 2024-03-18T01:43:53.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.866+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c06c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.866 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.866+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 0x55b7b2c0bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.866+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c0e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.866+0000 7f62cb98a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b7b2b6b1f0 con 0x55b7b2c07200 2024-03-18T01:43:53.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.866+0000 7f62cb98a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b7b2b13a80 con 0x55b7b2c06810 2024-03-18T01:43:53.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.866+0000 7f62cb98a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b7b2b67290 con 0x55b7b2c0c370 2024-03-18T01:43:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.866+0000 7f62c2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c06c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.867+0000 7f62c2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c06c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:60010/0 (socket says 172.21.15.7:60010) 2024-03-18T01:43:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.867+0000 7f62c2800640 1 -- 172.21.15.7:0/2366356267 learned_addr learned my addr 172.21.15.7:0/2366356267 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.867+0000 7f62c1400640 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c0e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.869 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.869+0000 7f62c1e00640 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 0x55b7b2c0bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.869+0000 7f62c1400640 1 -- 172.21.15.7:0/2366356267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 msgr2=0x55b7b2c06c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:53.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.869+0000 7f62c1400640 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c06c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.870 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.869+0000 7f62c1400640 1 -- 172.21.15.7:0/2366356267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 msgr2=0x55b7b2c0bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.869+0000 7f62c1400640 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 0x55b7b2c0bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.869+0000 7f62c1400640 1 -- 172.21.15.7:0/2366356267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b29497d0 con 0x55b7b2c0c370 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.870+0000 7f62c2800640 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c06c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.870+0000 7f62baa00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62a80a2880 0x7f62a80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.870+0000 7f62baa00640 1 -- 172.21.15.7:0/2958515321 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f62ac0cead0 con 0x7f62c4074240 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.870+0000 7f62c1400640 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c0e800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f62b400bc80 tx=0x7f62b40107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97099d4a9446fd81 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.870+0000 7f62c1e00640 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 0x55b7b2c0bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.870+0000 7f62c2800640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62a80a2880 0x7f62a80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.870+0000 7f62b3000640 1 -- 172.21.15.7:0/2366356267 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b4005220 con 0x55b7b2c0c370 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.871+0000 7f62b3000640 1 -- 172.21.15.7:0/2366356267 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62b400be30 con 0x55b7b2c0c370 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.871+0000 7f62b3000640 1 -- 172.21.15.7:0/2366356267 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b4011790 con 0x55b7b2c0c370 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.872+0000 7f62cb98a740 1 -- 172.21.15.7:0/2366356267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 msgr2=0x55b7b2c0e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:53.873 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.872+0000 7f62cb98a740 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c0e800 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f62b400bc80 tx=0x7f62b40107a0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.872+0000 7f62c2800640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62a80a2880 0x7f62a80a4d40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f62bc01aa70 tx=0x7f62bc03f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.872+0000 7f62cb98a740 1 -- 172.21.15.7:0/2366356267 shutdown_connections 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.872+0000 7f62cb98a740 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c0e800 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.872+0000 7f62cb98a740 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 0x55b7b2c0bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.872+0000 7f62cb98a740 1 --2- 172.21.15.7:0/2366356267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c06c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.872+0000 7f62cb98a740 1 -- 172.21.15.7:0/2366356267 >> 172.21.15.7:0/2366356267 conn(0x55b7b2c01d80 msgr2=0x55b7b2c02180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.873+0000 7f62cb98a740 1 -- 172.21.15.7:0/2366356267 shutdown_connections 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.873+0000 7f62cb98a740 1 -- 172.21.15.7:0/2366356267 wait complete. 2024-03-18T01:43:53.874 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.873+0000 7f62cb98a740 1 -- start start 2024-03-18T01:43:53.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.874+0000 7f62cb98a740 1 Processor -- start 2024-03-18T01:43:53.875 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.875+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c06810 0x55b7b2c10d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.876 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.875+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c07200 0x55b7b2c112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c182c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62cb98a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55b7b2c0f320 con 0x55b7b2c06810 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62cb98a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55b7b2b26ed0 con 0x55b7b2c07200 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62cb98a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b7b2c0f4a0 con 0x55b7b2c0c370 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62c1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c07200 0x55b7b2c112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62c2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c182c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62c2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c182c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:38994/0 (socket says 172.21.15.7:38994) 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62c2800640 1 -- 172.21.15.7:0/4244064557 learned_addr learned my addr 172.21.15.7:0/4244064557 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.876+0000 7f62c1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c07200 0x55b7b2c112c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:60018/0 (socket says 172.21.15.7:60018) 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.877+0000 7f62c2800640 1 -- 172.21.15.7:0/4244064557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c07200 msgr2=0x55b7b2c112c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.877+0000 7f62c2800640 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c07200 0x55b7b2c112c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.878 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.877+0000 7f62c2800640 1 -- 172.21.15.7:0/4244064557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c06810 msgr2=0x55b7b2c10d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:53.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.877+0000 7f62c2800640 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c06810 0x55b7b2c10d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.877+0000 7f62c2800640 1 -- 172.21.15.7:0/4244064557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7b287f3f0 con 0x55b7b2c0c370 2024-03-18T01:43:53.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.877+0000 7f62cb98a740 10 client.0 _ll_register_callbacks cb 0x55b7b2b62220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:43:53.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.877+0000 7f62cb98a740 1 -- 172.21.15.7:0/4244064557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62b400be30 con 0x55b7b2c0c370 2024-03-18T01:43:53.879 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.878+0000 7f62c2800640 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c182c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f62bc048040 tx=0x7f62bc01a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a3803303dd420a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:53.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.878+0000 7f62a7400640 10 client.26206 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:43:53.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.879+0000 7f62a7400640 1 -- 172.21.15.7:0/4244064557 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc04a040 con 0x55b7b2c0c370 2024-03-18T01:43:53.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.880+0000 7f62a7400640 1 -- 172.21.15.7:0/4244064557 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f62bc004240 con 0x55b7b2c0c370 2024-03-18T01:43:53.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.880+0000 7f62a7400640 1 -- 172.21.15.7:0/4244064557 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f62bc004440 con 0x55b7b2c0c370 2024-03-18T01:43:53.880 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.880+0000 7f62cb98a740 10 client.26206 fetch_fsmap learned FSMap version 55 2024-03-18T01:43:53.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.880+0000 7f62cb98a740 1 -- 172.21.15.7:0/4244064557 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=55}) -- 0x55b7b283f0f0 con 0x55b7b2c0c370 2024-03-18T01:43:53.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.881+0000 7f62a7400640 1 -- 172.21.15.7:0/4244064557 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 55) ==== 8366+0+0 (secure 0 0 0) 0x7f62bc0045e0 con 0x55b7b2c0c370 2024-03-18T01:43:53.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.881+0000 7f62cb98a740 10 client.26206 fetch_fsmap finished waiting for FSMap version 55 2024-03-18T01:43:53.881 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.881+0000 7f62cb98a740 10 client.26206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:43:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.881+0000 7f62cb98a740 20 client.26206 populate_metadata read hostname 'smithi007' 2024-03-18T01:43:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.881+0000 7f62cb98a740 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b7b2c1ca40 0x55b7b2c1ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.881+0000 7f62cb98a740 4 client.26206 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:43:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.881+0000 7f62cb98a740 1 -- 172.21.15.7:0/4244064557 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7b2b7ebe0 con 0x55b7b2c1ca40 2024-03-18T01:43:53.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.882+0000 7f62c1e00640 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b7b2c1ca40 0x55b7b2c1ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.883+0000 7f62c1e00640 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b7b2c1ca40 0x55b7b2c1ee90 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3f178f1d4b435d8 server_cookie=95366b297a939b18 in_seq=0 out_seq=0 2024-03-18T01:43:53.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.883+0000 7f62a7400640 10 client.26206 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:43:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.884+0000 7f62a7400640 1 -- 172.21.15.7:0/4244064557 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b7b2b7ebe0 con 0x55b7b2c1ca40 2024-03-18T01:43:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.884+0000 7f62a7400640 10 client.26206 handle_command_reply: tid=0 2024-03-18T01:43:53.884 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.884+0000 7f62cb98a740 1 client.26206 shutdown 2024-03-18T01:43:53.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62cb98a740 1 -- 172.21.15.7:0/4244064557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 msgr2=0x55b7b2c182c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:53.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62cb98a740 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c182c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f62bc048040 tx=0x7f62bc01a5d0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.886 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62cb98a740 1 -- 172.21.15.7:0/4244064557 shutdown_connections 2024-03-18T01:43:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62cb98a740 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b7b2c1ca40 0x55b7b2c1ee90 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62cb98a740 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c182c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62cb98a740 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c07200 0x55b7b2c112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62a7400640 0 client.26206 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:43:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62cb98a740 1 --2- 172.21.15.7:0/4244064557 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c06810 0x55b7b2c10d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.886+0000 7f62cb98a740 1 -- 172.21.15.7:0/4244064557 >> 172.21.15.7:0/4244064557 conn(0x55b7b2c01d80 msgr2=0x55b7b2c00ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:53.887 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.887+0000 7f62cb98a740 1 -- 172.21.15.7:0/4244064557 shutdown_connections 2024-03-18T01:43:53.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.887+0000 7f62cb98a740 1 -- 172.21.15.7:0/4244064557 wait complete. 2024-03-18T01:43:53.888 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.887+0000 7f62cb98a740 20 client.26206 trim_cache size 0 max 16384 2024-03-18T01:43:53.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:53 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:54.055 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:53 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:54.060 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.925+0000 7f62cb98a740 1 Processor -- start 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.926+0000 7f62cb98a740 1 -- start start 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.926+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c11220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.926+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c07200 0x55b7b2c18190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.926+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c0c370 0x55b7b2c00630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.927+0000 7f62cb98a740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41a80b0 con 0x55b7b2c0c370 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.927+0000 7f62cb98a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41a9c20 con 0x55b7b2c06810 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.927+0000 7f62cb98a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41a87b0 con 0x55b7b2c07200 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.927+0000 7f62c2800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c0c370 0x55b7b2c00630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.928+0000 7f62c2800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c0c370 0x55b7b2c00630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:36258/0 (socket says 172.21.15.7:36258) 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.928+0000 7f62c1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c07200 0x55b7b2c18190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.928+0000 7f62c2800640 1 -- 172.21.15.7:0/1544617789 learned_addr learned my addr 172.21.15.7:0/1544617789 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:54.061 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.928+0000 7f62c1e00640 1 --2- 172.21.15.7:0/1544617789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c11220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.928+0000 7f62c2800640 1 -- 172.21.15.7:0/1544617789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c07200 msgr2=0x55b7b2c18190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.929+0000 7f62c2800640 1 --2- 172.21.15.7:0/1544617789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c07200 0x55b7b2c18190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.929+0000 7f62c2800640 1 -- 172.21.15.7:0/1544617789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 msgr2=0x55b7b2c11220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.929+0000 7f62c2800640 1 --2- 172.21.15.7:0/1544617789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c11220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.929+0000 7f62c2800640 1 -- 172.21.15.7:0/1544617789 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b2bfc370 con 0x55b7b2c0c370 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.929+0000 7f62c2800640 1 --2- 172.21.15.7:0/1544617789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c0c370 0x55b7b2c00630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f62bc03f010 tx=0x7f62bc04f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d136df158e2c8f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.930+0000 7f62b2600640 1 -- 172.21.15.7:0/1544617789 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc072070 con 0x55b7b2c0c370 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.930+0000 7f62b2600640 1 -- 172.21.15.7:0/1544617789 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62b400be30 con 0x55b7b2c0c370 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.930+0000 7f62b2600640 1 -- 172.21.15.7:0/1544617789 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc04a070 con 0x55b7b2c0c370 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.931+0000 7f62cb98a740 1 -- 172.21.15.7:0/1544617789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c0c370 msgr2=0x55b7b2c00630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.931+0000 7f62cb98a740 1 --2- 172.21.15.7:0/1544617789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c0c370 0x55b7b2c00630 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f62bc03f010 tx=0x7f62bc04f000 comp rx=0 tx=0).stop 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.932+0000 7f62cb98a740 1 -- 172.21.15.7:0/1544617789 shutdown_connections 2024-03-18T01:43:54.062 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.932+0000 7f62cb98a740 1 --2- 172.21.15.7:0/1544617789 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c0c370 0x55b7b2c00630 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.932+0000 7f62cb98a740 1 --2- 172.21.15.7:0/1544617789 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c07200 0x55b7b2c18190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.932+0000 7f62cb98a740 1 --2- 172.21.15.7:0/1544617789 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c11220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.932+0000 7f62cb98a740 1 -- 172.21.15.7:0/1544617789 >> 172.21.15.7:0/1544617789 conn(0x55b7b2c01d80 msgr2=0x55b7b2c10b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.937+0000 7f62cb98a740 1 -- 172.21.15.7:0/1544617789 shutdown_connections 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.937+0000 7f62cb98a740 1 -- 172.21.15.7:0/1544617789 wait complete. 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.939+0000 7f62cb98a740 1 -- start start 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.939+0000 7f62cb98a740 1 Processor -- start 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.940+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c15f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.940+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 0x55b7b2c16440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.940+0000 7f62cb98a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c17450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.940+0000 7f62c1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c15f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.940+0000 7f62c1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c15f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:60046/0 (socket says 172.21.15.7:60046) 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62c1e00640 1 -- 172.21.15.7:0/228963600 learned_addr learned my addr 172.21.15.7:0/228963600 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:43:54.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62c2800640 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c17450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41a9550 con 0x55b7b2c07200 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41a8e80 con 0x55b7b2c06810 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41a6090 con 0x55b7b2c0c370 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62cb98a740 10 client.0 _ll_register_callbacks cb 0x7f62bc048070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62c1e00640 1 -- 172.21.15.7:0/228963600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 msgr2=0x55b7b2c17450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62c1e00640 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c17450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62c1e00640 1 -- 172.21.15.7:0/228963600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 msgr2=0x55b7b2c16440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62c1e00640 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 0x55b7b2c16440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.941+0000 7f62c1e00640 1 -- 172.21.15.7:0/228963600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7b2c079f0 con 0x55b7b2c06810 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.943+0000 7f62c2800640 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c17450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.943+0000 7f62c1e00640 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c15f00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f62ac09b040 tx=0x7f62ac005240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29d5e9980e5efc26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.943+0000 7f62a7400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:43:54.064 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.944+0000 7f62a7400640 1 -- 172.21.15.7:0/228963600 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac09aa10 con 0x55b7b2c06810 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.944+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62b400be30 con 0x55b7b2c06810 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.944+0000 7f62a7400640 1 -- 172.21.15.7:0/228963600 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f62ac097720 con 0x55b7b2c06810 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.944+0000 7f62a7400640 1 -- 172.21.15.7:0/228963600 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f62ac0b5020 con 0x55b7b2c06810 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.945+0000 7f62cb98a740 10 client.26240 fetch_fsmap learned FSMap version 55 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.945+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55b7b2c832b0 con 0x55b7b2c06810 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.945+0000 7f62a7400640 1 -- 172.21.15.7:0/228963600 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62b400be30 con 0x55b7b2c06810 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.945+0000 7f62a7400640 1 -- 172.21.15.7:0/228963600 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 55) ==== 8366+0+0 (secure 0 0 0) 0x7f62ac0a33c0 con 0x55b7b2c06810 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.947+0000 7f62cb98a740 10 client.26240 fetch_fsmap finished waiting for FSMap version 55 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.947+0000 7f62cb98a740 10 client.26240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.947+0000 7f62cb98a740 20 client.26240 populate_metadata read hostname 'smithi007' 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.947+0000 7f62cb98a740 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b7b2c83b00 0x55b7b2c85ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.948+0000 7f62cb98a740 4 client.26240 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.948+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b7b2b78730 con 0x55b7b2c83b00 2024-03-18T01:43:54.065 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.948+0000 7f62c1400640 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b7b2c83b00 0x55b7b2c85ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.950+0000 7f62c1400640 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b7b2c83b00 0x55b7b2c85ef0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3608a0ce08fa602 server_cookie=390d02d12634c2e1 in_seq=0 out_seq=0 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.950+0000 7f62a7400640 10 client.26240 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.950+0000 7f62a7400640 1 -- 172.21.15.7:0/228963600 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55b7b2b78730 con 0x55b7b2c83b00 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.950+0000 7f62a7400640 10 client.26240 handle_command_reply: tid=0 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.950+0000 7f62cb98a740 1 client.26240 shutdown 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 msgr2=0x55b7b2c15f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c15f00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f62ac09b040 tx=0x7f62ac005240 comp rx=0 tx=0).stop 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 shutdown_connections 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55b7b2c83b00 0x55b7b2c85ef0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b7b2c0c370 0x55b7b2c17450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55b7b2c07200 0x55b7b2c16440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62a7400640 0 client.26240 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 --2- 172.21.15.7:0/228963600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b7b2c06810 0x55b7b2c15f00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 >> 172.21.15.7:0/228963600 conn(0x55b7b2c01d80 msgr2=0x55b7b2c00630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 shutdown_connections 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.951+0000 7f62cb98a740 1 -- 172.21.15.7:0/228963600 wait complete. 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.952+0000 7f62cb98a740 20 client.26240 trim_cache size 0 max 16384 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (16 inodes in the stack)", 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:43:54.067 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:43:54.068 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:43:54.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:43:54.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:43:54.068 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.953+0000 7f62c8c00640 1 -- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62a80a2880 msgr2=0x7f62a80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.953+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62a80a2880 0x7f62a80a4d40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f62bc01aa70 tx=0x7f62bc03f040 comp rx=0 tx=0).stop 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.953+0000 7f62c8c00640 1 -- 172.21.15.7:0/2958515321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c4074240 msgr2=0x7f62c4195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.953+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c4074240 0x7f62c4195d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00d720 tx=0x7f62ac00dbe0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.960+0000 7f62c8c00640 1 -- 172.21.15.7:0/2958515321 shutdown_connections 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.960+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62a80a2880 0x7f62a80a4d40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.960+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62c410ddc0 0x7f62c419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.960+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62c4075820 0x7f62c41962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.960+0000 7f62c8c00640 1 --2- 172.21.15.7:0/2958515321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62c4074240 0x7f62c4195d80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:43:54.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.960+0000 7f62c8c00640 1 -- 172.21.15.7:0/2958515321 >> 172.21.15.7:0/2958515321 conn(0x7f62c40fcd60 msgr2=0x7f62c4101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:43:54.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.960+0000 7f62c8c00640 1 -- 172.21.15.7:0/2958515321 shutdown_connections 2024-03-18T01:43:54.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:43:53.960+0000 7f62c8c00640 1 -- 172.21.15.7:0/2958515321 wait complete. 2024-03-18T01:43:54.070 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:43:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:54 smithi046 ceph-mon[36108]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 MiB/s wr, 6.86k op/s 2024-03-18T01:43:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:54 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:54 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:54 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:54.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:54 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:54 smithi007 ceph-mon[31996]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 MiB/s wr, 6.86k op/s 2024-03-18T01:43:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:54 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:54 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:54.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:54 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:54.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:54 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:54.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:54 smithi203 ceph-mon[36345]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 47 MiB/s wr, 6.86k op/s 2024-03-18T01:43:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:54 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:43:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:54 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:54 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:54.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:54 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:55 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:55.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:55 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:55.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:55 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:55.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:55 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:55.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:55 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:55.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:55 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:56.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:56 smithi046 ceph-mon[36108]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 44 MiB/s wr, 6.30k op/s 2024-03-18T01:43:56.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:56 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:56.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:56 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:56.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:56 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:56.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:56 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 207 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:56.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:56 smithi007 ceph-mon[31996]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 44 MiB/s wr, 6.30k op/s 2024-03-18T01:43:56.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:56 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:56.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:56 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:56.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:56 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:56.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:56 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 207 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:56.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:56 smithi203 ceph-mon[36345]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 44 MiB/s wr, 6.30k op/s 2024-03-18T01:43:56.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:56 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:43:56.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:56 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:43:56.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:56 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:56.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:56 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 207 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:43:57.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:57 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:57.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:57 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:57.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:57 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:57.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:57 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:57 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:43:57.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:57 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:58.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:58 smithi046 ceph-mon[36108]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 42 MiB/s wr, 6.01k op/s 2024-03-18T01:43:58.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:58 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:58.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:58 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:58.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:58 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:58.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:58 smithi007 ceph-mon[31996]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 42 MiB/s wr, 6.01k op/s 2024-03-18T01:43:58.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:58 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:58.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:58 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:58.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:58 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:58.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:58 smithi203 ceph-mon[36345]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 42 MiB/s wr, 6.01k op/s 2024-03-18T01:43:58.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:58 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:43:58.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:58 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:43:58.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:58 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:59.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:59 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:59.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:43:59 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:43:59.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:59 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:43:59.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:43:59 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:00.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:59 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:00.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:43:59 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:00.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:00 smithi007 ceph-mon[31996]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 41 MiB/s wr, 6.03k op/s 2024-03-18T01:44:00.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:00 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:00.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:00 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:00.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:00 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:00 smithi203 ceph-mon[36345]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 41 MiB/s wr, 6.03k op/s 2024-03-18T01:44:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:00 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:00.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:00 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:00.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:00 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:01.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:00 smithi046 ceph-mon[36108]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 41 MiB/s wr, 6.03k op/s 2024-03-18T01:44:01.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:00 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:01.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:00 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:01.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:00 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:01.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:01 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:01.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:01 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:02.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:01 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:02.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:01 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:02.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:01 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:02.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:01 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:02.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:02 smithi203 ceph-mon[36345]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 MiB/s wr, 6.41k op/s 2024-03-18T01:44:02.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:02 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 213 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:02.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:02 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:02.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:02 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:02.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:02 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:02.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:02 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:03.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:02 smithi046 ceph-mon[36108]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 MiB/s wr, 6.41k op/s 2024-03-18T01:44:03.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:02 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 213 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:03.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:02 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:03.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:02 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:03.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:02 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:03.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:02 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:03.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:02 smithi007 ceph-mon[31996]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 MiB/s wr, 6.41k op/s 2024-03-18T01:44:03.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:02 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 213 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:03.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:02 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:03.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:02 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:03.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:02 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:03.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:02 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:04.861 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:04 smithi007 ceph-mon[31996]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 43 MiB/s wr, 6.34k op/s 2024-03-18T01:44:04.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:04 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:04.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:04 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:04.862 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:04 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:04.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:04 smithi203 ceph-mon[36345]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 43 MiB/s wr, 6.34k op/s 2024-03-18T01:44:04.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:04 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:04.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:04 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:04.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:04 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:05.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:04 smithi046 ceph-mon[36108]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 43 MiB/s wr, 6.34k op/s 2024-03-18T01:44:05.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:04 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:05.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:04 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:05.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:04 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:05.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:05 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:05.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:05 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:05 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:06.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:05 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:06.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:05 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:06.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:05 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:06.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:06 smithi203 ceph-mon[36345]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 4.93k op/s 2024-03-18T01:44:06.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:06 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:06.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:06 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:06 smithi046 ceph-mon[36108]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 4.93k op/s 2024-03-18T01:44:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:06 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:07.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:06 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:07.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:06 smithi007 ceph-mon[31996]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 33 MiB/s wr, 4.93k op/s 2024-03-18T01:44:07.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:06 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:07.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:06 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:07 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:07 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:08.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:07 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:44:08.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:07 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:08.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:07 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:08.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:07 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:44:08.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:07 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:08.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:07 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:08.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:07 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:44:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:08 smithi046 ceph-mon[36108]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 38 MiB/s wr, 5.73k op/s 2024-03-18T01:44:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:08 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 218 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:08 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:09.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:08 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:08 smithi007 ceph-mon[31996]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 38 MiB/s wr, 5.73k op/s 2024-03-18T01:44:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:08 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 218 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:08 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:09.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:08 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:09.209 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:08 smithi203 ceph-mon[36345]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 38 MiB/s wr, 5.73k op/s 2024-03-18T01:44:09.209 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:08 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 218 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:09.209 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:08 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:09.209 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:08 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:10.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi046 to 175.2M 2024-03-18T01:44:10.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: Unable to set osd_memory_target on smithi046 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-18T01:44:10.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:09 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi046 to 175.2M 2024-03-18T01:44:10.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: Unable to set osd_memory_target on smithi046 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-18T01:44:10.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:09 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi046 to 175.2M 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: Unable to set osd_memory_target on smithi046 to 183745228: error parsing value: Value '183745228' is below minimum 939524096 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:10.494 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:09 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi007 to 177.8M 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: Unable to set osd_memory_target on smithi007 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 39 MiB/s wr, 5.96k op/s 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: Adjusting osd_memory_target on smithi203 to 174.1M 2024-03-18T01:44:11.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: Unable to set osd_memory_target on smithi203 to 182607667: error parsing value: Value '182607667' is below minimum 939524096 2024-03-18T01:44:11.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:44:11.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:44:11.021 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:10 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:44:11.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi007 to 177.8M 2024-03-18T01:44:11.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: Unable to set osd_memory_target on smithi007 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T01:44:11.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 39 MiB/s wr, 5.96k op/s 2024-03-18T01:44:11.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: Adjusting osd_memory_target on smithi203 to 174.1M 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: Unable to set osd_memory_target on smithi203 to 182607667: error parsing value: Value '182607667' is below minimum 939524096 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:44:11.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:10 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:44:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi007 to 177.8M 2024-03-18T01:44:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: Unable to set osd_memory_target on smithi007 to 186497740: error parsing value: Value '186497740' is below minimum 939524096 2024-03-18T01:44:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 39 MiB/s wr, 5.96k op/s 2024-03-18T01:44:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:11.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T01:44:11.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: Adjusting osd_memory_target on smithi203 to 174.1M 2024-03-18T01:44:11.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: Unable to set osd_memory_target on smithi203 to 182607667: error parsing value: Value '182607667' is below minimum 939524096 2024-03-18T01:44:11.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:44:11.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:44:11.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:10 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:44:12.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:12 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:12.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:12 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:12.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:12 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:12.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:12 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:12.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:12 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:12.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:12 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:13.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:13 smithi203 ceph-mon[36345]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 39 MiB/s wr, 5.97k op/s 2024-03-18T01:44:13.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:13 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:13.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:13 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:13.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:13 smithi046 ceph-mon[36108]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 39 MiB/s wr, 5.97k op/s 2024-03-18T01:44:13.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:13 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:13.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:13 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:13 smithi007 ceph-mon[31996]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 39 MiB/s wr, 5.97k op/s 2024-03-18T01:44:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:13 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:13 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:14.411 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:14 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:14.412 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:14 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:14.412 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:14 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:14 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:14 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:14 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:14 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:14 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:14 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:15.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:15 smithi046 ceph-mon[36108]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 38 MiB/s wr, 5.79k op/s 2024-03-18T01:44:15.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:15 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:15.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:15 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:15.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:15 smithi007 ceph-mon[31996]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 38 MiB/s wr, 5.79k op/s 2024-03-18T01:44:15.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:15 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:15.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:15 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:15 smithi203 ceph-mon[36345]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 38 MiB/s wr, 5.79k op/s 2024-03-18T01:44:15.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:15 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:15.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:15 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:16 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:16.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:16 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:16.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:16 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:16.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:16 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:16 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:16 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:16 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:16 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:16 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:16 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:16 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:16 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 224 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:17.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:17 smithi046 ceph-mon[36108]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 MiB/s wr, 5.16k op/s 2024-03-18T01:44:17.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:17 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:17.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:17 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:17.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:17 smithi007 ceph-mon[31996]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 MiB/s wr, 5.16k op/s 2024-03-18T01:44:17.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:17 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:17.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:17 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:17.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:17 smithi203 ceph-mon[36345]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 MiB/s wr, 5.16k op/s 2024-03-18T01:44:17.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:17 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:17.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:17 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:18 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:18 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:18 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:18.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:18 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:18.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:18 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:18.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:18 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:18 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:18 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:18 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:19.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:19 smithi046 ceph-mon[36108]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 42 MiB/s wr, 6.41k op/s 2024-03-18T01:44:19.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:19 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:19.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:19 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:19 smithi007 ceph-mon[31996]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 42 MiB/s wr, 6.41k op/s 2024-03-18T01:44:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:19 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:19 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:19.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:19 smithi203 ceph-mon[36345]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 42 MiB/s wr, 6.41k op/s 2024-03-18T01:44:19.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:19 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:19.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:19 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:20.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:20 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:20.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:20 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:20.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:20 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:20.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:20 smithi046 ceph-mon[36108]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 39 MiB/s wr, 6.05k op/s 2024-03-18T01:44:20.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:20 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:20.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:20 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:20.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:20 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:20.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:20 smithi007 ceph-mon[31996]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 39 MiB/s wr, 6.05k op/s 2024-03-18T01:44:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:20 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:20 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:20.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:20 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:20.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:20 smithi203 ceph-mon[36345]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 39 MiB/s wr, 6.05k op/s 2024-03-18T01:44:21.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:21 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:21.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:21 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:21.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:21 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 229 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:21.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:21 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:21.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:21 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:21.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:21 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 229 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:21.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:21 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:21.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:21 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:21.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:21 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 229 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:22.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:22 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:22.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:22 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:22.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:22 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:22.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:22 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:22.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:22 smithi007 ceph-mon[31996]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 37 MiB/s wr, 5.71k op/s 2024-03-18T01:44:22.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:22 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:22.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:22 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:22.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:22 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:22.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:22 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:22.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:22 smithi203 ceph-mon[36345]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 37 MiB/s wr, 5.71k op/s 2024-03-18T01:44:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:22 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:22 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:22 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:22 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:22 smithi046 ceph-mon[36108]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 37 MiB/s wr, 5.71k op/s 2024-03-18T01:44:23.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:23 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:23.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:23 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:23.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:23 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:23.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:23 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:23.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:23 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:23.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:23 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:24.070 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:44:24.071 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:44:24.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:24 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:24.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:24 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:24.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:24 smithi203 ceph-mon[36345]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 38 MiB/s wr, 5.98k op/s 2024-03-18T01:44:24.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:24 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:24.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:24 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:24.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:24 smithi046 ceph-mon[36108]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 38 MiB/s wr, 5.98k op/s 2024-03-18T01:44:24.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:24 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:24.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:24 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:24.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:24 smithi007 ceph-mon[31996]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 38 MiB/s wr, 5.98k op/s 2024-03-18T01:44:25.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:25 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:25.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:25 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:25.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:25 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:25.665 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:25 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:25.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:25 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:25.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:25 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:26 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:26 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:26 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:26.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:26 smithi203 ceph-mon[36345]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 37 MiB/s wr, 6.16k op/s 2024-03-18T01:44:26.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:26 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 234 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:26 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:26 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:26 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:26 smithi046 ceph-mon[36108]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 37 MiB/s wr, 6.16k op/s 2024-03-18T01:44:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:26 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 234 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:26 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:26 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:26 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:26 smithi007 ceph-mon[31996]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 37 MiB/s wr, 6.16k op/s 2024-03-18T01:44:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:26 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 234 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:26.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 -- 172.21.15.7:0/3470959198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 msgr2=0x7fa7b0100320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 --2- 172.21.15.7:0/3470959198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 0x7fa7b0100320 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa798009900 tx=0x7fa79802bc40 comp rx=0 tx=0).stop 2024-03-18T01:44:26.815 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 -- 172.21.15.7:0/3470959198 shutdown_connections 2024-03-18T01:44:26.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 --2- 172.21.15.7:0/3470959198 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7b0101200 0x7fa7b0101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 --2- 172.21.15.7:0/3470959198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7b0100860 0x7fa7b0100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 --2- 172.21.15.7:0/3470959198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 0x7fa7b0100320 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 -- 172.21.15.7:0/3470959198 >> 172.21.15.7:0/3470959198 conn(0x7fa7b00fbac0 msgr2=0x7fa7b00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:26.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 -- 172.21.15.7:0/3470959198 shutdown_connections 2024-03-18T01:44:26.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.814+0000 7fa7afe00640 1 -- 172.21.15.7:0/3470959198 wait complete. 2024-03-18T01:44:26.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.816+0000 7fa7afe00640 1 Processor -- start 2024-03-18T01:44:26.817 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.816+0000 7fa7afe00640 1 -- start start 2024-03-18T01:44:26.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.816+0000 7fa7afe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 0x7fa7b0194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.816+0000 7fa7afe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7b0100860 0x7fa7b0195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.816+0000 7fa7afe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7b0101200 0x7fa7b019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.818 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7afe00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b010f470 con 0x7fa7b0101200 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7afe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b010f2f0 con 0x7fa7b0100860 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7afe00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b010f5f0 con 0x7fa7b00fff00 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7aea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 0x7fa7b0194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7aea00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 0x7fa7b0194f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44112/0 (socket says 172.21.15.7:44112) 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7aea00640 1 -- 172.21.15.7:0/1281988041 learned_addr learned my addr 172.21.15.7:0/1281988041 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7af400640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7b0101200 0x7fa7b019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7b0100860 0x7fa7b0195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7aea00640 1 -- 172.21.15.7:0/1281988041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7b0100860 msgr2=0x7fa7b0195490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7aea00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7b0100860 0x7fa7b0195490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7aea00640 1 -- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7b0101200 msgr2=0x7fa7b019c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7aea00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7b0101200 0x7fa7b019c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.819 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.817+0000 7fa7aea00640 1 -- 172.21.15.7:0/1281988041 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b019cab0 con 0x7fa7b00fff00 2024-03-18T01:44:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.818+0000 7fa7aea00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 0x7fa7b0194f50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa798009230 tx=0x7fa798005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.818+0000 7fa7a7400640 1 -- 172.21.15.7:0/1281988041 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798041070 con 0x7fa7b00fff00 2024-03-18T01:44:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.818+0000 7fa7afe00640 1 -- 172.21.15.7:0/1281988041 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b019cd40 con 0x7fa7b00fff00 2024-03-18T01:44:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.818+0000 7fa7a7400640 1 -- 172.21.15.7:0/1281988041 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7980054e0 con 0x7fa7b00fff00 2024-03-18T01:44:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.818+0000 7fa7a7400640 1 -- 172.21.15.7:0/1281988041 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa798034440 con 0x7fa7b00fff00 2024-03-18T01:44:26.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.819+0000 7fa7afe00640 1 -- 172.21.15.7:0/1281988041 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b01aeb70 con 0x7fa7b00fff00 2024-03-18T01:44:26.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.828+0000 7fa7b881b740 1 Processor -- start 2024-03-18T01:44:26.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.828+0000 7fa7b881b740 1 -- start start 2024-03-18T01:44:26.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5758a10 0x5574c5758e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c575df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7a7400640 1 -- 172.21.15.7:0/1281988041 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fa798044070 con 0x7fa7b00fff00 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c5760a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7b881b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5574c56bd1f0 con 0x5574c5758a10 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7b881b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5574c5665a80 con 0x5574c5759400 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7b881b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5574c56b9290 con 0x5574c575e570 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7af400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5758a10 0x5574c5758e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7a7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c5760a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.829+0000 7fa7a7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c5760a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44122/0 (socket says 172.21.15.7:44122) 2024-03-18T01:44:26.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.830+0000 7fa7af400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5758a10 0x5574c5758e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:57094/0 (socket says 172.21.15.7:57094) 2024-03-18T01:44:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.830+0000 7fa7a7e00640 1 -- 172.21.15.7:0/3070506384 learned_addr learned my addr 172.21.15.7:0/3070506384 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.830+0000 7fa7af400640 1 -- 172.21.15.7:0/3070506384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 msgr2=0x5574c5760a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.830+0000 7fa7af400640 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c5760a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.830+0000 7fa7af400640 1 -- 172.21.15.7:0/3070506384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 msgr2=0x5574c575df00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:44:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.831+0000 7fa7aea00640 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c575df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.831+0000 7fa7af400640 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c575df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.831+0000 7fa7af400640 1 -- 172.21.15.7:0/3070506384 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c549b7d0 con 0x5574c5758a10 2024-03-18T01:44:26.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.831+0000 7fa7aea00640 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c575df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:44:26.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.832+0000 7fa7a7400640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7940a2900 0x7fa7940a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.832+0000 7fa7a7400640 1 -- 172.21.15.7:0/1281988041 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa7980f0be0 con 0x7fa7b00fff00 2024-03-18T01:44:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.833+0000 7fa7af400640 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5758a10 0x5574c5758e30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8007d90 tx=0x7fa7a800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3999bfdad39f44b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:26.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.833+0000 7fa7aea00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7940a2900 0x7fa7940a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.834+0000 7fa79fe00640 1 -- 172.21.15.7:0/3070506384 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8014070 con 0x5574c5758a10 2024-03-18T01:44:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.834+0000 7fa79fe00640 1 -- 172.21.15.7:0/3070506384 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a8003e50 con 0x5574c5758a10 2024-03-18T01:44:26.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.834+0000 7fa79fe00640 1 -- 172.21.15.7:0/3070506384 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8005020 con 0x5574c5758a10 2024-03-18T01:44:26.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.835+0000 7fa7b881b740 1 -- 172.21.15.7:0/3070506384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5758a10 msgr2=0x5574c5758e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.835+0000 7fa7b881b740 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5758a10 0x5574c5758e30 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8007d90 tx=0x7fa7a800e720 comp rx=0 tx=0).stop 2024-03-18T01:44:26.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.836+0000 7fa7b881b740 1 -- 172.21.15.7:0/3070506384 shutdown_connections 2024-03-18T01:44:26.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.836+0000 7fa7b881b740 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c5760a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.836+0000 7fa7b881b740 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c575df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.836+0000 7fa7b881b740 1 --2- 172.21.15.7:0/3070506384 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5758a10 0x5574c5758e30 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.836+0000 7fa7b881b740 1 -- 172.21.15.7:0/3070506384 >> 172.21.15.7:0/3070506384 conn(0x5574c5753f80 msgr2=0x5574c5754380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:26.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.837+0000 7fa7b881b740 1 -- 172.21.15.7:0/3070506384 shutdown_connections 2024-03-18T01:44:26.838 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.837+0000 7fa7b881b740 1 -- 172.21.15.7:0/3070506384 wait complete. 2024-03-18T01:44:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.837+0000 7fa7b881b740 1 -- start start 2024-03-18T01:44:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.838+0000 7fa7b881b740 1 Processor -- start 2024-03-18T01:44:26.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.838+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5758a10 0x5574c5774090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.839+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c57745d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.841 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.839+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c577b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.839+0000 7fa7af400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5758a10 0x5574c5774090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7af400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5758a10 0x5574c5774090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:54052/0 (socket says 172.21.15.7:54052) 2024-03-18T01:44:26.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7b881b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5574c5772530 con 0x5574c5759400 2024-03-18T01:44:26.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7af400640 1 -- 172.21.15.7:0/1110168626 learned_addr learned my addr 172.21.15.7:0/1110168626 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:26.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7aea00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7940a2900 0x7fa7940a4dc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa7980be130 tx=0x7fa7980dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:26.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c57745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5574c5678ed0 con 0x5574c5758a10 2024-03-18T01:44:26.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5574c57726b0 con 0x5574c575e570 2024-03-18T01:44:26.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7b881b740 10 client.0 _ll_register_callbacks cb 0x5574c56b4220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:44:26.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.840+0000 7fa7af400640 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c577b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.841+0000 7fa7a7e00640 1 -- 172.21.15.7:0/1110168626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 msgr2=0x5574c577b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:26.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.841+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c577b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.841+0000 7fa7a7e00640 1 -- 172.21.15.7:0/1110168626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5758a10 msgr2=0x5574c5774090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:26.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.841+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5758a10 0x5574c5774090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.841+0000 7fa7a7e00640 1 -- 172.21.15.7:0/1110168626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c53d13f0 con 0x5574c5759400 2024-03-18T01:44:26.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.842+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c57745d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x5574c5754ef0 tx=0x7fa7a0012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ab72f04160b03c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:26.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.842+0000 7fa79cc00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:44:26.844 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.843+0000 7fa79cc00640 1 -- 172.21.15.7:0/1110168626 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a001f060 con 0x5574c5759400 2024-03-18T01:44:26.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.843+0000 7fa79cc00640 1 -- 172.21.15.7:0/1110168626 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa7a00107e0 con 0x5574c5759400 2024-03-18T01:44:26.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.843+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7a8003e50 con 0x5574c5759400 2024-03-18T01:44:26.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.844+0000 7fa79cc00640 1 -- 172.21.15.7:0/1110168626 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fa7a00109e0 con 0x5574c5759400 2024-03-18T01:44:26.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.844+0000 7fa7b881b740 10 client.15957 fetch_fsmap learned FSMap version 84 2024-03-18T01:44:26.845 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.844+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=4+}) -- 0x5574c53910f0 con 0x5574c5759400 2024-03-18T01:44:26.849 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.848+0000 7fa79cc00640 1 -- 172.21.15.7:0/1110168626 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a8003e50 con 0x5574c5759400 2024-03-18T01:44:26.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.848+0000 7fa79cc00640 1 -- 172.21.15.7:0/1110168626 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 84) ==== 8370+0+0 (secure 0 0 0) 0x7fa7a0019030 con 0x5574c5759400 2024-03-18T01:44:26.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.848+0000 7fa7b881b740 10 client.15957 fetch_fsmap finished waiting for FSMap version 84 2024-03-18T01:44:26.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.849+0000 7fa7b881b740 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:44:26.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.849+0000 7fa7b881b740 20 client.15957 populate_metadata read hostname 'smithi007' 2024-03-18T01:44:26.850 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.849+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5574c577fe20 0x5574c57822c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.849+0000 7fa7b881b740 4 client.15957 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:44:26.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.849+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574c56d0be0 con 0x5574c577fe20 2024-03-18T01:44:26.851 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.850+0000 7fa7af400640 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5574c577fe20 0x5574c57822c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.851+0000 7fa7af400640 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5574c577fe20 0x5574c57822c0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d3176bf0010d247 server_cookie=184f7aa9b2becdbb in_seq=0 out_seq=0 2024-03-18T01:44:26.852 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.851+0000 7fa79cc00640 10 client.15957 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:44:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.852+0000 7fa79cc00640 1 -- 172.21.15.7:0/1110168626 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5574c56d0be0 con 0x5574c577fe20 2024-03-18T01:44:26.853 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.852+0000 7fa79cc00640 10 client.15957 handle_command_reply: tid=0 2024-03-18T01:44:26.854 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.853+0000 7fa7b881b740 1 client.15957 shutdown 2024-03-18T01:44:26.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.854+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 msgr2=0x5574c57745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.855 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.854+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c57745d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x5574c5754ef0 tx=0x7fa7a0012820 comp rx=0 tx=0).stop 2024-03-18T01:44:26.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.855+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 shutdown_connections 2024-03-18T01:44:26.856 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.855+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5574c577fe20 0x5574c57822c0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.855+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c575e570 0x5574c577b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.855+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c57745d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.855+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1110168626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5758a10 0x5574c5774090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.855+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 >> 172.21.15.7:0/1110168626 conn(0x5574c5753f80 msgr2=0x5574c5753110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:26.857 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.855+0000 7fa79cc00640 0 client.15957 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:44:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.857+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 shutdown_connections 2024-03-18T01:44:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.857+0000 7fa7b881b740 1 -- 172.21.15.7:0/1110168626 wait complete. 2024-03-18T01:44:26.858 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.857+0000 7fa7b881b740 20 client.15957 trim_cache size 0 max 16384 2024-03-18T01:44:26.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7b881b740 1 Processor -- start 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7b881b740 1 -- start start 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c575aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c575af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c575e570 0x5574c577b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7b881b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b01b06c0 con 0x5574c5759400 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7b881b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b01b2260 con 0x5574c575e570 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7b881b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b01b0dc0 con 0x5574c5758a10 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.905+0000 7fa7af400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c575aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.906+0000 7fa7af400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c575aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44136/0 (socket says 172.21.15.7:44136) 2024-03-18T01:44:26.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.906+0000 7fa7af400640 1 -- 172.21.15.7:0/1778839519 learned_addr learned my addr 172.21.15.7:0/1778839519 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.906+0000 7fa7af400640 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c575e570 0x5574c577b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.906+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c575af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.906+0000 7fa7af400640 1 -- 172.21.15.7:0/1778839519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c575e570 msgr2=0x5574c577b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.906+0000 7fa7af400640 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c575e570 0x5574c577b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.907+0000 7fa7af400640 1 -- 172.21.15.7:0/1778839519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 msgr2=0x5574c575af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.907+0000 7fa7af400640 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c575af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.907+0000 7fa7af400640 1 -- 172.21.15.7:0/1778839519 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c577fc00 con 0x5574c5758a10 2024-03-18T01:44:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.907+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c575af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:44:26.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.907+0000 7fa7af400640 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c575aa40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8000f80 tx=0x7fa7a8004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a90d2858998f10e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.908+0000 7fa79f400640 1 -- 172.21.15.7:0/1778839519 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8023040 con 0x5574c5758a10 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.908+0000 7fa79f400640 1 -- 172.21.15.7:0/1778839519 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a8024040 con 0x5574c5758a10 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.908+0000 7fa79f400640 1 -- 172.21.15.7:0/1778839519 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8014040 con 0x5574c5758a10 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.909+0000 7fa7b881b740 1 -- 172.21.15.7:0/1778839519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 msgr2=0x5574c575aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.909+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c575aa40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8000f80 tx=0x7fa7a8004360 comp rx=0 tx=0).stop 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.910+0000 7fa7b881b740 1 -- 172.21.15.7:0/1778839519 shutdown_connections 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.910+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c575e570 0x5574c577b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.910+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c5759400 0x5574c575af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.910+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1778839519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c575aa40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.912 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.910+0000 7fa7b881b740 1 -- 172.21.15.7:0/1778839519 >> 172.21.15.7:0/1778839519 conn(0x5574c5753f80 msgr2=0x5574c5774090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.913+0000 7fa7b881b740 1 -- 172.21.15.7:0/1778839519 shutdown_connections 2024-03-18T01:44:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.913+0000 7fa7b881b740 1 -- 172.21.15.7:0/1778839519 wait complete. 2024-03-18T01:44:26.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.913+0000 7fa7b881b740 1 -- start start 2024-03-18T01:44:26.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.914+0000 7fa7b881b740 1 Processor -- start 2024-03-18T01:44:26.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.915+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c5778c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.916+0000 7fa7af400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c5778c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.916+0000 7fa7af400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c5778c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44150/0 (socket says 172.21.15.7:44150) 2024-03-18T01:44:26.918 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c5779140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7b881b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c575e570 0x5574c577a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7af400640 1 -- 172.21.15.7:0/1284228883 learned_addr learned my addr 172.21.15.7:0/1284228883 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b01b1b90 con 0x5574c575e570 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b01b1490 con 0x5574c5759400 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b00732b0 con 0x5574c5758a10 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7b881b740 10 client.0 _ll_register_callbacks cb 0x7fa7a000b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c5779140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7a7e00640 1 -- 172.21.15.7:0/1284228883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 msgr2=0x5574c5778c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:26.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c5778c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7a7e00640 1 -- 172.21.15.7:0/1284228883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c575e570 msgr2=0x5574c577a150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.917+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c575e570 0x5574c577a150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.918+0000 7fa7a7e00640 1 -- 172.21.15.7:0/1284228883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c57549f0 con 0x5574c5759400 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.918+0000 7fa79cc00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.918+0000 7fa7a7e00640 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c5779140 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0023420 tx=0x7fa7a0012e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b1062b6d540dd5f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.919+0000 7fa7af400640 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c5778c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.919+0000 7fa79cc00640 1 -- 172.21.15.7:0/1284228883 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0010b80 con 0x5574c5759400 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.919+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7a8024040 con 0x5574c5759400 2024-03-18T01:44:26.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.919+0000 7fa79cc00640 1 -- 172.21.15.7:0/1284228883 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa7a0029070 con 0x5574c5759400 2024-03-18T01:44:26.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.920+0000 7fa79cc00640 1 -- 172.21.15.7:0/1284228883 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=84) ==== 24+0+0 (secure 0 0 0) 0x7fa7a0010b80 con 0x5574c5759400 2024-03-18T01:44:26.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.921+0000 7fa7b881b740 10 client.26255 fetch_fsmap learned FSMap version 84 2024-03-18T01:44:26.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.921+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=84,monmap=4+}) -- 0x5574c577b8f0 con 0x5574c5759400 2024-03-18T01:44:26.923 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.922+0000 7fa79cc00640 1 -- 172.21.15.7:0/1284228883 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a8024040 con 0x5574c5759400 2024-03-18T01:44:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.922+0000 7fa79cc00640 1 -- 172.21.15.7:0/1284228883 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 84) ==== 8370+0+0 (secure 0 0 0) 0x7fa7a0019040 con 0x5574c5759400 2024-03-18T01:44:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.922+0000 7fa7b881b740 10 client.26255 fetch_fsmap finished waiting for FSMap version 84 2024-03-18T01:44:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.922+0000 7fa7b881b740 10 client.26255 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:44:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.922+0000 7fa7b881b740 20 client.26255 populate_metadata read hostname 'smithi007' 2024-03-18T01:44:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.923+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5574c57e68b0 0x5574c57e8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.923+0000 7fa7b881b740 4 client.26255 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:44:26.924 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.923+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5574c56ca730 con 0x5574c57e68b0 2024-03-18T01:44:26.926 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.925+0000 7fa7af400640 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5574c57e68b0 0x5574c57e8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:26.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.927+0000 7fa7af400640 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5574c57e68b0 0x5574c57e8cd0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=315eac0fb76a60eb server_cookie=b40986f66ad3ec9e in_seq=0 out_seq=0 2024-03-18T01:44:26.928 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.927+0000 7fa79cc00640 10 client.26255 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:44:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.927+0000 7fa79cc00640 1 -- 172.21.15.7:0/1284228883 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x5574c56ca730 con 0x5574c57e68b0 2024-03-18T01:44:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.927+0000 7fa79cc00640 10 client.26255 handle_command_reply: tid=0 2024-03-18T01:44:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.928+0000 7fa7b881b740 1 client.26255 shutdown 2024-03-18T01:44:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.928+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 msgr2=0x5574c5779140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.929 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.928+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c5779140 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0023420 tx=0x7fa7a0012e20 comp rx=0 tx=0).stop 2024-03-18T01:44:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.930+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 shutdown_connections 2024-03-18T01:44:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.930+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5574c57e68b0 0x5574c57e8cd0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.930+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5574c575e570 0x5574c577a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.930+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5574c5759400 0x5574c5779140 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.930+0000 7fa79cc00640 0 client.26255 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:44:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.930+0000 7fa7b881b740 1 --2- 172.21.15.7:0/1284228883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5574c5758a10 0x5574c5778c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.930+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 >> 172.21.15.7:0/1284228883 conn(0x5574c5753f80 msgr2=0x5574c5774050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:26.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.930+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 shutdown_connections 2024-03-18T01:44:26.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.933+0000 7fa7b881b740 1 -- 172.21.15.7:0/1284228883 wait complete. 2024-03-18T01:44:26.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.934+0000 7fa7b881b740 20 client.26255 trim_cache size 0 max 16384 2024-03-18T01:44:26.939 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:44:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (16 inodes in the stack)", 2024-03-18T01:44:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:44:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:44:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:44:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:44:26.939 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:44:26.940 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:44:26.940 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:44:26.940 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:44:26.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 -- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7940a2900 msgr2=0x7fa7940a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7940a2900 0x7fa7940a4dc0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa7980be130 tx=0x7fa7980dd040 comp rx=0 tx=0).stop 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 -- 172.21.15.7:0/1281988041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 msgr2=0x7fa7b0194f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 0x7fa7b0194f50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa798009230 tx=0x7fa798005dd0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 -- 172.21.15.7:0/1281988041 shutdown_connections 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fa7940a2900 0x7fa7940a4dc0 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa7980be130 tx=0x7fa7980dd040 comp rx=0 tx=0).stop 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fa7b0101200 0x7fa7b019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa7b0100860 0x7fa7b0195490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 --2- 172.21.15.7:0/1281988041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7b00fff00 0x7fa7b0194f50 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa798009230 tx=0x7fa798005dd0 comp rx=0 tx=0).stop 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.939+0000 7fa7afe00640 1 -- 172.21.15.7:0/1281988041 >> 172.21.15.7:0/1281988041 conn(0x7fa7b00fbac0 msgr2=0x7fa7b00fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.940+0000 7fa7afe00640 1 -- 172.21.15.7:0/1281988041 shutdown_connections 2024-03-18T01:44:26.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:26.940+0000 7fa7afe00640 1 -- 172.21.15.7:0/1281988041 wait complete. 2024-03-18T01:44:26.972 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:44:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:28 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:28 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:28 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:28 smithi203 ceph-mon[36345]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 6.65k op/s 2024-03-18T01:44:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:28 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:28 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:28 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:28 smithi046 ceph-mon[36108]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 6.65k op/s 2024-03-18T01:44:28.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:28 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:28.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:28 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:28.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:28 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:28.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:28 smithi007 ceph-mon[31996]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 6.65k op/s 2024-03-18T01:44:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:30 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:30 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:30 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:30 smithi046 ceph-mon[36108]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 39 MiB/s wr, 6.70k op/s 2024-03-18T01:44:30.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:30 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:30.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:30 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:30 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:30 smithi007 ceph-mon[31996]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 39 MiB/s wr, 6.70k op/s 2024-03-18T01:44:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:30 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:30 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:30 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:30 smithi203 ceph-mon[36345]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 39 MiB/s wr, 6.70k op/s 2024-03-18T01:44:31.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:31 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:31.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:31 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:31.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:31 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:31.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:31 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:31.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:31 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:31.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:31 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:31.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:31 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:31.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:31 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:31.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:31 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:32 smithi046 ceph-mon[36108]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 36 MiB/s wr, 6.20k op/s 2024-03-18T01:44:32.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:32 smithi007 ceph-mon[31996]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 36 MiB/s wr, 6.20k op/s 2024-03-18T01:44:32.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:32 smithi203 ceph-mon[36345]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 36 MiB/s wr, 6.20k op/s 2024-03-18T01:44:33.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:33.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:33 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:33.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:33.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:33 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:33.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:33.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:33 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:34.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:34 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:34.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:34 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:34.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:34 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:34.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:34 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:34.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:34 smithi046 ceph-mon[36108]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 39 MiB/s wr, 6.51k op/s 2024-03-18T01:44:34.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:34 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:34.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:34 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:34.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:34 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:34.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:34 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:34.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:34 smithi007 ceph-mon[31996]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 39 MiB/s wr, 6.51k op/s 2024-03-18T01:44:34.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:34 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:34.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:34 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:34.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:34 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:34.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:34 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:34.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:34 smithi203 ceph-mon[36345]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 39 MiB/s wr, 6.51k op/s 2024-03-18T01:44:35.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:35 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:35.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:35 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:35.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:35 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:35.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:35 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:35.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:35 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:35.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:35 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:36.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:36 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:36.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:36 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:36.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:36 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:36.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:36 smithi046 ceph-mon[36108]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 37 MiB/s wr, 6.22k op/s 2024-03-18T01:44:36.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:36 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 244 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:36.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:36 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:36.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:36 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:36.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:36 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:36.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:36 smithi007 ceph-mon[31996]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 37 MiB/s wr, 6.22k op/s 2024-03-18T01:44:36.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:36 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 244 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:36.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:36 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:36.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:36 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:36.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:36 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:36.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:36 smithi203 ceph-mon[36345]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 37 MiB/s wr, 6.22k op/s 2024-03-18T01:44:36.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:36 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 244 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:37.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:37 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:37.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:37 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:37.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:37 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:37.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:37 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:37.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:37 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:37.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:37 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:38.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:38 smithi203 ceph-mon[36345]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 34 MiB/s wr, 5.58k op/s 2024-03-18T01:44:38.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:38 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:38.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:38 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:38.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:38 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:38 smithi046 ceph-mon[36108]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 34 MiB/s wr, 5.58k op/s 2024-03-18T01:44:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:38 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:38 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:38 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:38 smithi007 ceph-mon[31996]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 34 MiB/s wr, 5.58k op/s 2024-03-18T01:44:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:38 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:38 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:38 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:39.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:39 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:39.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:39 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:40.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:39 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:40.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:39 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:40.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:39 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:40.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:39 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:40 smithi046 ceph-mon[36108]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 32 MiB/s wr, 5.36k op/s 2024-03-18T01:44:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:40 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:40 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:40 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:41.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:40 smithi007 ceph-mon[31996]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 32 MiB/s wr, 5.36k op/s 2024-03-18T01:44:41.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:40 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:41.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:40 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:41.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:40 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:40 smithi203 ceph-mon[36345]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 32 MiB/s wr, 5.36k op/s 2024-03-18T01:44:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:40 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:40 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:41.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:40 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:42.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:41 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:42.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:41 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:42.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:41 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:42.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:41 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:42.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:41 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:42.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:41 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:41 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:42.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:41 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:42.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:41 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 250 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:43.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:42 smithi046 ceph-mon[36108]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 33 MiB/s wr, 5.45k op/s 2024-03-18T01:44:43.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:42 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:43.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:42 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:43.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:42 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:43.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:42 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:42 smithi007 ceph-mon[31996]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 33 MiB/s wr, 5.45k op/s 2024-03-18T01:44:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:42 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:42 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:42 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:43.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:42 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:42 smithi203 ceph-mon[36345]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 33 MiB/s wr, 5.45k op/s 2024-03-18T01:44:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:42 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:42 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:43.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:42 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:43.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:42 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:45.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:44 smithi046 ceph-mon[36108]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 34 MiB/s wr, 5.65k op/s 2024-03-18T01:44:45.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:44 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:45.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:44 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:45.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:44 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:44 smithi007 ceph-mon[31996]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 34 MiB/s wr, 5.65k op/s 2024-03-18T01:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:44 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:44 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:45.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:44 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:44 smithi203 ceph-mon[36345]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 34 MiB/s wr, 5.65k op/s 2024-03-18T01:44:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:44 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:44 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:44 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:46.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:45 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:46.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:45 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:46.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:45 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:46.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:45 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:45 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:45 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:47.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:46 smithi046 ceph-mon[36108]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 32 MiB/s wr, 5.34k op/s 2024-03-18T01:44:47.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:46 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:47.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:46 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:47.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:46 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 255 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:47.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:46 smithi007 ceph-mon[31996]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 32 MiB/s wr, 5.34k op/s 2024-03-18T01:44:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:46 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:46 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:47.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:46 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 255 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:46 smithi203 ceph-mon[36345]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 32 MiB/s wr, 5.34k op/s 2024-03-18T01:44:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:46 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:46 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:46 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 255 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:48.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:47 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:48.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:47 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:48.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:47 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:48.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:47 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:48.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:47 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:48.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:47 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:48 smithi007 ceph-mon[31996]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 33 MiB/s wr, 5.71k op/s 2024-03-18T01:44:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:48 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:49.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:48 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:49.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:48 smithi203 ceph-mon[36345]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 33 MiB/s wr, 5.71k op/s 2024-03-18T01:44:49.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:48 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:49.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:48 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:48 smithi046 ceph-mon[36108]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 33 MiB/s wr, 5.71k op/s 2024-03-18T01:44:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:48 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:48 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:51.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:50 smithi007 ceph-mon[31996]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 33 MiB/s wr, 5.65k op/s 2024-03-18T01:44:51.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:50 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:51.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:50 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:51.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:50 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:51.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:50 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:50 smithi203 ceph-mon[36345]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 33 MiB/s wr, 5.65k op/s 2024-03-18T01:44:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:50 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:50 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:50 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:50 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:50 smithi046 ceph-mon[36108]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 33 MiB/s wr, 5.65k op/s 2024-03-18T01:44:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:50 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:50 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:50 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:50 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:52.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:51 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:52.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:51 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:52.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:51 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 261 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:52.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:51 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:52.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:51 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:52.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:51 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 261 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:51 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:51 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:51 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 261 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:53.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:52 smithi203 ceph-mon[36345]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 32 MiB/s wr, 5.54k op/s 2024-03-18T01:44:53.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:52 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:53.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:52 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:53.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:52 smithi046 ceph-mon[36108]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 32 MiB/s wr, 5.54k op/s 2024-03-18T01:44:53.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:52 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:53.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:52 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:53.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:52 smithi007 ceph-mon[31996]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 32 MiB/s wr, 5.54k op/s 2024-03-18T01:44:53.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:52 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:53.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:52 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:54.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:53 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:54.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:53 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:54.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:53 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:54.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:53 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:54.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:53 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:54.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:53 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:55.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:54 smithi203 ceph-mon[36345]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 5.38k op/s 2024-03-18T01:44:55.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:54 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:55.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:54 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:55.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:54 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:54 smithi046 ceph-mon[36108]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 5.38k op/s 2024-03-18T01:44:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:54 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:54 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:54 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:55.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:54 smithi007 ceph-mon[31996]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 31 MiB/s wr, 5.38k op/s 2024-03-18T01:44:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:54 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:54 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:54 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:56.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:55 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:56.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:55 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:55 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:55 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:56.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:55 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:44:56.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:55 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:56.974 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:44:56.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:44:57.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:56 smithi007 ceph-mon[31996]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 5.16k op/s 2024-03-18T01:44:57.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:56 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:57.117 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:56 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:56 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:57.118 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:56 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 266 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:57.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:56 smithi203 ceph-mon[36345]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 5.16k op/s 2024-03-18T01:44:57.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:56 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:57.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:56 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:57.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:56 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:57.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:56 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 266 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:56 smithi046 ceph-mon[36108]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 29 MiB/s wr, 5.16k op/s 2024-03-18T01:44:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:56 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:44:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:56 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:44:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:56 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:56 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 266 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:44:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.321+0000 7f9c70800640 1 -- 172.21.15.7:0/794941641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 msgr2=0x7f9c68103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:57.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.321+0000 7f9c70800640 1 --2- 172.21.15.7:0/794941641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 0x7f9c68103e40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009ca0 tx=0x7f9c6402c100 comp rx=0 tx=0).stop 2024-03-18T01:44:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.321+0000 7f9c70800640 1 -- 172.21.15.7:0/794941641 shutdown_connections 2024-03-18T01:44:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.321+0000 7f9c70800640 1 --2- 172.21.15.7:0/794941641 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c681055d0 0x7f9c6810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.321+0000 7f9c70800640 1 --2- 172.21.15.7:0/794941641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c68104c10 0x7f9c68105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.321+0000 7f9c70800640 1 --2- 172.21.15.7:0/794941641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 0x7f9c68103e40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.321+0000 7f9c70800640 1 -- 172.21.15.7:0/794941641 >> 172.21.15.7:0/794941641 conn(0x7f9c680fef90 msgr2=0x7f9c681013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:57.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.324+0000 7f9c70800640 1 -- 172.21.15.7:0/794941641 shutdown_connections 2024-03-18T01:44:57.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.324+0000 7f9c70800640 1 -- 172.21.15.7:0/794941641 wait complete. 2024-03-18T01:44:57.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.326+0000 7f9c70800640 1 Processor -- start 2024-03-18T01:44:57.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.326+0000 7f9c70800640 1 -- start start 2024-03-18T01:44:57.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.327+0000 7f9c70800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 0x7f9c681959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c70800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c68104c10 0x7f9c68195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c70800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c681055d0 0x7f9c6819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c70800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6810f6f0 con 0x7f9c68104c10 2024-03-18T01:44:57.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c70800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6810f570 con 0x7f9c681055d0 2024-03-18T01:44:57.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c70800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6810f870 con 0x7f9c68103a20 2024-03-18T01:44:57.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c6de00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 0x7f9c681959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.329 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c6de00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 0x7f9c681959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44490/0 (socket says 172.21.15.7:44490) 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c6de00640 1 -- 172.21.15.7:0/39617023 learned_addr learned my addr 172.21.15.7:0/39617023 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.328+0000 7f9c6e800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c681055d0 0x7f9c6819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.329+0000 7f9c6de00640 1 -- 172.21.15.7:0/39617023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c681055d0 msgr2=0x7f9c6819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.329+0000 7f9c6de00640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c681055d0 0x7f9c6819cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.329+0000 7f9c6de00640 1 -- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c68104c10 msgr2=0x7f9c68195f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.329+0000 7f9c6d400640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c68104c10 0x7f9c68195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.329+0000 7f9c6de00640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c68104c10 0x7f9c68195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.329+0000 7f9c6de00640 1 -- 172.21.15.7:0/39617023 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c681ae5f0 con 0x7f9c68103a20 2024-03-18T01:44:57.330 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.329+0000 7f9c6d400640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c68104c10 0x7f9c68195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:44:57.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.331+0000 7f9c6de00640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 0x7f9c681959f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9c64000f80 tx=0x7f9c64035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:57.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.331+0000 7f9c62a00640 1 -- 172.21.15.7:0/39617023 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64039040 con 0x7f9c68103a20 2024-03-18T01:44:57.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.331+0000 7f9c70800640 1 -- 172.21.15.7:0/39617023 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c681ae880 con 0x7f9c68103a20 2024-03-18T01:44:57.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.331+0000 7f9c62a00640 1 -- 172.21.15.7:0/39617023 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c64044070 con 0x7f9c68103a20 2024-03-18T01:44:57.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.331+0000 7f9c62a00640 1 -- 172.21.15.7:0/39617023 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64004610 con 0x7f9c68103a20 2024-03-18T01:44:57.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.333+0000 7f9c70800640 1 -- 172.21.15.7:0/39617023 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c681aee10 con 0x7f9c68103a20 2024-03-18T01:44:57.340 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.340+0000 7f9c62a00640 1 -- 172.21.15.7:0/39617023 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9c64003970 con 0x7f9c68103a20 2024-03-18T01:44:57.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.341+0000 7f9c7356b740 1 Processor -- start 2024-03-18T01:44:57.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.343+0000 7f9c62a00640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9c500a2800 0x7f9c500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.343+0000 7f9c62a00640 1 -- 172.21.15.7:0/39617023 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9c640f1490 con 0x7f9c68103a20 2024-03-18T01:44:57.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.344+0000 7f9c7356b740 1 -- start start 2024-03-18T01:44:57.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.344+0000 7f9c6e800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9c500a2800 0x7f9c500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.344+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439e8810 0x5644439e8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.344+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e9200 0x5644439edd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.344+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439ee370 0x5644439f0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.344+0000 7f9c7356b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x56444394d1f0 con 0x5644439ee370 2024-03-18T01:44:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.344+0000 7f9c7356b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5644438f5a80 con 0x5644439e8810 2024-03-18T01:44:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.344+0000 7f9c7356b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564443949290 con 0x5644439e9200 2024-03-18T01:44:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e9200 0x5644439edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e9200 0x5644439edd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44498/0 (socket says 172.21.15.7:44498) 2024-03-18T01:44:57.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6d400640 1 -- 172.21.15.7:0/2069842654 learned_addr learned my addr 172.21.15.7:0/2069842654 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:57.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6e800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9c500a2800 0x7f9c500a4cc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800b710 tx=0x7f9c580041b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:57.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6e800640 1 --2- 172.21.15.7:0/2069842654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439ee370 0x5644439f0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6d400640 1 -- 172.21.15.7:0/2069842654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439e8810 msgr2=0x5644439e8c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:44:57.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6d400640 1 --2- 172.21.15.7:0/2069842654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439e8810 0x5644439e8c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6d400640 1 -- 172.21.15.7:0/2069842654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439ee370 msgr2=0x5644439f0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:57.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6d400640 1 --2- 172.21.15.7:0/2069842654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439ee370 0x5644439f0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.345+0000 7f9c6d400640 1 -- 172.21.15.7:0/2069842654 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56444372b7d0 con 0x5644439e9200 2024-03-18T01:44:57.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.347+0000 7f9c6d400640 1 --2- 172.21.15.7:0/2069842654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e9200 0x5644439edd00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00a9d0 tx=0x7f9c5c00ae90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=867cb76931ea14d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:57.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.347+0000 7f9c57000640 1 -- 172.21.15.7:0/2069842654 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c015070 con 0x5644439e9200 2024-03-18T01:44:57.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.348+0000 7f9c57000640 1 -- 172.21.15.7:0/2069842654 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c5c00cc60 con 0x5644439e9200 2024-03-18T01:44:57.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.348+0000 7f9c57000640 1 -- 172.21.15.7:0/2069842654 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c0105b0 con 0x5644439e9200 2024-03-18T01:44:57.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.349+0000 7f9c7356b740 1 -- 172.21.15.7:0/2069842654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e9200 msgr2=0x5644439edd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:57.349 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.349+0000 7f9c7356b740 1 --2- 172.21.15.7:0/2069842654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e9200 0x5644439edd00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c00a9d0 tx=0x7f9c5c00ae90 comp rx=0 tx=0).stop 2024-03-18T01:44:57.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.350+0000 7f9c7356b740 1 -- 172.21.15.7:0/2069842654 shutdown_connections 2024-03-18T01:44:57.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.350+0000 7f9c7356b740 1 --2- 172.21.15.7:0/2069842654 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439ee370 0x5644439f0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.350+0000 7f9c7356b740 1 --2- 172.21.15.7:0/2069842654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e9200 0x5644439edd00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.350+0000 7f9c7356b740 1 --2- 172.21.15.7:0/2069842654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439e8810 0x5644439e8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.351 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.350+0000 7f9c7356b740 1 -- 172.21.15.7:0/2069842654 >> 172.21.15.7:0/2069842654 conn(0x5644439e3d80 msgr2=0x5644439e4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:57.354 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.354+0000 7f9c7356b740 1 -- 172.21.15.7:0/2069842654 shutdown_connections 2024-03-18T01:44:57.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.354+0000 7f9c7356b740 1 -- 172.21.15.7:0/2069842654 wait complete. 2024-03-18T01:44:57.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.355+0000 7f9c7356b740 1 -- start start 2024-03-18T01:44:57.355 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.355+0000 7f9c7356b740 1 Processor -- start 2024-03-18T01:44:57.356 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.356+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e8810 0x5644439f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.357 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.357+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439e9200 0x5644439f3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.357+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439ee370 0x5644439fa360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.358 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.357+0000 7f9c7356b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5644439f1320 con 0x5644439e8810 2024-03-18T01:44:57.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.357+0000 7f9c7356b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564443908ed0 con 0x5644439e9200 2024-03-18T01:44:57.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.357+0000 7f9c7356b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5644439f14a0 con 0x5644439ee370 2024-03-18T01:44:57.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.357+0000 7f9c7356b740 10 client.0 _ll_register_callbacks cb 0x564443944220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:44:57.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.358+0000 7f9c6e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439ee370 0x5644439fa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.358+0000 7f9c6e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439ee370 0x5644439fa360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44510/0 (socket says 172.21.15.7:44510) 2024-03-18T01:44:57.359 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.358+0000 7f9c6e800640 1 -- 172.21.15.7:0/4059212616 learned_addr learned my addr 172.21.15.7:0/4059212616 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:57.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.358+0000 7f9c6e800640 1 -- 172.21.15.7:0/4059212616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439e9200 msgr2=0x5644439f3320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:44:57.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.358+0000 7f9c6e800640 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439e9200 0x5644439f3320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.358+0000 7f9c6e800640 1 -- 172.21.15.7:0/4059212616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e8810 msgr2=0x5644439f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:57.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.358+0000 7f9c6e800640 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e8810 0x5644439f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.358+0000 7f9c6e800640 1 -- 172.21.15.7:0/4059212616 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644436613f0 con 0x5644439ee370 2024-03-18T01:44:57.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.359+0000 7f9c6e800640 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439ee370 0x5644439fa360 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9c58003eb0 tx=0x7f9c58004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64c07df3201970d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:57.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.359+0000 7f9c4b400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:44:57.360 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.359+0000 7f9c4b400640 1 -- 172.21.15.7:0/4059212616 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5801d610 con 0x5644439ee370 2024-03-18T01:44:57.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.360+0000 7f9c4b400640 1 -- 172.21.15.7:0/4059212616 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9c5801d7b0 con 0x5644439ee370 2024-03-18T01:44:57.361 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.361+0000 7f9c7356b740 1 -- 172.21.15.7:0/4059212616 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c5c00cc60 con 0x5644439ee370 2024-03-18T01:44:57.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.362+0000 7f9c4b400640 1 -- 172.21.15.7:0/4059212616 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f9c5801d9b0 con 0x5644439ee370 2024-03-18T01:44:57.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.362+0000 7f9c7356b740 10 client.26239 fetch_fsmap learned FSMap version 110 2024-03-18T01:44:57.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.362+0000 7f9c7356b740 1 -- 172.21.15.7:0/4059212616 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x5644436210f0 con 0x5644439ee370 2024-03-18T01:44:57.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.362+0000 7f9c4b400640 1 -- 172.21.15.7:0/4059212616 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c5c00cc60 con 0x5644439ee370 2024-03-18T01:44:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.363+0000 7f9c4b400640 1 -- 172.21.15.7:0/4059212616 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 110) ==== 8366+0+0 (secure 0 0 0) 0x7f9c5800f3c0 con 0x5644439ee370 2024-03-18T01:44:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.363+0000 7f9c7356b740 10 client.26239 fetch_fsmap finished waiting for FSMap version 110 2024-03-18T01:44:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.363+0000 7f9c7356b740 10 client.26239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:44:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.363+0000 7f9c7356b740 20 client.26239 populate_metadata read hostname 'smithi007' 2024-03-18T01:44:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.363+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5644439ff170 0x564443a01610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.363+0000 7f9c7356b740 4 client.26239 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:44:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.363+0000 7f9c7356b740 1 -- 172.21.15.7:0/4059212616 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564443960be0 con 0x5644439ff170 2024-03-18T01:44:57.364 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.364+0000 7f9c6de00640 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5644439ff170 0x564443a01610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.368+0000 7f9c6de00640 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5644439ff170 0x564443a01610 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8ee71e15935c882 server_cookie=5f5a444dc10d10f4 in_seq=0 out_seq=0 2024-03-18T01:44:57.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.368+0000 7f9c4b400640 10 client.26239 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:44:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.370+0000 7f9c4b400640 1 -- 172.21.15.7:0/4059212616 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564443960be0 con 0x5644439ff170 2024-03-18T01:44:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.370+0000 7f9c4b400640 10 client.26239 handle_command_reply: tid=0 2024-03-18T01:44:57.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.370+0000 7f9c7356b740 1 client.26239 shutdown 2024-03-18T01:44:57.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 -- 172.21.15.7:0/4059212616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439ee370 msgr2=0x5644439fa360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:57.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439ee370 0x5644439fa360 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9c58003eb0 tx=0x7f9c58004730 comp rx=0 tx=0).stop 2024-03-18T01:44:57.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 -- 172.21.15.7:0/4059212616 shutdown_connections 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5644439ff170 0x564443a01610 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439ee370 0x5644439fa360 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439e9200 0x5644439f3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4059212616 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e8810 0x5644439f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 -- 172.21.15.7:0/4059212616 >> 172.21.15.7:0/4059212616 conn(0x5644439e3d80 msgr2=0x5644439e2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c4b400640 0 client.26239 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 -- 172.21.15.7:0/4059212616 shutdown_connections 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.371+0000 7f9c7356b740 1 -- 172.21.15.7:0/4059212616 wait complete. 2024-03-18T01:44:57.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.372+0000 7f9c7356b740 20 client.26239 trim_cache size 0 max 16384 2024-03-18T01:44:57.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.414+0000 7f9c7356b740 1 Processor -- start 2024-03-18T01:44:57.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.414+0000 7f9c7356b740 1 -- start start 2024-03-18T01:44:57.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.415+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.415 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.415+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439fa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.415+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.415+0000 7f9c7356b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9c681b9260 con 0x5644439e9200 2024-03-18T01:44:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.415+0000 7f9c7356b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9c681badd0 con 0x5644439ee370 2024-03-18T01:44:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.415+0000 7f9c7356b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9c681b9930 con 0x5644439e8810 2024-03-18T01:44:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.416+0000 7f9c6de00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.416+0000 7f9c6de00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:44514/0 (socket says 172.21.15.7:44514) 2024-03-18T01:44:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.416+0000 7f9c6de00640 1 -- 172.21.15.7:0/2752185914 learned_addr learned my addr 172.21.15.7:0/2752185914 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.416+0000 7f9c6e800640 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.417+0000 7f9c6de00640 1 -- 172.21.15.7:0/2752185914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 msgr2=0x5644439f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.417+0000 7f9c6d400640 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439fa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.417+0000 7f9c6de00640 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.417+0000 7f9c6de00640 1 -- 172.21.15.7:0/2752185914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 msgr2=0x5644439fa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:44:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.417+0000 7f9c6de00640 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439fa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.418+0000 7f9c6de00640 1 -- 172.21.15.7:0/2752185914 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5644439eda50 con 0x5644439e8810 2024-03-18T01:44:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.418+0000 7f9c6e800640 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:44:57.418 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.418+0000 7f9c6d400640 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439fa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:44:57.419 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.419+0000 7f9c6de00640 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f1870 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9c640bd7d0 tx=0x7f9c640095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c11d8a7e706bd8ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.420+0000 7f9c56600640 1 -- 172.21.15.7:0/2752185914 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c640bdad0 con 0x5644439e8810 2024-03-18T01:44:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.420+0000 7f9c56600640 1 -- 172.21.15.7:0/2752185914 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c640c1c20 con 0x5644439e8810 2024-03-18T01:44:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.420+0000 7f9c56600640 1 -- 172.21.15.7:0/2752185914 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c64032920 con 0x5644439e8810 2024-03-18T01:44:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.421+0000 7f9c7356b740 1 -- 172.21.15.7:0/2752185914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 msgr2=0x5644439f1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:57.421 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.421+0000 7f9c7356b740 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f1870 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9c640bd7d0 tx=0x7f9c640095c0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.421+0000 7f9c7356b740 1 -- 172.21.15.7:0/2752185914 shutdown_connections 2024-03-18T01:44:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.422+0000 7f9c7356b740 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.422+0000 7f9c7356b740 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439fa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.422+0000 7f9c7356b740 1 --2- 172.21.15.7:0/2752185914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f1870 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.422+0000 7f9c7356b740 1 -- 172.21.15.7:0/2752185914 >> 172.21.15.7:0/2752185914 conn(0x5644439e3d80 msgr2=0x5644439e2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.422+0000 7f9c7356b740 1 -- 172.21.15.7:0/2752185914 shutdown_connections 2024-03-18T01:44:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.422+0000 7f9c7356b740 1 -- 172.21.15.7:0/2752185914 wait complete. 2024-03-18T01:44:57.423 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.423+0000 7f9c7356b740 1 -- start start 2024-03-18T01:44:57.424 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.423+0000 7f9c7356b740 1 Processor -- start 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.424+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.424+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439f8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.424+0000 7f9c7356b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.424+0000 7f9c7356b740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9c681ba700 con 0x5644439e9200 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.424+0000 7f9c7356b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9c681ba000 con 0x5644439ee370 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.424+0000 7f9c7356b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9c681b7240 con 0x5644439e8810 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.424+0000 7f9c7356b740 10 client.0 _ll_register_callbacks cb 0x7f9c58004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.425+0000 7f9c6e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.425+0000 7f9c6e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:56772/0 (socket says 172.21.15.7:56772) 2024-03-18T01:44:57.426 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.425+0000 7f9c6e800640 1 -- 172.21.15.7:0/4293603127 learned_addr learned my addr 172.21.15.7:0/4293603127 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:44:57.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.426+0000 7f9c6e800640 1 -- 172.21.15.7:0/4293603127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 msgr2=0x5644439f7e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:44:57.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.426+0000 7f9c6e800640 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f7e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.426+0000 7f9c6e800640 1 -- 172.21.15.7:0/4293603127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 msgr2=0x5644439f8360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:44:57.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.426+0000 7f9c6d400640 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439f8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.427 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.426+0000 7f9c6e800640 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439f8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.427+0000 7f9c6e800640 1 -- 172.21.15.7:0/4293603127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644439f9960 con 0x5644439ee370 2024-03-18T01:44:57.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.427+0000 7f9c6e800640 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f9370 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800f900 tx=0x7f9c58003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82fa49006fb66be4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:44:57.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.427+0000 7f9c4b400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:44:57.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.428+0000 7f9c4b400640 1 -- 172.21.15.7:0/4293603127 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c5800fcb0 con 0x5644439ee370 2024-03-18T01:44:57.428 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.428+0000 7f9c7356b740 1 -- 172.21.15.7:0/4293603127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c640c1c20 con 0x5644439ee370 2024-03-18T01:44:57.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.429+0000 7f9c4b400640 1 -- 172.21.15.7:0/4293603127 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9c5800f5d0 con 0x5644439ee370 2024-03-18T01:44:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.430+0000 7f9c4b400640 1 -- 172.21.15.7:0/4293603127 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=110) ==== 24+0+0 (secure 0 0 0) 0x7f9c5801db50 con 0x5644439ee370 2024-03-18T01:44:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.430+0000 7f9c7356b740 10 client.26264 fetch_fsmap learned FSMap version 110 2024-03-18T01:44:57.430 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.430+0000 7f9c7356b740 1 -- 172.21.15.7:0/4293603127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=110,monmap=4+}) -- 0x564443a76960 con 0x5644439ee370 2024-03-18T01:44:57.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.431+0000 7f9c4b400640 1 -- 172.21.15.7:0/4293603127 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c640c1c20 con 0x5644439ee370 2024-03-18T01:44:57.431 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.431+0000 7f9c4b400640 1 -- 172.21.15.7:0/4293603127 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 110) ==== 8366+0+0 (secure 0 0 0) 0x7f9c58004a70 con 0x5644439ee370 2024-03-18T01:44:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.431+0000 7f9c7356b740 10 client.26264 fetch_fsmap finished waiting for FSMap version 110 2024-03-18T01:44:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.431+0000 7f9c7356b740 10 client.26264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:44:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.432+0000 7f9c7356b740 20 client.26264 populate_metadata read hostname 'smithi007' 2024-03-18T01:44:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.432+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564443a772e0 0x564443a79700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:44:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.432+0000 7f9c7356b740 4 client.26264 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:44:57.432 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.432+0000 7f9c7356b740 1 -- 172.21.15.7:0/4293603127 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56444395a730 con 0x564443a772e0 2024-03-18T01:44:57.433 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.433+0000 7f9c6de00640 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564443a772e0 0x564443a79700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:44:57.437 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.437+0000 7f9c6de00640 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564443a772e0 0x564443a79700 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40aab1ef0a660504 server_cookie=96a62ed89604ba27 in_seq=0 out_seq=0 2024-03-18T01:44:57.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.437+0000 7f9c4b400640 10 client.26264 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:44:57.438 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.437+0000 7f9c4b400640 1 -- 172.21.15.7:0/4293603127 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x56444395a730 con 0x564443a772e0 2024-03-18T01:44:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.437+0000 7f9c4b400640 10 client.26264 handle_command_reply: tid=0 2024-03-18T01:44:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.438+0000 7f9c7356b740 1 client.26264 shutdown 2024-03-18T01:44:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.439+0000 7f9c7356b740 1 -- 172.21.15.7:0/4293603127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 msgr2=0x5644439f9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:57.439 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.439+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f9370 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800f900 tx=0x7f9c58003b90 comp rx=0 tx=0).stop 2024-03-18T01:44:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.440+0000 7f9c7356b740 1 -- 172.21.15.7:0/4293603127 shutdown_connections 2024-03-18T01:44:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.440+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564443a772e0 0x564443a79700 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.440+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644439ee370 0x5644439f9370 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.440+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5644439e9200 0x5644439f8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.440+0000 7f9c7356b740 1 --2- 172.21.15.7:0/4293603127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5644439e8810 0x5644439f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.440+0000 7f9c7356b740 1 -- 172.21.15.7:0/4293603127 >> 172.21.15.7:0/4293603127 conn(0x5644439e3d80 msgr2=0x5644439e6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.440+0000 7f9c4b400640 0 client.26264 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:44:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.440+0000 7f9c7356b740 1 -- 172.21.15.7:0/4293603127 shutdown_connections 2024-03-18T01:44:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.441+0000 7f9c7356b740 1 -- 172.21.15.7:0/4293603127 wait complete. 2024-03-18T01:44:57.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.441+0000 7f9c7356b740 20 client.26264 trim_cache size 0 max 16384 2024-03-18T01:44:57.444 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:44:57.444 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (16 inodes in the stack)", 2024-03-18T01:44:57.444 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:44:57.444 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:44:57.444 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:44:57.444 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:44:57.444 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:44:57.445 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:44:57.445 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:44:57.445 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:44:57.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.444+0000 7f9c70800640 1 -- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9c500a2800 msgr2=0x7f9c500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:57.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.444+0000 7f9c70800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9c500a2800 0x7f9c500a4cc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9c5800b710 tx=0x7f9c580041b0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.444+0000 7f9c70800640 1 -- 172.21.15.7:0/39617023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 msgr2=0x7f9c681959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:44:57.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.445+0000 7f9c70800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 0x7f9c681959f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9c64000f80 tx=0x7f9c64035d10 comp rx=0 tx=0).stop 2024-03-18T01:44:57.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.445+0000 7f9c70800640 1 -- 172.21.15.7:0/39617023 shutdown_connections 2024-03-18T01:44:57.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.445+0000 7f9c70800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c681055d0 0x7f9c6819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.445+0000 7f9c70800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9c500a2800 0x7f9c500a4cc0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.445+0000 7f9c70800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9c68104c10 0x7f9c68195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.445+0000 7f9c70800640 1 --2- 172.21.15.7:0/39617023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c68103a20 0x7f9c681959f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:44:57.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.445+0000 7f9c70800640 1 -- 172.21.15.7:0/39617023 >> 172.21.15.7:0/39617023 conn(0x7f9c680fef90 msgr2=0x7f9c68077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:44:57.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.446+0000 7f9c70800640 1 -- 172.21.15.7:0/39617023 shutdown_connections 2024-03-18T01:44:57.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:44:57.446+0000 7f9c70800640 1 -- 172.21.15.7:0/39617023 wait complete. 2024-03-18T01:44:57.477 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:44:58.170 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:57 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:58.170 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:57 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:58.170 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:57 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:57 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:57 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:57 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:58.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:57 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:44:58.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:57 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:44:58.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:57 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:58 smithi046 ceph-mon[36108]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 33 MiB/s wr, 5.84k op/s 2024-03-18T01:44:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:58 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:58 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:58 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:58 smithi007 ceph-mon[31996]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 33 MiB/s wr, 5.84k op/s 2024-03-18T01:44:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:58 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:58 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:58 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:44:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:58 smithi203 ceph-mon[36345]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 33 MiB/s wr, 5.84k op/s 2024-03-18T01:44:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:58 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:44:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:58 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:44:59.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:58 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:59 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:44:59 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:00.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:59 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:00.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:44:59 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:59 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:44:59 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:00 smithi046 ceph-mon[36108]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 32 MiB/s wr, 5.62k op/s 2024-03-18T01:45:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:00 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:45:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:00 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:45:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:00 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:00 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:00 smithi007 ceph-mon[31996]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 32 MiB/s wr, 5.62k op/s 2024-03-18T01:45:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:00 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:45:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:00 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:45:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:00 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:00 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:01.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:00 smithi203 ceph-mon[36345]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 32 MiB/s wr, 5.62k op/s 2024-03-18T01:45:01.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:00 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:45:01.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:00 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:45:01.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:00 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:01.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:00 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 271 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:01 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:45:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:01 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:45:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:01 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:01 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:01 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:01 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:02.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:01 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:45:02.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:01 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:45:02.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:01 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:02.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:01 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:02.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:01 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:02.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:01 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:02.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:01 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:45:02.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:01 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:45:02.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:01 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:02.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:01 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:02.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:01 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:02.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:01 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:45:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:03 smithi046 ceph-mon[36108]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 34 MiB/s wr, 5.96k op/s 2024-03-18T01:45:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:03 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:03 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:45:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:03 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:03.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:03 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 270 secs 2024-03-18T01:45:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:03 smithi007 ceph-mon[31996]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 34 MiB/s wr, 5.96k op/s 2024-03-18T01:45:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:03 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:03 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:45:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:03 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:03 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 270 secs 2024-03-18T01:45:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:03 smithi203 ceph-mon[36345]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 34 MiB/s wr, 5.96k op/s 2024-03-18T01:45:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:03 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:03 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:45:03.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:03 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:03.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:03 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 270 secs 2024-03-18T01:45:04.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:04 smithi046 ceph-mon[36108]: Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:45:04.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:04 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:45:04.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:04 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:04.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:04 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 5 slow metadata IOs are blocked > 180 secs, oldest blocked for 266 secs 2024-03-18T01:45:04.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:04 smithi007 ceph-mon[31996]: Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:45:04.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:04 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:45:04.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:04 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:04.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:04 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 5 slow metadata IOs are blocked > 180 secs, oldest blocked for 266 secs 2024-03-18T01:45:04.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:04 smithi203 ceph-mon[36345]: Health check update: 6 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:45:04.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:04 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6834/2878820671,v1:172.21.15.46:6835/2878820671] up:standby-replay 2024-03-18T01:45:04.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:04 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:04.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:04 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 5 slow metadata IOs are blocked > 180 secs, oldest blocked for 266 secs 2024-03-18T01:45:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:05 smithi007 ceph-mon[31996]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 35 MiB/s wr, 6.21k op/s 2024-03-18T01:45:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:05 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:45:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:05 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:05 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 2 slow metadata IOs are blocked > 180 secs, oldest blocked for 266 secs 2024-03-18T01:45:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:05 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 271 secs 2024-03-18T01:45:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:05 smithi203 ceph-mon[36345]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 35 MiB/s wr, 6.21k op/s 2024-03-18T01:45:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:05 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:45:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:05 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:05 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 2 slow metadata IOs are blocked > 180 secs, oldest blocked for 266 secs 2024-03-18T01:45:05.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:05 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 271 secs 2024-03-18T01:45:05.521 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:05 smithi046 ceph-mon[36108]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 35 MiB/s wr, 6.21k op/s 2024-03-18T01:45:05.521 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:05 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] up:active 2024-03-18T01:45:05.521 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:05 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:05.521 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:05 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 2 slow metadata IOs are blocked > 180 secs, oldest blocked for 266 secs 2024-03-18T01:45:05.521 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:05 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 271 secs 2024-03-18T01:45:06.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:06 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:45:06.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:06 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:45:06.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:06 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:06.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:06 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:45:06.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:06 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:45:06.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:06 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:06.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:06 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] up:active 2024-03-18T01:45:06.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:06 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/2100401567,v1:172.21.15.203:6833/2100401567] up:standby-replay 2024-03-18T01:45:06.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:06 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:07.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:07 smithi203 ceph-mon[36345]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 32 MiB/s wr, 5.54k op/s 2024-03-18T01:45:07.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:07 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:07.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:07 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 3 slow metadata IOs are blocked > 180 secs, oldest blocked for 259 secs 2024-03-18T01:45:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:07 smithi046 ceph-mon[36108]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 32 MiB/s wr, 5.54k op/s 2024-03-18T01:45:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:07 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:07 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 3 slow metadata IOs are blocked > 180 secs, oldest blocked for 259 secs 2024-03-18T01:45:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:07 smithi007 ceph-mon[31996]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 32 MiB/s wr, 5.54k op/s 2024-03-18T01:45:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:07 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 240 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:07 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 3 slow metadata IOs are blocked > 180 secs, oldest blocked for 259 secs 2024-03-18T01:45:08.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:08 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:08.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:08 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:08.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:08 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:08.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:08 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:08.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:08 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:08.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:08 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:09.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:09 smithi203 ceph-mon[36345]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 38 MiB/s wr, 6.56k op/s 2024-03-18T01:45:09.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:09 smithi046 ceph-mon[36108]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 38 MiB/s wr, 6.56k op/s 2024-03-18T01:45:09.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:09 smithi007 ceph-mon[31996]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 38 MiB/s wr, 6.56k op/s 2024-03-18T01:45:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:10 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:10 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:10 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 268 secs 2024-03-18T01:45:10.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:10 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:10.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:10 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:10.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:10 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 268 secs 2024-03-18T01:45:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:10 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:10 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:10.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:10 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 4 slow metadata IOs are blocked > 180 secs, oldest blocked for 268 secs 2024-03-18T01:45:11.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:11 smithi046 ceph-mon[36108]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 33 MiB/s wr, 5.81k op/s 2024-03-18T01:45:11.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:11 smithi046 ceph-mon[36108]: Health check update: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:45:11.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:11 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:45:11.520 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:11 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:11.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:11 smithi007 ceph-mon[31996]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 33 MiB/s wr, 5.81k op/s 2024-03-18T01:45:11.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:11 smithi007 ceph-mon[31996]: Health check update: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:45:11.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:11 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:45:11.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:11 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:11.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:11 smithi203 ceph-mon[36345]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 33 MiB/s wr, 5.81k op/s 2024-03-18T01:45:11.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:11 smithi203 ceph-mon[36345]: Health check update: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-03-18T01:45:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:11 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] up:active 2024-03-18T01:45:11.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:11 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:12.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:12 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:45:12.589 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:12 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:45:12.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:12 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:45:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:13 smithi007 ceph-mon[31996]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 34 MiB/s wr, 6.03k op/s 2024-03-18T01:45:13.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:13 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:13.608 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:13 smithi046 ceph-mon[36108]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 34 MiB/s wr, 6.03k op/s 2024-03-18T01:45:13.608 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:13 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:13 smithi203 ceph-mon[36345]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 34 MiB/s wr, 6.03k op/s 2024-03-18T01:45:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:13 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 245 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:14.543 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:14 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:14.543 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:14 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:14.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:14 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:14.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:14 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:14.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:14 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:14.654 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:14 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:15.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:15 smithi007 ceph-mon[31996]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 33 MiB/s wr, 5.69k op/s 2024-03-18T01:45:15.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:15 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:45:15.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:15 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:45:15.556 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:15 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:45:15.600 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:15 smithi046 ceph-mon[36108]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 33 MiB/s wr, 5.69k op/s 2024-03-18T01:45:15.600 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:15 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:45:15.600 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:15 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:45:15.601 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:15 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:45:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:15 smithi203 ceph-mon[36345]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 33 MiB/s wr, 5.69k op/s 2024-03-18T01:45:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:15 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:45:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:15 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:45:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:15 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:45:17.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:17 smithi046 ceph-mon[36108]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 31 MiB/s wr, 5.39k op/s 2024-03-18T01:45:17.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:17 smithi007 ceph-mon[31996]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 31 MiB/s wr, 5.39k op/s 2024-03-18T01:45:17.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:17 smithi203 ceph-mon[36345]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 31 MiB/s wr, 5.39k op/s 2024-03-18T01:45:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:18 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:18 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:18.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:18 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:18.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:18 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:18 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:18 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:19.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:19 smithi046 ceph-mon[36108]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 34 MiB/s wr, 5.83k op/s 2024-03-18T01:45:19.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:19 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 252 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:19 smithi007 ceph-mon[31996]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 34 MiB/s wr, 5.83k op/s 2024-03-18T01:45:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:19 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 252 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:19.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:19 smithi203 ceph-mon[36345]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 34 MiB/s wr, 5.83k op/s 2024-03-18T01:45:19.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:19 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 252 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:21.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:21 smithi046 ceph-mon[36108]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 29 MiB/s wr, 5.21k op/s 2024-03-18T01:45:21.555 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:21 smithi007 ceph-mon[31996]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 29 MiB/s wr, 5.21k op/s 2024-03-18T01:45:21.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:21 smithi203 ceph-mon[36345]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 29 MiB/s wr, 5.21k op/s 2024-03-18T01:45:22.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:22 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:22.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:22 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:22.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:22 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:22.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:22 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:22 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:22 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:23.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:23 smithi007 ceph-mon[31996]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 33 MiB/s wr, 5.92k op/s 2024-03-18T01:45:23.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:23 smithi203 ceph-mon[36345]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 33 MiB/s wr, 5.92k op/s 2024-03-18T01:45:23.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:23 smithi046 ceph-mon[36108]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 33 MiB/s wr, 5.92k op/s 2024-03-18T01:45:24.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:24 smithi203 ceph-mon[36345]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 28 MiB/s wr, 5.02k op/s 2024-03-18T01:45:24.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:24 smithi046 ceph-mon[36108]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 28 MiB/s wr, 5.02k op/s 2024-03-18T01:45:24.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:24 smithi007 ceph-mon[31996]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 28 MiB/s wr, 5.02k op/s 2024-03-18T01:45:25.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:25 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 258 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:25.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:25 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 258 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:25.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:25 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 258 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:26 smithi203 ceph-mon[36345]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 30 MiB/s wr, 5.32k op/s 2024-03-18T01:45:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:26 smithi046 ceph-mon[36108]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 30 MiB/s wr, 5.32k op/s 2024-03-18T01:45:26.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:26 smithi007 ceph-mon[31996]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 30 MiB/s wr, 5.32k op/s 2024-03-18T01:45:27.478 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:45:27.478 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:45:28.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:28 smithi203 ceph-mon[36345]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 34 MiB/s wr, 6.03k op/s 2024-03-18T01:45:29.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:28 smithi046 ceph-mon[36108]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 34 MiB/s wr, 6.03k op/s 2024-03-18T01:45:29.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:28 smithi007 ceph-mon[31996]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 34 MiB/s wr, 6.03k op/s 2024-03-18T01:45:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.659+0000 7fdabac00640 1 -- 172.21.15.7:0/3460604448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab4110060 msgr2=0x7fdab4112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.659+0000 7fdabac00640 1 --2- 172.21.15.7:0/3460604448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab4110060 0x7fdab4112450 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009860 tx=0x7fdaa402bb70 comp rx=0 tx=0).stop 2024-03-18T01:45:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.659+0000 7fdabac00640 1 -- 172.21.15.7:0/3460604448 shutdown_connections 2024-03-18T01:45:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.659+0000 7fdabac00640 1 --2- 172.21.15.7:0/3460604448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab4110060 0x7fdab4112450 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.659+0000 7fdabac00640 1 --2- 172.21.15.7:0/3460604448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdab4109140 0x7fdab410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.661 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.659+0000 7fdabac00640 1 --2- 172.21.15.7:0/3460604448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab41087e0 0x7fdab4108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.659+0000 7fdabac00640 1 -- 172.21.15.7:0/3460604448 >> 172.21.15.7:0/3460604448 conn(0x7fdab406e780 msgr2=0x7fdab4070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:45:30.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.660+0000 7fdabac00640 1 -- 172.21.15.7:0/3460604448 shutdown_connections 2024-03-18T01:45:30.662 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.660+0000 7fdabac00640 1 -- 172.21.15.7:0/3460604448 wait complete. 2024-03-18T01:45:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.661+0000 7fdabac00640 1 Processor -- start 2024-03-18T01:45:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.661+0000 7fdabac00640 1 -- start start 2024-03-18T01:45:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdabac00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab41087e0 0x7fdab41a2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdabac00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdab4109140 0x7fdab41a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdabac00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab4110060 0x7fdab41a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdabac00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdab4115c80 con 0x7fdab4110060 2024-03-18T01:45:30.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdabac00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdab4115b00 con 0x7fdab41087e0 2024-03-18T01:45:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdabac00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdab4115e00 con 0x7fdab4109140 2024-03-18T01:45:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdab3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab41087e0 0x7fdab41a2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdab3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdab4109140 0x7fdab41a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdab3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab41087e0 0x7fdab41a2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:47748/0 (socket says 172.21.15.7:47748) 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdab3e00640 1 -- 172.21.15.7:0/3455186446 learned_addr learned my addr 172.21.15.7:0/3455186446 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.662+0000 7fdab3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdab4109140 0x7fdab41a29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:57386/0 (socket says 172.21.15.7:57386) 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.663+0000 7fdab3e00640 1 -- 172.21.15.7:0/3455186446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdab4109140 msgr2=0x7fdab41a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.663+0000 7fdab3e00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdab4109140 0x7fdab41a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.663+0000 7fdab3e00640 1 -- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab4110060 msgr2=0x7fdab41a9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.663+0000 7fdab3e00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab4110060 0x7fdab41a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.663+0000 7fdab3e00640 1 -- 172.21.15.7:0/3455186446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab41aa090 con 0x7fdab41087e0 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.663+0000 7fdab3400640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdab4109140 0x7fdab41a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:45:30.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.663+0000 7fdab3e00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab41087e0 0x7fdab41a2470 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00e9f0 tx=0x7fda9c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:45:30.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.665+0000 7fdab0c00640 1 -- 172.21.15.7:0/3455186446 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c00cd80 con 0x7fdab41087e0 2024-03-18T01:45:30.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.667+0000 7fdab0c00640 1 -- 172.21.15.7:0/3455186446 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda9c010910 con 0x7fdab41087e0 2024-03-18T01:45:30.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.667+0000 7fdab0c00640 1 -- 172.21.15.7:0/3455186446 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c018980 con 0x7fdab41087e0 2024-03-18T01:45:30.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.667+0000 7fdabac00640 1 -- 172.21.15.7:0/3455186446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab41aa380 con 0x7fdab41087e0 2024-03-18T01:45:30.669 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.667+0000 7fdabac00640 1 -- 172.21.15.7:0/3455186446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab41aa880 con 0x7fdab41087e0 2024-03-18T01:45:30.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.672+0000 7fdab0c00640 1 -- 172.21.15.7:0/3455186446 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fda9c018b20 con 0x7fdab41087e0 2024-03-18T01:45:30.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.674+0000 7fdab0c00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda980a2880 0x7fda980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.675+0000 7fdab0c00640 1 -- 172.21.15.7:0/3455186446 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fda9c0d0190 con 0x7fdab41087e0 2024-03-18T01:45:30.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.679+0000 7fdab3400640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda980a2880 0x7fda980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.697 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.696+0000 7fdab3400640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda980a2880 0x7fda980a4d40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdab41a3980 tx=0x7fdaac010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:45:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.698+0000 7fdabd7fc740 1 Processor -- start 2024-03-18T01:45:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.698+0000 7fdabd7fc740 1 -- start start 2024-03-18T01:45:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559716171b50 0x559716171f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x559716177040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5597161776b0 0x55971618fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5597160db1f0 con 0x559716171b50 2024-03-18T01:45:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559716083a80 con 0x559716172540 2024-03-18T01:45:30.700 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5597160d7290 con 0x5597161776b0 2024-03-18T01:45:30.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdab3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x559716177040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.701 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdab3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x559716177040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:47750/0 (socket says 172.21.15.7:47750) 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.699+0000 7fdab3400640 1 -- 172.21.15.7:0/1963358279 learned_addr learned my addr 172.21.15.7:0/1963358279 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.700+0000 7fdab3400640 1 -- 172.21.15.7:0/1963358279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5597161776b0 msgr2=0x55971618fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.700+0000 7fdab8c00640 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5597161776b0 0x55971618fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.700+0000 7fdab3400640 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5597161776b0 0x55971618fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.700+0000 7fdab3400640 1 -- 172.21.15.7:0/1963358279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559716171b50 msgr2=0x559716171f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.700+0000 7fdab3400640 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559716171b50 0x559716171f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.700+0000 7fdab3400640 1 -- 172.21.15.7:0/1963358279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559715eb97d0 con 0x559716172540 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.700+0000 7fdab8c00640 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5597161776b0 0x55971618fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:45:30.702 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.700+0000 7fdab3400640 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x559716177040 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00dd50 tx=0x7fdaac014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4df202445f1f0bda server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:45:30.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.704+0000 7fdaa9200640 1 -- 172.21.15.7:0/1963358279 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0233f0 con 0x559716172540 2024-03-18T01:45:30.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.704+0000 7fdaa9200640 1 -- 172.21.15.7:0/1963358279 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaac028070 con 0x559716172540 2024-03-18T01:45:30.705 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.704+0000 7fdaa9200640 1 -- 172.21.15.7:0/1963358279 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0189a0 con 0x559716172540 2024-03-18T01:45:30.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.705+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1963358279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 msgr2=0x559716177040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.706 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.705+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x559716177040 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00dd50 tx=0x7fdaac014000 comp rx=0 tx=0).stop 2024-03-18T01:45:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.705+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1963358279 shutdown_connections 2024-03-18T01:45:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.705+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5597161776b0 0x55971618fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.705+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x559716177040 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.705+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/1963358279 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x559716171b50 0x559716171f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.706+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1963358279 >> 172.21.15.7:0/1963358279 conn(0x55971617a590 msgr2=0x55971617a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:45:30.707 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.706+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1963358279 shutdown_connections 2024-03-18T01:45:30.708 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.707+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1963358279 wait complete. 2024-03-18T01:45:30.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.707+0000 7fdabd7fc740 1 -- start start 2024-03-18T01:45:30.709 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.708+0000 7fdabd7fc740 1 Processor -- start 2024-03-18T01:45:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.710+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716191e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.710+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x5597161923c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.710+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716199400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.712 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdab3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716191e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x559716190330 con 0x5597161776b0 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdab3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x5597161923c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x559716096ed0 con 0x559716172540 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5597161904b0 con 0x559716171b50 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdab3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x5597161923c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:47764/0 (socket says 172.21.15.7:47764) 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdab3400640 1 -- 172.21.15.7:0/3696388388 learned_addr learned my addr 172.21.15.7:0/3696388388 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdabd7fc740 10 client.0 _ll_register_callbacks cb 0x5597160d2220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.711+0000 7fdab8c00640 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.713 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.712+0000 7fdab3400640 1 -- 172.21.15.7:0/3696388388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 msgr2=0x559716191e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:45:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.712+0000 7fdab3400640 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716191e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.712+0000 7fdab3400640 1 -- 172.21.15.7:0/3696388388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 msgr2=0x559716199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:45:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.712+0000 7fdab3400640 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.712+0000 7fdab3400640 1 -- 172.21.15.7:0/3696388388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559715def3f0 con 0x559716172540 2024-03-18T01:45:30.714 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.712+0000 7fdab8c00640 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:45:30.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.713+0000 7fdab3400640 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x5597161923c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdaac018e70 tx=0x7fdaac018ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfb9ed1f3018cd86 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:45:30.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.713+0000 7fdaa1600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:45:30.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.713+0000 7fdaa1600640 1 -- 172.21.15.7:0/3696388388 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac004c40 con 0x559716172540 2024-03-18T01:45:30.715 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.714+0000 7fdaa1600640 1 -- 172.21.15.7:0/3696388388 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdaac004de0 con 0x559716172540 2024-03-18T01:45:30.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.717+0000 7fdabd7fc740 1 -- 172.21.15.7:0/3696388388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaac028070 con 0x559716172540 2024-03-18T01:45:30.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.717+0000 7fdaa1600640 1 -- 172.21.15.7:0/3696388388 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=125) ==== 24+0+0 (secure 0 0 0) 0x7fdaac005030 con 0x559716172540 2024-03-18T01:45:30.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.718+0000 7fdabd7fc740 10 client.26276 fetch_fsmap learned FSMap version 125 2024-03-18T01:45:30.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.718+0000 7fdabd7fc740 1 -- 172.21.15.7:0/3696388388 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=125,monmap=4+}) -- 0x559715daf0f0 con 0x559716172540 2024-03-18T01:45:30.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.718+0000 7fdaa1600640 1 -- 172.21.15.7:0/3696388388 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaac028070 con 0x559716172540 2024-03-18T01:45:30.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.718+0000 7fdaa1600640 1 -- 172.21.15.7:0/3696388388 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 125) ==== 8370+0+0 (secure 0 0 0) 0x7fdaac00b040 con 0x559716172540 2024-03-18T01:45:30.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.719+0000 7fdabd7fc740 10 client.26276 fetch_fsmap finished waiting for FSMap version 125 2024-03-18T01:45:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.719+0000 7fdabd7fc740 10 client.26276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:45:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.719+0000 7fdabd7fc740 20 client.26276 populate_metadata read hostname 'smithi007' 2024-03-18T01:45:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.719+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55971619d8a0 0x55971619fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.719+0000 7fdabd7fc740 4 client.26276 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:45:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.719+0000 7fdabd7fc740 1 -- 172.21.15.7:0/3696388388 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597160eebe0 con 0x55971619d8a0 2024-03-18T01:45:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.719+0000 7fdab3e00640 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55971619d8a0 0x55971619fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.721+0000 7fdaa1600640 10 client.26276 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:45:30.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.721+0000 7fdab3e00640 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55971619d8a0 0x55971619fc90 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8c74ac944e3336a server_cookie=441dc35eeb94dfb in_seq=0 out_seq=0 2024-03-18T01:45:30.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.723+0000 7fdaa1600640 1 -- 172.21.15.7:0/3696388388 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5597160eebe0 con 0x55971619d8a0 2024-03-18T01:45:30.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.723+0000 7fdaa1600640 10 client.26276 handle_command_reply: tid=0 2024-03-18T01:45:30.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.723+0000 7fdabd7fc740 1 client.26276 shutdown 2024-03-18T01:45:30.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.723+0000 7fdabd7fc740 1 -- 172.21.15.7:0/3696388388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 msgr2=0x5597161923c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.723+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x5597161923c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdaac018e70 tx=0x7fdaac018ea0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.724+0000 7fdabd7fc740 1 -- 172.21.15.7:0/3696388388 shutdown_connections 2024-03-18T01:45:30.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.724+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55971619d8a0 0x55971619fc90 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.724+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.724+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716172540 0x5597161923c0 secure :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdaac018e70 tx=0x7fdaac018ea0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.724+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/3696388388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716191e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.724+0000 7fdabd7fc740 1 -- 172.21.15.7:0/3696388388 >> 172.21.15.7:0/3696388388 conn(0x55971617a590 msgr2=0x559716170ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:45:30.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.724+0000 7fdaa1600640 0 client.26276 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:45:30.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.725+0000 7fdabd7fc740 1 -- 172.21.15.7:0/3696388388 shutdown_connections 2024-03-18T01:45:30.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.725+0000 7fdabd7fc740 1 -- 172.21.15.7:0/3696388388 wait complete. 2024-03-18T01:45:30.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.725+0000 7fdabd7fc740 20 client.26276 trim_cache size 0 max 16384 2024-03-18T01:45:30.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.767+0000 7fdabd7fc740 1 Processor -- start 2024-03-18T01:45:30.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.767+0000 7fdabd7fc740 1 -- start start 2024-03-18T01:45:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716173b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5597161776b0 0x559716174050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55971619ffc0 0x5597161993c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b09a0 con 0x55971619ffc0 2024-03-18T01:45:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b2540 con 0x5597161776b0 2024-03-18T01:45:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b1070 con 0x559716171b50 2024-03-18T01:45:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdab3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716173b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdab3e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716173b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:57412/0 (socket says 172.21.15.7:57412) 2024-03-18T01:45:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdab3e00640 1 -- 172.21.15.7:0/1253018563 learned_addr learned my addr 172.21.15.7:0/1253018563 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:45:30.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.768+0000 7fdab3400640 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5597161776b0 0x559716174050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.769+0000 7fdab3e00640 1 -- 172.21.15.7:0/1253018563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5597161776b0 msgr2=0x559716174050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:45:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.769+0000 7fdab3e00640 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5597161776b0 0x559716174050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.769+0000 7fdab3e00640 1 -- 172.21.15.7:0/1253018563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55971619ffc0 msgr2=0x5597161993c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:45:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.769+0000 7fdab8c00640 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55971619ffc0 0x5597161993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.770+0000 7fdab3e00640 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55971619ffc0 0x5597161993c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.770+0000 7fdab3e00640 1 -- 172.21.15.7:0/1253018563 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55971616c370 con 0x559716171b50 2024-03-18T01:45:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.770+0000 7fdab8c00640 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55971619ffc0 0x5597161993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:45:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.770+0000 7fdab3e00640 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716173b10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fda9c09c4d0 tx=0x7fda9c087040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7532403517636ec7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:45:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.771+0000 7fdaa3e00640 1 -- 172.21.15.7:0/1253018563 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c0b5070 con 0x559716171b50 2024-03-18T01:45:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.771+0000 7fdaa3e00640 1 -- 172.21.15.7:0/1253018563 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda9c09a870 con 0x559716171b50 2024-03-18T01:45:30.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.771+0000 7fdaa3e00640 1 -- 172.21.15.7:0/1253018563 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c010c50 con 0x559716171b50 2024-03-18T01:45:30.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1253018563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 msgr2=0x559716173b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716173b10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fda9c09c4d0 tx=0x7fda9c087040 comp rx=0 tx=0).stop 2024-03-18T01:45:30.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1253018563 shutdown_connections 2024-03-18T01:45:30.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55971619ffc0 0x5597161993c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5597161776b0 0x559716174050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/1253018563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559716171b50 0x559716173b10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1253018563 >> 172.21.15.7:0/1253018563 conn(0x55971617a590 msgr2=0x55971619f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:45:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1253018563 shutdown_connections 2024-03-18T01:45:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.772+0000 7fdabd7fc740 1 -- 172.21.15.7:0/1253018563 wait complete. 2024-03-18T01:45:30.774 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.773+0000 7fdabd7fc740 1 -- start start 2024-03-18T01:45:30.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.773+0000 7fdabd7fc740 1 Processor -- start 2024-03-18T01:45:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.774+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716171b50 0x559716196930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.774+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716196e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.774+0000 7fdabd7fc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55971619ffc0 0x559716197e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.774+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b1e70 con 0x5597161776b0 2024-03-18T01:45:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.774+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41b1740 con 0x559716171b50 2024-03-18T01:45:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.774+0000 7fdabd7fc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdab41ae980 con 0x55971619ffc0 2024-03-18T01:45:30.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.774+0000 7fdabd7fc740 10 client.0 _ll_register_callbacks cb 0x7fdaac00dc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:45:30.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.775+0000 7fdab8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55971619ffc0 0x559716197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.775+0000 7fdab8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55971619ffc0 0x559716197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:57414/0 (socket says 172.21.15.7:57414) 2024-03-18T01:45:30.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.776+0000 7fdab8c00640 1 -- 172.21.15.7:0/550544414 learned_addr learned my addr 172.21.15.7:0/550544414 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:45:30.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.775+0000 7fdab3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716171b50 0x559716196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.776+0000 7fdab3400640 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.777+0000 7fdab8c00640 1 -- 172.21.15.7:0/550544414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716171b50 msgr2=0x559716196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:45:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.777+0000 7fdab8c00640 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716171b50 0x559716196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.777+0000 7fdab8c00640 1 -- 172.21.15.7:0/550544414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 msgr2=0x559716196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:45:30.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.777+0000 7fdab8c00640 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.777+0000 7fdab8c00640 1 -- 172.21.15.7:0/550544414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559716198470 con 0x55971619ffc0 2024-03-18T01:45:30.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.777+0000 7fdab3e00640 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716171b50 0x559716196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:45:30.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.778+0000 7fdab8c00640 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55971619ffc0 0x559716197e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4037040 tx=0x7fdaa4034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7604cb5611718a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:45:30.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.778+0000 7fdaa1600640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:45:30.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.779+0000 7fdaa1600640 1 -- 172.21.15.7:0/550544414 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4039040 con 0x55971619ffc0 2024-03-18T01:45:30.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.779+0000 7fdaa1600640 1 -- 172.21.15.7:0/550544414 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdaa4004230 con 0x55971619ffc0 2024-03-18T01:45:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.779+0000 7fdabd7fc740 1 -- 172.21.15.7:0/550544414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda9c09a870 con 0x55971619ffc0 2024-03-18T01:45:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.779+0000 7fdaa1600640 1 -- 172.21.15.7:0/550544414 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=125) ==== 24+0+0 (secure 0 0 0) 0x7fdaa404d050 con 0x55971619ffc0 2024-03-18T01:45:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.780+0000 7fdabd7fc740 10 client.26257 fetch_fsmap learned FSMap version 125 2024-03-18T01:45:30.781 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.780+0000 7fdabd7fc740 1 -- 172.21.15.7:0/550544414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=125,monmap=4+}) -- 0x559716199760 con 0x55971619ffc0 2024-03-18T01:45:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdaa1600640 1 -- 172.21.15.7:0/550544414 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda9c09a870 con 0x55971619ffc0 2024-03-18T01:45:30.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdaa1600640 1 -- 172.21.15.7:0/550544414 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 125) ==== 8370+0+0 (secure 0 0 0) 0x7fdaa4004970 con 0x55971619ffc0 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdabd7fc740 10 client.26257 fetch_fsmap finished waiting for FSMap version 125 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdabd7fc740 10 client.26257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdabd7fc740 20 client.26257 populate_metadata read hostname 'smithi007' 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559716204210 0x559716206630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdabd7fc740 4 client.26257 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdabd7fc740 1 -- 172.21.15.7:0/550544414 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5597160e8730 con 0x559716204210 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.784+0000 7fdab3e00640 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559716204210 0x559716206630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.785+0000 7fdaa1600640 10 client.26257 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.785+0000 7fdab3e00640 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559716204210 0x559716206630 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7252a2c0552d4aa8 server_cookie=899972d2cc1f4108 in_seq=0 out_seq=0 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.785+0000 7fdaa1600640 1 -- 172.21.15.7:0/550544414 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5597160e8730 con 0x559716204210 2024-03-18T01:45:30.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.785+0000 7fdaa1600640 10 client.26257 handle_command_reply: tid=0 2024-03-18T01:45:30.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.785+0000 7fdabd7fc740 1 client.26257 shutdown 2024-03-18T01:45:30.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 -- 172.21.15.7:0/550544414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55971619ffc0 msgr2=0x559716197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55971619ffc0 0x559716197e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4037040 tx=0x7fdaa4034670 comp rx=0 tx=0).stop 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 -- 172.21.15.7:0/550544414 shutdown_connections 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x559716204210 0x559716206630 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55971619ffc0 0x559716197e80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5597161776b0 0x559716196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 --2- 172.21.15.7:0/550544414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559716171b50 0x559716196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 -- 172.21.15.7:0/550544414 >> 172.21.15.7:0/550544414 conn(0x55971617a590 msgr2=0x559716191e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdaa1600640 0 client.26257 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 -- 172.21.15.7:0/550544414 shutdown_connections 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 1 -- 172.21.15.7:0/550544414 wait complete. 2024-03-18T01:45:30.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.790+0000 7fdabd7fc740 20 client.26257 trim_cache size 0 max 16384 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:45:30.793 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:45:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 -- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda980a2880 msgr2=0x7fda980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda980a2880 0x7fda980a4d40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdab41a3980 tx=0x7fdaac010040 comp rx=0 tx=0).stop 2024-03-18T01:45:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 -- 172.21.15.7:0/3455186446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab41087e0 msgr2=0x7fdab41a2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:45:30.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab41087e0 0x7fdab41a2470 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00e9f0 tx=0x7fda9c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 -- 172.21.15.7:0/3455186446 shutdown_connections 2024-03-18T01:45:30.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fda980a2880 0x7fda980a4d40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdab4110060 0x7fdab41a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdab4109140 0x7fdab41a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 --2- 172.21.15.7:0/3455186446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdab41087e0 0x7fdab41a2470 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:45:30.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 -- 172.21.15.7:0/3455186446 >> 172.21.15.7:0/3455186446 conn(0x7fdab406e780 msgr2=0x7fdab41098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:45:30.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 -- 172.21.15.7:0/3455186446 shutdown_connections 2024-03-18T01:45:30.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:45:30.793+0000 7fdabac00640 1 -- 172.21.15.7:0/3455186446 wait complete. 2024-03-18T01:45:30.824 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:45:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:30 smithi203 ceph-mon[36345]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 31 MiB/s wr, 5.61k op/s 2024-03-18T01:45:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:30 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:30.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:30 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:31.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:30 smithi046 ceph-mon[36108]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 31 MiB/s wr, 5.61k op/s 2024-03-18T01:45:31.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:30 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:31.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:30 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:31.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:30 smithi007 ceph-mon[31996]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 31 MiB/s wr, 5.61k op/s 2024-03-18T01:45:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:30 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:30 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:32.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:31 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 263 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:32.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:31 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 263 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:32.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:31 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 263 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:32 smithi046 ceph-mon[36108]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 31 MiB/s wr, 5.60k op/s 2024-03-18T01:45:33.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:32 smithi007 ceph-mon[31996]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 31 MiB/s wr, 5.60k op/s 2024-03-18T01:45:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:32 smithi203 ceph-mon[36345]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 31 MiB/s wr, 5.60k op/s 2024-03-18T01:45:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:34 smithi046 ceph-mon[36108]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 27 MiB/s wr, 4.86k op/s 2024-03-18T01:45:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:34 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:34 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:34 smithi007 ceph-mon[31996]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 27 MiB/s wr, 4.86k op/s 2024-03-18T01:45:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:34 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:34 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:34 smithi203 ceph-mon[36345]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 27 MiB/s wr, 4.86k op/s 2024-03-18T01:45:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:34 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:34 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:36 smithi046 ceph-mon[36108]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 30 MiB/s wr, 5.42k op/s 2024-03-18T01:45:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:36 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 268 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:37.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:36 smithi007 ceph-mon[31996]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 30 MiB/s wr, 5.42k op/s 2024-03-18T01:45:37.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:36 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 268 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:36 smithi203 ceph-mon[36345]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 30 MiB/s wr, 5.42k op/s 2024-03-18T01:45:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:36 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 268 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:38 smithi046 ceph-mon[36108]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 32 MiB/s wr, 5.78k op/s 2024-03-18T01:45:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:38 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:38 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:38 smithi046 ceph-mon[36108]: 1 slow requests, 1 included below; oldest blocked for > 182.612038 secs 2024-03-18T01:45:39.020 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:38 smithi046 ceph-mon[36108]: slow request 182.612037 seconds old, received at 2024-03-18T01:42:35.251740+0000: internal op exportdir:mds.4:1 currently failed to wrlock, waiting 2024-03-18T01:45:39.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:38 smithi007 ceph-mon[31996]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 32 MiB/s wr, 5.78k op/s 2024-03-18T01:45:39.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:38 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:39.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:38 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:39.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:38 smithi007 ceph-mon[31996]: 1 slow requests, 1 included below; oldest blocked for > 182.612038 secs 2024-03-18T01:45:39.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:38 smithi007 ceph-mon[31996]: slow request 182.612037 seconds old, received at 2024-03-18T01:42:35.251740+0000: internal op exportdir:mds.4:1 currently failed to wrlock, waiting 2024-03-18T01:45:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:38 smithi203 ceph-mon[36345]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 32 MiB/s wr, 5.78k op/s 2024-03-18T01:45:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:38 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:38 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:38 smithi203 ceph-mon[36345]: 1 slow requests, 1 included below; oldest blocked for > 182.612038 secs 2024-03-18T01:45:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:38 smithi203 ceph-mon[36345]: slow request 182.612037 seconds old, received at 2024-03-18T01:42:35.251740+0000: internal op exportdir:mds.4:1 currently failed to wrlock, waiting 2024-03-18T01:45:40.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:39 smithi046 ceph-mon[36108]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:45:40.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:39 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:40.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:39 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:40.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:39 smithi007 ceph-mon[31996]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:45:40.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:39 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:40.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:39 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:40.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:39 smithi203 ceph-mon[36345]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:45:40.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:39 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:45:40.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:39 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:40 smithi046 ceph-mon[36108]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 27 MiB/s wr, 4.95k op/s 2024-03-18T01:45:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:40 smithi046 ceph-mon[36108]: 1 slow requests, 1 included below; oldest blocked for > 182.155034 secs 2024-03-18T01:45:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:40 smithi046 ceph-mon[36108]: slow request 182.155034 seconds old, received at 2024-03-18T01:42:38.106213+0000: peer_request:mds.4:1 currently dispatched 2024-03-18T01:45:41.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:40 smithi007 ceph-mon[31996]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 27 MiB/s wr, 4.95k op/s 2024-03-18T01:45:41.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:40 smithi007 ceph-mon[31996]: 1 slow requests, 1 included below; oldest blocked for > 182.155034 secs 2024-03-18T01:45:41.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:40 smithi007 ceph-mon[31996]: slow request 182.155034 seconds old, received at 2024-03-18T01:42:38.106213+0000: peer_request:mds.4:1 currently dispatched 2024-03-18T01:45:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:40 smithi203 ceph-mon[36345]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 27 MiB/s wr, 4.95k op/s 2024-03-18T01:45:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:40 smithi203 ceph-mon[36345]: 1 slow requests, 1 included below; oldest blocked for > 182.155034 secs 2024-03-18T01:45:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:40 smithi203 ceph-mon[36345]: slow request 182.155034 seconds old, received at 2024-03-18T01:42:38.106213+0000: peer_request:mds.4:1 currently dispatched 2024-03-18T01:45:42.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:41 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 273 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:42.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:41 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 273 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:41 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 273 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:43.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:42 smithi007 ceph-mon[31996]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 28 MiB/s wr, 5.16k op/s 2024-03-18T01:45:43.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:42 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:43.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:42 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:42 smithi203 ceph-mon[36345]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 28 MiB/s wr, 5.16k op/s 2024-03-18T01:45:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:42 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:42 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:43.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:42 smithi046 ceph-mon[36108]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 28 MiB/s wr, 5.16k op/s 2024-03-18T01:45:43.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:42 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:43.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:42 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:44.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:43 smithi007 ceph-mon[31996]: 1 slow requests, 0 included below; oldest blocked for > 187.612397 secs 2024-03-18T01:45:44.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:43 smithi203 ceph-mon[36345]: 1 slow requests, 0 included below; oldest blocked for > 187.612397 secs 2024-03-18T01:45:44.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:43 smithi046 ceph-mon[36108]: 1 slow requests, 0 included below; oldest blocked for > 187.612397 secs 2024-03-18T01:45:45.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:44 smithi203 ceph-mon[36345]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 28 MiB/s wr, 5.01k op/s 2024-03-18T01:45:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:44 smithi046 ceph-mon[36108]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 28 MiB/s wr, 5.01k op/s 2024-03-18T01:45:45.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:44 smithi007 ceph-mon[31996]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 28 MiB/s wr, 5.01k op/s 2024-03-18T01:45:46.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:45 smithi203 ceph-mon[36345]: 1 slow requests, 0 included below; oldest blocked for > 187.155108 secs 2024-03-18T01:45:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:45 smithi046 ceph-mon[36108]: 1 slow requests, 0 included below; oldest blocked for > 187.155108 secs 2024-03-18T01:45:46.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:45 smithi007 ceph-mon[31996]: 1 slow requests, 0 included below; oldest blocked for > 187.155108 secs 2024-03-18T01:45:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:46 smithi203 ceph-mon[36345]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 27 MiB/s wr, 4.90k op/s 2024-03-18T01:45:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:46 smithi203 ceph-mon[36345]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:45:47.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:46 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 278 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:47.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:46 smithi046 ceph-mon[36108]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 27 MiB/s wr, 4.90k op/s 2024-03-18T01:45:47.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:46 smithi046 ceph-mon[36108]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:45:47.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:46 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 278 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:47.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:46 smithi007 ceph-mon[31996]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 27 MiB/s wr, 4.90k op/s 2024-03-18T01:45:47.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:46 smithi007 ceph-mon[31996]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:45:47.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:46 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 278 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:49.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:48 smithi203 ceph-mon[36345]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 32 MiB/s wr, 5.80k op/s 2024-03-18T01:45:49.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:48 smithi203 ceph-mon[36345]: 1 slow requests, 0 included below; oldest blocked for > 192.612460 secs 2024-03-18T01:45:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:48 smithi046 ceph-mon[36108]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 32 MiB/s wr, 5.80k op/s 2024-03-18T01:45:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:48 smithi046 ceph-mon[36108]: 1 slow requests, 0 included below; oldest blocked for > 192.612460 secs 2024-03-18T01:45:49.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:48 smithi007 ceph-mon[31996]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 32 MiB/s wr, 5.80k op/s 2024-03-18T01:45:49.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:48 smithi007 ceph-mon[31996]: 1 slow requests, 0 included below; oldest blocked for > 192.612460 secs 2024-03-18T01:45:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:50 smithi203 ceph-mon[36345]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 27 MiB/s wr, 4.86k op/s 2024-03-18T01:45:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:50 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:50 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:50 smithi203 ceph-mon[36345]: 1 slow requests, 0 included below; oldest blocked for > 192.155175 secs 2024-03-18T01:45:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:50 smithi046 ceph-mon[36108]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 27 MiB/s wr, 4.86k op/s 2024-03-18T01:45:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:50 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:50 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:50 smithi046 ceph-mon[36108]: 1 slow requests, 0 included below; oldest blocked for > 192.155175 secs 2024-03-18T01:45:51.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:50 smithi007 ceph-mon[31996]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 27 MiB/s wr, 4.86k op/s 2024-03-18T01:45:51.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:50 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:50 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:50 smithi007 ceph-mon[31996]: 1 slow requests, 0 included below; oldest blocked for > 192.155175 secs 2024-03-18T01:45:52.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:51 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 283 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:52.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:51 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 283 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:52.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:51 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 283 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:53.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:52 smithi046 ceph-mon[36108]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 MiB/s wr, 5.03k op/s 2024-03-18T01:45:53.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:52 smithi046 ceph-mon[36108]: 1 slow requests, 0 included below; oldest blocked for > 197.612524 secs 2024-03-18T01:45:53.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:52 smithi007 ceph-mon[31996]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 MiB/s wr, 5.03k op/s 2024-03-18T01:45:53.306 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:52 smithi007 ceph-mon[31996]: 1 slow requests, 0 included below; oldest blocked for > 197.612524 secs 2024-03-18T01:45:53.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:52 smithi203 ceph-mon[36345]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 MiB/s wr, 5.03k op/s 2024-03-18T01:45:53.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:52 smithi203 ceph-mon[36345]: 1 slow requests, 0 included below; oldest blocked for > 197.612524 secs 2024-03-18T01:45:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:54 smithi046 ceph-mon[36108]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 30 MiB/s wr, 5.41k op/s 2024-03-18T01:45:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:54 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:54 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:55.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:54 smithi007 ceph-mon[31996]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 30 MiB/s wr, 5.41k op/s 2024-03-18T01:45:55.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:54 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:54 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:54 smithi203 ceph-mon[36345]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 30 MiB/s wr, 5.41k op/s 2024-03-18T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:54 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:55.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:54 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:55 smithi046 ceph-mon[36108]: 1 slow requests, 0 included below; oldest blocked for > 197.155199 secs 2024-03-18T01:45:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:55 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 289 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:56.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:55 smithi007 ceph-mon[31996]: 1 slow requests, 0 included below; oldest blocked for > 197.155199 secs 2024-03-18T01:45:56.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:55 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 289 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:56.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:55 smithi203 ceph-mon[36345]: 1 slow requests, 0 included below; oldest blocked for > 197.155199 secs 2024-03-18T01:45:56.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:55 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 289 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:45:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:56 smithi046 ceph-mon[36108]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 28 MiB/s wr, 5.18k op/s 2024-03-18T01:45:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:56 smithi007 ceph-mon[31996]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 28 MiB/s wr, 5.18k op/s 2024-03-18T01:45:57.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:56 smithi203 ceph-mon[36345]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 28 MiB/s wr, 5.18k op/s 2024-03-18T01:45:58.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:58 smithi046 ceph-mon[36108]: 1 slow requests, 0 included below; oldest blocked for > 202.612541 secs 2024-03-18T01:45:58.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:58 smithi007 ceph-mon[31996]: 1 slow requests, 0 included below; oldest blocked for > 202.612541 secs 2024-03-18T01:45:58.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:58 smithi203 ceph-mon[36345]: 1 slow requests, 0 included below; oldest blocked for > 202.612541 secs 2024-03-18T01:45:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:59 smithi046 ceph-mon[36108]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 31 MiB/s wr, 5.86k op/s 2024-03-18T01:45:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:59 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:45:59 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:59 smithi007 ceph-mon[31996]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 31 MiB/s wr, 5.86k op/s 2024-03-18T01:45:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:59 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:45:59 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:45:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:59 smithi203 ceph-mon[36345]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 31 MiB/s wr, 5.86k op/s 2024-03-18T01:45:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:59 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:45:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:45:59 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:00.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:46:00.825 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:46:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:01 smithi007 ceph-mon[31996]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 28 MiB/s wr, 5.21k op/s 2024-03-18T01:46:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:01 smithi007 ceph-mon[31996]: 2 slow requests, 1 included below; oldest blocked for > 202.155269 secs 2024-03-18T01:46:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:01 smithi007 ceph-mon[31996]: slow request 180.003341 seconds old, received at 2024-03-18T01:43:00.258141+0000: peer_request:mds.4:2 currently dispatched 2024-03-18T01:46:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:01 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 295 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:01.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:01 smithi203 ceph-mon[36345]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 28 MiB/s wr, 5.21k op/s 2024-03-18T01:46:01.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:01 smithi203 ceph-mon[36345]: 2 slow requests, 1 included below; oldest blocked for > 202.155269 secs 2024-03-18T01:46:01.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:01 smithi203 ceph-mon[36345]: slow request 180.003341 seconds old, received at 2024-03-18T01:43:00.258141+0000: peer_request:mds.4:2 currently dispatched 2024-03-18T01:46:01.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:01 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 295 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:01.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:01 smithi046 ceph-mon[36108]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 28 MiB/s wr, 5.21k op/s 2024-03-18T01:46:01.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:01 smithi046 ceph-mon[36108]: 2 slow requests, 1 included below; oldest blocked for > 202.155269 secs 2024-03-18T01:46:01.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:01 smithi046 ceph-mon[36108]: slow request 180.003341 seconds old, received at 2024-03-18T01:43:00.258141+0000: peer_request:mds.4:2 currently dispatched 2024-03-18T01:46:01.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:01 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 295 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:01.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.558+0000 7f72cc800640 1 -- 172.21.15.7:0/1911384317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4110060 msgr2=0x7f72c4112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.558 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.558+0000 7f72cc800640 1 --2- 172.21.15.7:0/1911384317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4110060 0x7f72c4112450 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009900 tx=0x7f72b402bc40 comp rx=0 tx=0).stop 2024-03-18T01:46:01.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.558+0000 7f72cc800640 1 -- 172.21.15.7:0/1911384317 shutdown_connections 2024-03-18T01:46:01.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.558+0000 7f72cc800640 1 --2- 172.21.15.7:0/1911384317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4110060 0x7f72c4112450 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.558+0000 7f72cc800640 1 --2- 172.21.15.7:0/1911384317 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c4109140 0x7f72c410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.558+0000 7f72cc800640 1 --2- 172.21.15.7:0/1911384317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72c41087e0 0x7f72c4108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.558+0000 7f72cc800640 1 -- 172.21.15.7:0/1911384317 >> 172.21.15.7:0/1911384317 conn(0x7f72c406e780 msgr2=0x7f72c4070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:01.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.559+0000 7f72cc800640 1 -- 172.21.15.7:0/1911384317 shutdown_connections 2024-03-18T01:46:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.559+0000 7f72cc800640 1 -- 172.21.15.7:0/1911384317 wait complete. 2024-03-18T01:46:01.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.560+0000 7f72cc800640 1 Processor -- start 2024-03-18T01:46:01.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.561+0000 7f72cc800640 1 -- start start 2024-03-18T01:46:01.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.562+0000 7f72cc800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c41087e0 0x7f72c41a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.562+0000 7f72cc800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4109140 0x7f72c41a2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.562+0000 7f72cc800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72c4110060 0x7f72c41a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72c9e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c41087e0 0x7f72c41a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72c9400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4109140 0x7f72c41a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72c9400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4109140 0x7f72c41a2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59060/0 (socket says 172.21.15.7:59060) 2024-03-18T01:46:01.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72ca800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72c4110060 0x7f72c41a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72c9e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c41087e0 0x7f72c41a2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:48738/0 (socket says 172.21.15.7:48738) 2024-03-18T01:46:01.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72c9e00640 1 -- 172.21.15.7:0/2473228134 learned_addr learned my addr 172.21.15.7:0/2473228134 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:01.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f72c410a1b0 con 0x7f72c41087e0 2024-03-18T01:46:01.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72c410a030 con 0x7f72c4109140 2024-03-18T01:46:01.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.563+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72c410a330 con 0x7f72c4110060 2024-03-18T01:46:01.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.564+0000 7f72c9400640 1 -- 172.21.15.7:0/2473228134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72c4110060 msgr2=0x7f72c41a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.564+0000 7f72c9400640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72c4110060 0x7f72c41a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.564+0000 7f72c9400640 1 -- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c41087e0 msgr2=0x7f72c41a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.564+0000 7f72c9400640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c41087e0 0x7f72c41a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.565+0000 7f72c9400640 1 -- 172.21.15.7:0/2473228134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c41aa070 con 0x7f72c4109140 2024-03-18T01:46:01.567 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.567+0000 7f72c9400640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4109140 0x7f72c41a2990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f72b800c970 tx=0x7f72b800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:01.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.567+0000 7f72bea00640 1 -- 172.21.15.7:0/2473228134 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8007bb0 con 0x7f72c4109140 2024-03-18T01:46:01.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.567+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c41aa360 con 0x7f72c4109140 2024-03-18T01:46:01.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.567+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c41aa900 con 0x7f72c4109140 2024-03-18T01:46:01.568 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.567+0000 7f72bea00640 1 -- 172.21.15.7:0/2473228134 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b8007d50 con 0x7f72c4109140 2024-03-18T01:46:01.569 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.567+0000 7f72bea00640 1 -- 172.21.15.7:0/2473228134 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b801d060 con 0x7f72c4109140 2024-03-18T01:46:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.573+0000 7f72cf5e7740 1 Processor -- start 2024-03-18T01:46:01.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.573+0000 7f72cf5e7740 1 -- start start 2024-03-18T01:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.573+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931bd810 0x55f3931bdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.574+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931be200 0x55f3931c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.574+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931c5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.574+0000 7f72ca800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931bd810 0x55f3931bdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.574+0000 7f72ca800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931bd810 0x55f3931bdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:48874/0 (socket says 172.21.15.7:48874) 2024-03-18T01:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.574+0000 7f72ca800640 1 -- 172.21.15.7:0/353454996 learned_addr learned my addr 172.21.15.7:0/353454996 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.574+0000 7f72cf5e7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f3931221f0 con 0x55f3931c3370 2024-03-18T01:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.574+0000 7f72cf5e7740 1 -- 172.21.15.7:0/353454996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f3930caa80 con 0x55f3931be200 2024-03-18T01:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.574+0000 7f72cf5e7740 1 -- 172.21.15.7:0/353454996 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f39311e290 con 0x55f3931bd810 2024-03-18T01:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.575+0000 7f72bea00640 1 -- 172.21.15.7:0/2473228134 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f72b8005020 con 0x7f72c4109140 2024-03-18T01:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.576+0000 7f72ca800640 1 -- 172.21.15.7:0/353454996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931be200 msgr2=0x55f3931c2d00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.576+0000 7f72ca800640 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931be200 0x55f3931c2d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.577 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.576+0000 7f72ca800640 1 -- 172.21.15.7:0/353454996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 msgr2=0x55f3931c5800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:46:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.576+0000 7f72c9400640 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.577+0000 7f72ca800640 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.577+0000 7f72bea00640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72ac0a2880 0x7f72ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.577+0000 7f72bea00640 1 -- 172.21.15.7:0/2473228134 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72b80ce970 con 0x7f72c4109140 2024-03-18T01:46:01.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.577+0000 7f72ca800640 1 -- 172.21.15.7:0/353454996 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f392f007d0 con 0x55f3931bd810 2024-03-18T01:46:01.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.577+0000 7f72c9e00640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72ac0a2880 0x7f72ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.577+0000 7f72c9400640 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:46:01.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.579+0000 7f72c9e00640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72ac0a2880 0x7f72ac0a4d40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55f3931be600 tx=0x7f72c0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:01.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.582+0000 7f72ca800640 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931bd810 0x55f3931bdc30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f72b4005ab0 tx=0x7f72b402c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ffda6070b493dbb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:01.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.582+0000 7f72b3000640 1 -- 172.21.15.7:0/353454996 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4042070 con 0x55f3931bd810 2024-03-18T01:46:01.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.583+0000 7f72b3000640 1 -- 172.21.15.7:0/353454996 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b402cb90 con 0x55f3931bd810 2024-03-18T01:46:01.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.583+0000 7f72b3000640 1 -- 172.21.15.7:0/353454996 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4004920 con 0x55f3931bd810 2024-03-18T01:46:01.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.584+0000 7f72cf5e7740 1 -- 172.21.15.7:0/353454996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931bd810 msgr2=0x55f3931bdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.584+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931bd810 0x55f3931bdc30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f72b4005ab0 tx=0x7f72b402c890 comp rx=0 tx=0).stop 2024-03-18T01:46:01.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.584+0000 7f72cf5e7740 1 -- 172.21.15.7:0/353454996 shutdown_connections 2024-03-18T01:46:01.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.584+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.584+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931be200 0x55f3931c2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.584+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/353454996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931bd810 0x55f3931bdc30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.584+0000 7f72cf5e7740 1 -- 172.21.15.7:0/353454996 >> 172.21.15.7:0/353454996 conn(0x55f3931b8d80 msgr2=0x55f3931b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.587+0000 7f72cf5e7740 1 -- 172.21.15.7:0/353454996 shutdown_connections 2024-03-18T01:46:01.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.587+0000 7f72cf5e7740 1 -- 172.21.15.7:0/353454996 wait complete. 2024-03-18T01:46:01.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.587+0000 7f72cf5e7740 1 -- start start 2024-03-18T01:46:01.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.588+0000 7f72cf5e7740 1 Processor -- start 2024-03-18T01:46:01.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.588+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.588+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.588+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931d7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.589+0000 7f72cf5e7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f3931c62c0 con 0x55f3931c3370 2024-03-18T01:46:01.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.589+0000 7f72cf5e7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f3930dded0 con 0x55f3931bd810 2024-03-18T01:46:01.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.589+0000 7f72cf5e7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f3931c6440 con 0x55f3931be200 2024-03-18T01:46:01.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.589+0000 7f72c9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.590+0000 7f72c9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:48886/0 (socket says 172.21.15.7:48886) 2024-03-18T01:46:01.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.590+0000 7f72c9400640 1 -- 172.21.15.7:0/2960131960 learned_addr learned my addr 172.21.15.7:0/2960131960 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:01.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.590+0000 7f72cf5e7740 10 client.0 _ll_register_callbacks cb 0x55f393119220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:46:01.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.590+0000 7f72c9e00640 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.591+0000 7f72c9400640 1 -- 172.21.15.7:0/2960131960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 msgr2=0x55f3931d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:01.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.591+0000 7f72c9400640 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.591+0000 7f72c9400640 1 -- 172.21.15.7:0/2960131960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 msgr2=0x55f3931d7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:01.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.591+0000 7f72c9400640 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931d7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.591+0000 7f72c9400640 1 -- 172.21.15.7:0/2960131960 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f392e363f0 con 0x55f3931be200 2024-03-18T01:46:01.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.591+0000 7f72c9400640 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d0a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f72b8097980 tx=0x7f72b809aaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcfd1b6ea6395c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:01.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.591+0000 7f72a7400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:46:01.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.593+0000 7f72a7400640 1 -- 172.21.15.7:0/2960131960 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b809afb0 con 0x55f3931be200 2024-03-18T01:46:01.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.593+0000 7f72a7400640 1 -- 172.21.15.7:0/2960131960 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72b809a400 con 0x55f3931be200 2024-03-18T01:46:01.593 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.593+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2960131960 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b402cb90 con 0x55f3931be200 2024-03-18T01:46:01.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.593+0000 7f72a7400640 1 -- 172.21.15.7:0/2960131960 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f72b80b7020 con 0x55f3931be200 2024-03-18T01:46:01.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.593+0000 7f72cf5e7740 10 client.26269 fetch_fsmap learned FSMap version 132 2024-03-18T01:46:01.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.593+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2960131960 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=132,monmap=4+}) -- 0x55f392df60f0 con 0x55f3931be200 2024-03-18T01:46:01.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.594+0000 7f72a7400640 1 -- 172.21.15.7:0/2960131960 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b402cb90 con 0x55f3931be200 2024-03-18T01:46:01.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.594+0000 7f72a7400640 1 -- 172.21.15.7:0/2960131960 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 132) ==== 8370+0+0 (secure 0 0 0) 0x7f72b8096e20 con 0x55f3931be200 2024-03-18T01:46:01.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.595+0000 7f72cf5e7740 10 client.26269 fetch_fsmap finished waiting for FSMap version 132 2024-03-18T01:46:01.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.595+0000 7f72cf5e7740 10 client.26269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:46:01.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.595+0000 7f72cf5e7740 20 client.26269 populate_metadata read hostname 'smithi007' 2024-03-18T01:46:01.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.595+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f3931dc3d0 0x55f3931de820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.595+0000 7f72cf5e7740 4 client.26269 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:46:01.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.595+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2960131960 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f393135be0 con 0x55f3931dc3d0 2024-03-18T01:46:01.596 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.595+0000 7f72c9e00640 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f3931dc3d0 0x55f3931de820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.596+0000 7f72c9e00640 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f3931dc3d0 0x55f3931de820 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b63c51918b37408 server_cookie=ed4841aa1080beab in_seq=0 out_seq=0 2024-03-18T01:46:01.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.596+0000 7f72a7400640 10 client.26269 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:46:01.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.598+0000 7f72a7400640 1 -- 172.21.15.7:0/2960131960 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f393135be0 con 0x55f3931dc3d0 2024-03-18T01:46:01.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.598+0000 7f72a7400640 10 client.26269 handle_command_reply: tid=0 2024-03-18T01:46:01.598 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.598+0000 7f72cf5e7740 1 client.26269 shutdown 2024-03-18T01:46:01.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.598+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2960131960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 msgr2=0x55f3931d0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.599 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.598+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d0a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f72b8097980 tx=0x7f72b809aaf0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.599+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2960131960 shutdown_connections 2024-03-18T01:46:01.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.599+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f3931dc3d0 0x55f3931de820 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.599+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931d7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.599+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d0a80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.599+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2960131960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.599+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2960131960 >> 172.21.15.7:0/2960131960 conn(0x55f3931b8d80 msgr2=0x55f3931b7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:01.600 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.599+0000 7f72a7400640 0 client.26269 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:46:01.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.600+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2960131960 shutdown_connections 2024-03-18T01:46:01.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.600+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2960131960 wait complete. 2024-03-18T01:46:01.601 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.600+0000 7f72cf5e7740 20 client.26269 trim_cache size 0 max 16384 2024-03-18T01:46:01.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.654+0000 7f72cf5e7740 1 Processor -- start 2024-03-18T01:46:01.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.654+0000 7f72cf5e7740 1 -- start start 2024-03-18T01:46:01.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.655+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931bf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.655+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931bfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.655+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931d7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.655+0000 7f72cf5e7740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41b0910 con 0x55f3931c3370 2024-03-18T01:46:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.655+0000 7f72cf5e7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41b24b0 con 0x55f3931bd810 2024-03-18T01:46:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.655+0000 7f72cf5e7740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41b0fe0 con 0x55f3931be200 2024-03-18T01:46:01.656 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.655+0000 7f72c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931bf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.656+0000 7f72c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931bf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59076/0 (socket says 172.21.15.7:59076) 2024-03-18T01:46:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.656+0000 7f72c9e00640 1 -- 172.21.15.7:0/2381128248 learned_addr learned my addr 172.21.15.7:0/2381128248 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.656+0000 7f72ca800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931d7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.656+0000 7f72c9400640 1 --2- 172.21.15.7:0/2381128248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931bfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.658 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.658+0000 7f72ca800640 1 -- 172.21.15.7:0/2381128248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 msgr2=0x55f3931bfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:01.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.658+0000 7f72ca800640 1 --2- 172.21.15.7:0/2381128248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931bfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.658+0000 7f72ca800640 1 -- 172.21.15.7:0/2381128248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 msgr2=0x55f3931bf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:01.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.658+0000 7f72ca800640 1 --2- 172.21.15.7:0/2381128248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931bf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.659 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.658+0000 7f72ca800640 1 -- 172.21.15.7:0/2381128248 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3931b3370 con 0x55f3931c3370 2024-03-18T01:46:01.660 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.660+0000 7f72ca800640 1 --2- 172.21.15.7:0/2381128248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931d7a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55f3931bd560 tx=0x7f72b4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40cb6eb1d4741cde server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:01.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.662+0000 7f72b2600640 1 -- 172.21.15.7:0/2381128248 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b404a020 con 0x55f3931c3370 2024-03-18T01:46:01.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.664+0000 7f72b2600640 1 -- 172.21.15.7:0/2381128248 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b402cc50 con 0x55f3931c3370 2024-03-18T01:46:01.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.664+0000 7f72b2600640 1 -- 172.21.15.7:0/2381128248 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4042070 con 0x55f3931c3370 2024-03-18T01:46:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.665+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2381128248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 msgr2=0x55f3931d7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.665+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2381128248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931d7a80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55f3931bd560 tx=0x7f72b4005dd0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.665+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2381128248 shutdown_connections 2024-03-18T01:46:01.665 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.665+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2381128248 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931c3370 0x55f3931d7a80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.665+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2381128248 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931bfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.665+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/2381128248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931bd810 0x55f3931bf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.665+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2381128248 >> 172.21.15.7:0/2381128248 conn(0x55f3931b8d80 msgr2=0x55f3931d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.666+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2381128248 shutdown_connections 2024-03-18T01:46:01.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.666+0000 7f72cf5e7740 1 -- 172.21.15.7:0/2381128248 wait complete. 2024-03-18T01:46:01.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.667+0000 7f72cf5e7740 1 -- start start 2024-03-18T01:46:01.668 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.668+0000 7f72cf5e7740 1 Processor -- start 2024-03-18T01:46:01.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.670+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931bd810 0x55f3931d4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.670+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.670+0000 7f72cf5e7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931c3370 0x55f3931d63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.670+0000 7f72c9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72c9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:48916/0 (socket says 172.21.15.7:48916) 2024-03-18T01:46:01.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72c9400640 1 -- 172.21.15.7:0/1893367374 learned_addr learned my addr 172.21.15.7:0/1893367374 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:01.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72ca800640 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931c3370 0x55f3931d63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41b1de0 con 0x55f3931bd810 2024-03-18T01:46:01.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41b16b0 con 0x55f3931c3370 2024-03-18T01:46:01.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72c41ae910 con 0x55f3931be200 2024-03-18T01:46:01.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72ca800640 1 -- 172.21.15.7:0/1893367374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 msgr2=0x55f3931d5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:01.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72ca800640 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72ca800640 1 -- 172.21.15.7:0/1893367374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931bd810 msgr2=0x55f3931d4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:01.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72c9e00640 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931bd810 0x55f3931d4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.671+0000 7f72cf5e7740 10 client.0 _ll_register_callbacks cb 0x7f72b800a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:46:01.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.672+0000 7f72ca800640 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931bd810 0x55f3931d4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.672+0000 7f72ca800640 1 -- 172.21.15.7:0/1893367374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3931d6990 con 0x55f3931c3370 2024-03-18T01:46:01.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.672+0000 7f72c9e00640 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931bd810 0x55f3931d4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:46:01.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.673+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b402cc50 con 0x55f3931c3370 2024-03-18T01:46:01.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.673+0000 7f72ca800640 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931c3370 0x55f3931d63a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f72b4005ec0 tx=0x7f72b402ce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63984fb93ad464e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:01.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.673+0000 7f72a7400640 10 client.26297 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:46:01.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.675+0000 7f72a7400640 1 -- 172.21.15.7:0/1893367374 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b404a030 con 0x55f3931c3370 2024-03-18T01:46:01.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.675+0000 7f72a7400640 1 -- 172.21.15.7:0/1893367374 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72b402ca50 con 0x55f3931c3370 2024-03-18T01:46:01.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.675+0000 7f72a7400640 1 -- 172.21.15.7:0/1893367374 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f72b4042040 con 0x55f3931c3370 2024-03-18T01:46:01.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.676+0000 7f72cf5e7740 10 client.26297 fetch_fsmap learned FSMap version 132 2024-03-18T01:46:01.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.676+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({fsmap=132}) -- 0x55f3931d7b40 con 0x55f3931c3370 2024-03-18T01:46:01.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.677+0000 7f72a7400640 1 -- 172.21.15.7:0/1893367374 <== mon.2 v2:172.21.15.46:3300/0 4 ==== fsmap(e 132) ==== 8370+0+0 (secure 0 0 0) 0x7f72b4046500 con 0x55f3931c3370 2024-03-18T01:46:01.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.677+0000 7f72cf5e7740 10 client.26297 fetch_fsmap finished waiting for FSMap version 132 2024-03-18T01:46:01.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.677+0000 7f72cf5e7740 10 client.26297 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:46:01.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.677+0000 7f72cf5e7740 20 client.26297 populate_metadata read hostname 'smithi007' 2024-03-18T01:46:01.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.677+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f393242720 0x55f393244b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:01.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.677+0000 7f72cf5e7740 4 client.26297 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:46:01.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.677+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f39312f730 con 0x55f393242720 2024-03-18T01:46:01.678 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.678+0000 7f72c9e00640 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f393242720 0x55f393244b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:01.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.679+0000 7f72c9e00640 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f393242720 0x55f393244b10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=965ea214644dbd36 server_cookie=d4a7dc55de491d7b in_seq=0 out_seq=0 2024-03-18T01:46:01.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.680+0000 7f72a7400640 10 client.26297 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:46:01.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.681+0000 7f72a7400640 1 -- 172.21.15.7:0/1893367374 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55f39312f730 con 0x55f393242720 2024-03-18T01:46:01.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.681+0000 7f72a7400640 10 client.26297 handle_command_reply: tid=0 2024-03-18T01:46:01.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.681+0000 7f72cf5e7740 1 client.26297 shutdown 2024-03-18T01:46:01.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.682+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931c3370 msgr2=0x55f3931d63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.682+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931c3370 0x55f3931d63a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f72b4005ec0 tx=0x7f72b402ce00 comp rx=0 tx=0).stop 2024-03-18T01:46:01.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.684+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 shutdown_connections 2024-03-18T01:46:01.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.684+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f393242720 0x55f393244b10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.684+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f3931c3370 0x55f3931d63a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.684+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f3931be200 0x55f3931d5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.684+0000 7f72cf5e7740 1 --2- 172.21.15.7:0/1893367374 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f3931bd810 0x55f3931d4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.684+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 >> 172.21.15.7:0/1893367374 conn(0x55f3931b8d80 msgr2=0x55f3931d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:01.685 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.684+0000 7f72a7400640 0 client.26297 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:46:01.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.685+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 shutdown_connections 2024-03-18T01:46:01.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.685+0000 7f72cf5e7740 1 -- 172.21.15.7:0/1893367374 wait complete. 2024-03-18T01:46:01.686 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.685+0000 7f72cf5e7740 20 client.26297 trim_cache size 0 max 16384 2024-03-18T01:46:01.686 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:46:01.686 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-18T01:46:01.686 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:46:01.687 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:46:01.687 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:46:01.687 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:46:01.687 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:46:01.687 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:46:01.687 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:46:01.687 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72ac0a2880 msgr2=0x7f72ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72ac0a2880 0x7f72ac0a4d40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55f3931be600 tx=0x7f72c0009500 comp rx=0 tx=0).stop 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4109140 msgr2=0x7f72c41a2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4109140 0x7f72c41a2990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f72b800c970 tx=0x7f72b800ce30 comp rx=0 tx=0).stop 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 shutdown_connections 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72c4110060 0x7f72c41a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f72c4109140 0x7f72c41a2990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f72ac0a2880 0x7f72ac0a4d40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 --2- 172.21.15.7:0/2473228134 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f72c41087e0 0x7f72c41a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 >> 172.21.15.7:0/2473228134 conn(0x7f72c406e780 msgr2=0x7f72c4070330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:01.688 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 shutdown_connections 2024-03-18T01:46:01.689 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:01.687+0000 7f72cc800640 1 -- 172.21.15.7:0/2473228134 wait complete. 2024-03-18T01:46:01.719 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:46:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:03 smithi007 ceph-mon[31996]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 31 MiB/s wr, 5.99k op/s 2024-03-18T01:46:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:03 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:03 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:03 smithi007 ceph-mon[31996]: 2 slow requests, 1 included below; oldest blocked for > 207.612610 secs 2024-03-18T01:46:03.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:03 smithi007 ceph-mon[31996]: slow request 182.606201 seconds old, received at 2024-03-18T01:43:00.258149+0000: internal op exportdir:mds.4:2 currently failed to wrlock, waiting 2024-03-18T01:46:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:03 smithi203 ceph-mon[36345]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 31 MiB/s wr, 5.99k op/s 2024-03-18T01:46:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:03 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:03.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:03 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:03.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:03 smithi203 ceph-mon[36345]: 2 slow requests, 1 included below; oldest blocked for > 207.612610 secs 2024-03-18T01:46:03.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:03 smithi203 ceph-mon[36345]: slow request 182.606201 seconds old, received at 2024-03-18T01:43:00.258149+0000: internal op exportdir:mds.4:2 currently failed to wrlock, waiting 2024-03-18T01:46:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:03 smithi046 ceph-mon[36108]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 31 MiB/s wr, 5.99k op/s 2024-03-18T01:46:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:03 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:03 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:03 smithi046 ceph-mon[36108]: 2 slow requests, 1 included below; oldest blocked for > 207.612610 secs 2024-03-18T01:46:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:03 smithi046 ceph-mon[36108]: slow request 182.606201 seconds old, received at 2024-03-18T01:43:00.258149+0000: internal op exportdir:mds.4:2 currently failed to wrlock, waiting 2024-03-18T01:46:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:05 smithi203 ceph-mon[36345]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 33 MiB/s wr, 6.39k op/s 2024-03-18T01:46:05.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:05 smithi046 ceph-mon[36108]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 33 MiB/s wr, 6.39k op/s 2024-03-18T01:46:05.558 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:05 smithi007 ceph-mon[31996]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 33 MiB/s wr, 6.39k op/s 2024-03-18T01:46:06.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:06 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 207.158768 secs 2024-03-18T01:46:06.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:06 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 207.158768 secs 2024-03-18T01:46:06.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:06 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 207.158768 secs 2024-03-18T01:46:07.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:07 smithi203 ceph-mon[36345]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 30 MiB/s wr, 5.86k op/s 2024-03-18T01:46:07.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:07 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 300 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:07 smithi046 ceph-mon[36108]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 30 MiB/s wr, 5.86k op/s 2024-03-18T01:46:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:07 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 300 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:07 smithi007 ceph-mon[31996]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 30 MiB/s wr, 5.86k op/s 2024-03-18T01:46:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:07 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 300 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:08.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:08 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:46:08.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:08 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:08.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:08 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 212.613442 secs 2024-03-18T01:46:08.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:08 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:46:08.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:08 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:08.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:08 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 212.613442 secs 2024-03-18T01:46:08.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:08 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:46:08.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:08 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:08.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:08 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 212.613442 secs 2024-03-18T01:46:09.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:09 smithi203 ceph-mon[36345]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 33 MiB/s wr, 6.63k op/s 2024-03-18T01:46:09.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:09 smithi046 ceph-mon[36108]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 33 MiB/s wr, 6.63k op/s 2024-03-18T01:46:09.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:09 smithi007 ceph-mon[31996]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 33 MiB/s wr, 6.63k op/s 2024-03-18T01:46:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:10 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:10.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:10 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:10.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:10 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:10.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:10 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:10 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:10.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:10 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:11.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:11 smithi046 ceph-mon[36108]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 29 MiB/s wr, 5.79k op/s 2024-03-18T01:46:11.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:11 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 212.158837 secs 2024-03-18T01:46:11.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:11 smithi007 ceph-mon[31996]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 29 MiB/s wr, 5.79k op/s 2024-03-18T01:46:11.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:11 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 212.158837 secs 2024-03-18T01:46:11.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:11 smithi203 ceph-mon[36345]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 29 MiB/s wr, 5.79k op/s 2024-03-18T01:46:11.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:11 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 212.158837 secs 2024-03-18T01:46:13.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:13 smithi046 ceph-mon[36108]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 31 MiB/s wr, 6.25k op/s 2024-03-18T01:46:13.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:13 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 217.613509 secs 2024-03-18T01:46:13.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:13 smithi007 ceph-mon[31996]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 31 MiB/s wr, 6.25k op/s 2024-03-18T01:46:13.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:13 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 217.613509 secs 2024-03-18T01:46:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:13 smithi203 ceph-mon[36345]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 31 MiB/s wr, 6.25k op/s 2024-03-18T01:46:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:13 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 217.613509 secs 2024-03-18T01:46:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:14 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:14.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:14 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:14.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:14 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:14.542 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:14 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:14.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:14 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:14.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:14 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:15.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:15 smithi046 ceph-mon[36108]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 28 MiB/s wr, 5.68k op/s 2024-03-18T01:46:15.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:15 smithi007 ceph-mon[31996]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 28 MiB/s wr, 5.68k op/s 2024-03-18T01:46:15.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:15 smithi203 ceph-mon[36345]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 28 MiB/s wr, 5.68k op/s 2024-03-18T01:46:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:16 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 217.158984 secs 2024-03-18T01:46:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:16 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 305 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:16.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:16 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:46:16.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:16 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 217.158984 secs 2024-03-18T01:46:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:16 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 305 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:16.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:16 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:46:16.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:16 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 217.158984 secs 2024-03-18T01:46:16.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:16 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 305 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:16.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:16 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:46:17.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:17 smithi046 ceph-mon[36108]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 28 MiB/s wr, 5.65k op/s 2024-03-18T01:46:17.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:17 smithi007 ceph-mon[31996]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 28 MiB/s wr, 5.65k op/s 2024-03-18T01:46:17.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:17 smithi203 ceph-mon[36345]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 28 MiB/s wr, 5.65k op/s 2024-03-18T01:46:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:18 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:18 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:18.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:18 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 222.613578 secs 2024-03-18T01:46:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:18 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:18.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:18 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:18.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:18 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 222.613578 secs 2024-03-18T01:46:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:18 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:18.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:18 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:18.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:18 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 222.613578 secs 2024-03-18T01:46:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:19 smithi007 ceph-mon[31996]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 31 MiB/s wr, 6.35k op/s 2024-03-18T01:46:19.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:19 smithi203 ceph-mon[36345]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 31 MiB/s wr, 6.35k op/s 2024-03-18T01:46:19.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:19 smithi046 ceph-mon[36108]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 31 MiB/s wr, 6.35k op/s 2024-03-18T01:46:21.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:21 smithi007 ceph-mon[31996]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 5.07k op/s 2024-03-18T01:46:21.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:21 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 222.159046 secs 2024-03-18T01:46:21.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:21 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 310 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:21.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:21 smithi203 ceph-mon[36345]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 5.07k op/s 2024-03-18T01:46:21.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:21 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 222.159046 secs 2024-03-18T01:46:21.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:21 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 310 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:21.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:21 smithi046 ceph-mon[36108]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 24 MiB/s wr, 5.07k op/s 2024-03-18T01:46:21.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:21 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 222.159046 secs 2024-03-18T01:46:21.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:21 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 310 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:22 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:22 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:22.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:22 smithi046 ceph-mon[36108]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 28 MiB/s wr, 5.84k op/s 2024-03-18T01:46:22.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:22 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:22.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:22 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:22.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:22 smithi007 ceph-mon[31996]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 28 MiB/s wr, 5.84k op/s 2024-03-18T01:46:22.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:22 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:22.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:22 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:22.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:22 smithi203 ceph-mon[36345]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 28 MiB/s wr, 5.84k op/s 2024-03-18T01:46:23.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:23 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 227.613640 secs 2024-03-18T01:46:23.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:23 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:46:23.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:23 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:46:23.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:23 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:46:23.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:23 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 227.613640 secs 2024-03-18T01:46:23.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:23 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:46:23.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:23 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:46:23.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:23 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:46:23.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:23 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 227.613640 secs 2024-03-18T01:46:23.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:23 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:46:23.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:23 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:46:23.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:23 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:46:24.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:24 smithi046 ceph-mon[36108]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 25 MiB/s wr, 5.23k op/s 2024-03-18T01:46:24.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:24 smithi007 ceph-mon[31996]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 25 MiB/s wr, 5.23k op/s 2024-03-18T01:46:24.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:24 smithi203 ceph-mon[36345]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 25 MiB/s wr, 5.23k op/s 2024-03-18T01:46:25.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:25 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 227.159723 secs 2024-03-18T01:46:25.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:25 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 227.159723 secs 2024-03-18T01:46:25.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:25 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 227.159723 secs 2024-03-18T01:46:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:26 smithi046 ceph-mon[36108]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 5.22k op/s 2024-03-18T01:46:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:26 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 318 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:26.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:26 smithi007 ceph-mon[31996]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 5.22k op/s 2024-03-18T01:46:26.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:26 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 318 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:26.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:26 smithi203 ceph-mon[36345]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 MiB/s wr, 5.22k op/s 2024-03-18T01:46:26.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:26 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 318 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:29.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:28 smithi046 ceph-mon[36108]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 23 MiB/s wr, 4.85k op/s 2024-03-18T01:46:29.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:28 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 232.613704 secs 2024-03-18T01:46:29.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:28 smithi007 ceph-mon[31996]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 23 MiB/s wr, 4.85k op/s 2024-03-18T01:46:29.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:28 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 232.613704 secs 2024-03-18T01:46:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:28 smithi203 ceph-mon[36345]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 23 MiB/s wr, 4.85k op/s 2024-03-18T01:46:29.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:28 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 232.613704 secs 2024-03-18T01:46:31.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:30 smithi046 ceph-mon[36108]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 3.75k op/s 2024-03-18T01:46:31.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:30 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:31.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:30 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:31.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:30 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 232.159767 secs 2024-03-18T01:46:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:30 smithi007 ceph-mon[31996]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 3.75k op/s 2024-03-18T01:46:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:30 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:30 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:31.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:30 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 232.159767 secs 2024-03-18T01:46:31.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:30 smithi203 ceph-mon[36345]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 3.75k op/s 2024-03-18T01:46:31.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:30 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:31.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:30 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:31.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:30 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 232.159767 secs 2024-03-18T01:46:31.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:46:31.721 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:46:32.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:31 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 324 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:32.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:31 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 324 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:32.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:31 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 324 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.162+0000 7f9981c00640 1 -- 172.21.15.7:0/231190337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 msgr2=0x7f997c10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.162+0000 7f9981c00640 1 --2- 172.21.15.7:0/231190337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 0x7f997c10df80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f996c009900 tx=0x7f996c02bc50 comp rx=0 tx=0).stop 2024-03-18T01:46:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.162+0000 7f9981c00640 1 -- 172.21.15.7:0/231190337 shutdown_connections 2024-03-18T01:46:32.163 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.163+0000 7f9981c00640 1 --2- 172.21.15.7:0/231190337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 0x7f997c10df80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.163+0000 7f9981c00640 1 --2- 172.21.15.7:0/231190337 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c10b150 0x7f997c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.163+0000 7f9981c00640 1 --2- 172.21.15.7:0/231190337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f997c1087e0 0x7f997c10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.163+0000 7f9981c00640 1 -- 172.21.15.7:0/231190337 >> 172.21.15.7:0/231190337 conn(0x7f997c06e780 msgr2=0x7f997c070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.163+0000 7f9981c00640 1 -- 172.21.15.7:0/231190337 shutdown_connections 2024-03-18T01:46:32.164 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.163+0000 7f9981c00640 1 -- 172.21.15.7:0/231190337 wait complete. 2024-03-18T01:46:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.164+0000 7f9981c00640 1 Processor -- start 2024-03-18T01:46:32.165 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.165+0000 7f9981c00640 1 -- start start 2024-03-18T01:46:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.166+0000 7f9981c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c1087e0 0x7f997c1a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.166+0000 7f9981c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f997c10b150 0x7f997c1a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.166 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.166+0000 7f9981c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 0x7f997c1adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f997c10b150 0x7f997c1a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f9981c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f997c11c390 con 0x7f997c1087e0 2024-03-18T01:46:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f9981c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f997c11c210 con 0x7f997c10db00 2024-03-18T01:46:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f9981c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f997c11c510 con 0x7f997c10b150 2024-03-18T01:46:32.167 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997aa00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c1087e0 0x7f997c1a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 0x7f997c1adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f997c10b150 0x7f997c1a6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:49940/0 (socket says 172.21.15.7:49940) 2024-03-18T01:46:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 0x7f997c1adee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:43418/0 (socket says 172.21.15.7:43418) 2024-03-18T01:46:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997a000640 1 -- 172.21.15.7:0/4216100938 learned_addr learned my addr 172.21.15.7:0/4216100938 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997b400640 1 -- 172.21.15.7:0/4216100938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f997c10b150 msgr2=0x7f997c1a6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997b400640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f997c10b150 0x7f997c1a6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.168 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.167+0000 7f997b400640 1 -- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c1087e0 msgr2=0x7f997c1a6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.168+0000 7f997b400640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c1087e0 0x7f997c1a6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.168+0000 7f997b400640 1 -- 172.21.15.7:0/4216100938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c1ae550 con 0x7f997c10db00 2024-03-18T01:46:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.168+0000 7f997aa00640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c1087e0 0x7f997c1a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:46:32.169 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.169+0000 7f997b400640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 0x7f997c1adee0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f996c036040 tx=0x7f996c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:32.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.169+0000 7f9973400640 1 -- 172.21.15.7:0/4216100938 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c043020 con 0x7f997c10db00 2024-03-18T01:46:32.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.169+0000 7f9973400640 1 -- 172.21.15.7:0/4216100938 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f996c003ad0 con 0x7f997c10db00 2024-03-18T01:46:32.170 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.169+0000 7f9973400640 1 -- 172.21.15.7:0/4216100938 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c03ea00 con 0x7f997c10db00 2024-03-18T01:46:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.169+0000 7f9981c00640 1 -- 172.21.15.7:0/4216100938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c1ae7e0 con 0x7f997c10db00 2024-03-18T01:46:32.173 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.169+0000 7f9981c00640 1 -- 172.21.15.7:0/4216100938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c1aece0 con 0x7f997c10db00 2024-03-18T01:46:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.175+0000 7f9973400640 1 -- 172.21.15.7:0/4216100938 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f996c046020 con 0x7f997c10db00 2024-03-18T01:46:32.175 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.175+0000 7f9984986740 1 Processor -- start 2024-03-18T01:46:32.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.178+0000 7f9984986740 1 -- start start 2024-03-18T01:46:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.178+0000 7f9984986740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b5fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.178+0000 7f9984986740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b60200 0x563fc7b64d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.178+0000 7f9984986740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b65370 0x563fc7b67800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.178+0000 7f9984986740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x563fc7ac41f0 con 0x563fc7b65370 2024-03-18T01:46:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.178+0000 7f9984986740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563fc7a6ca80 con 0x563fc7b5f810 2024-03-18T01:46:32.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.178+0000 7f9984986740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563fc7ac0290 con 0x563fc7b60200 2024-03-18T01:46:32.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.180+0000 7f997aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b60200 0x563fc7b64d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.180+0000 7f997aa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b60200 0x563fc7b64d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:49956/0 (socket says 172.21.15.7:49956) 2024-03-18T01:46:32.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.180+0000 7f997aa00640 1 -- 172.21.15.7:0/3429275087 learned_addr learned my addr 172.21.15.7:0/3429275087 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:32.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.180+0000 7f9973400640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f99600a2840 0x7f99600a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.180+0000 7f997a000640 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b5fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.180+0000 7f997b400640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f99600a2840 0x7f99600a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.181+0000 7f997aa00640 1 -- 172.21.15.7:0/3429275087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 msgr2=0x563fc7b5fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:32.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.181+0000 7f997aa00640 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b5fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.181+0000 7f997a000640 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b65370 0x563fc7b67800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.181+0000 7f997aa00640 1 -- 172.21.15.7:0/3429275087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b65370 msgr2=0x563fc7b67800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.181+0000 7f997aa00640 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b65370 0x563fc7b67800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.181+0000 7f997aa00640 1 -- 172.21.15.7:0/3429275087 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc78a27d0 con 0x563fc7b60200 2024-03-18T01:46:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.181+0000 7f997a000640 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b65370 0x563fc7b67800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:46:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.182+0000 7f997aa00640 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b60200 0x563fc7b64d00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f997400c450 tx=0x7f997400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d0dac154504d94c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.182+0000 7f996be00640 1 -- 172.21.15.7:0/3429275087 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997401a070 con 0x563fc7b60200 2024-03-18T01:46:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.182+0000 7f996be00640 1 -- 172.21.15.7:0/3429275087 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9974007b00 con 0x563fc7b60200 2024-03-18T01:46:32.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.182+0000 7f996be00640 1 -- 172.21.15.7:0/3429275087 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974015c70 con 0x563fc7b60200 2024-03-18T01:46:32.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.184+0000 7f9973400640 1 -- 172.21.15.7:0/4216100938 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f996c0f16c0 con 0x7f997c10db00 2024-03-18T01:46:32.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.184+0000 7f9984986740 1 -- 172.21.15.7:0/3429275087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b60200 msgr2=0x563fc7b64d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.184+0000 7f997b400640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f99600a2840 0x7f99600a4d00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f996c009620 tx=0x7f996c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.184+0000 7f9984986740 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b60200 0x563fc7b64d00 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f997400c450 tx=0x7f997400c910 comp rx=0 tx=0).stop 2024-03-18T01:46:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.184+0000 7f9984986740 1 -- 172.21.15.7:0/3429275087 shutdown_connections 2024-03-18T01:46:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.184+0000 7f9984986740 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b65370 0x563fc7b67800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.185+0000 7f9984986740 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b60200 0x563fc7b64d00 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.185+0000 7f9984986740 1 --2- 172.21.15.7:0/3429275087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b5fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.185+0000 7f9984986740 1 -- 172.21.15.7:0/3429275087 >> 172.21.15.7:0/3429275087 conn(0x563fc7b5ad80 msgr2=0x563fc7b5b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.186+0000 7f9984986740 1 -- 172.21.15.7:0/3429275087 shutdown_connections 2024-03-18T01:46:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.186+0000 7f9984986740 1 -- 172.21.15.7:0/3429275087 wait complete. 2024-03-18T01:46:32.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.186+0000 7f9984986740 1 -- start start 2024-03-18T01:46:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.187+0000 7f9984986740 1 Processor -- start 2024-03-18T01:46:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.188+0000 7f9984986740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b69d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.188+0000 7f9984986740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.188+0000 7f9984986740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 0x563fc7b712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.188+0000 7f9984986740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x563fc7b68320 con 0x563fc7b60200 2024-03-18T01:46:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.188+0000 7f9984986740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x563fc7a7fed0 con 0x563fc7b65370 2024-03-18T01:46:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.188+0000 7f9984986740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563fc7b684a0 con 0x563fc7b5f810 2024-03-18T01:46:32.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.188+0000 7f9984986740 10 client.0 _ll_register_callbacks cb 0x563fc7abb220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:46:32.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.190+0000 7f997a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b69d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.190+0000 7f997a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b69d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:49960/0 (socket says 172.21.15.7:49960) 2024-03-18T01:46:32.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.190+0000 7f997a000640 1 -- 172.21.15.7:0/4119626219 learned_addr learned my addr 172.21.15.7:0/4119626219 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:32.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.190+0000 7f997a000640 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 0x563fc7b712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.192+0000 7f997aa00640 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.193+0000 7f997aa00640 1 -- 172.21.15.7:0/4119626219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 msgr2=0x563fc7b69d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:32.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.193+0000 7f997aa00640 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b69d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.193+0000 7f997aa00640 1 -- 172.21.15.7:0/4119626219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 msgr2=0x563fc7b712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.193+0000 7f997aa00640 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 0x563fc7b712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.193+0000 7f997aa00640 1 -- 172.21.15.7:0/4119626219 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc77d83f0 con 0x563fc7b60200 2024-03-18T01:46:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.194+0000 7f997aa00640 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6a2c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x563fc7b60600 tx=0x7f9974015590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb0db7feb74b154 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.194+0000 7f9968c00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:46:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.195+0000 7f9968c00640 1 -- 172.21.15.7:0/4119626219 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9974023020 con 0x563fc7b60200 2024-03-18T01:46:32.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.195+0000 7f9984986740 1 -- 172.21.15.7:0/4119626219 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9974007b00 con 0x563fc7b60200 2024-03-18T01:46:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.196+0000 7f9968c00640 1 -- 172.21.15.7:0/4119626219 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9974007620 con 0x563fc7b60200 2024-03-18T01:46:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.196+0000 7f9968c00640 1 -- 172.21.15.7:0/4119626219 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f997401a040 con 0x563fc7b60200 2024-03-18T01:46:32.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.196+0000 7f9984986740 10 client.15984 fetch_fsmap learned FSMap version 139 2024-03-18T01:46:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.196+0000 7f9984986740 1 -- 172.21.15.7:0/4119626219 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x563fc77980f0 con 0x563fc7b60200 2024-03-18T01:46:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.197+0000 7f9968c00640 1 -- 172.21.15.7:0/4119626219 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9974007b00 con 0x563fc7b60200 2024-03-18T01:46:32.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.197+0000 7f9968c00640 1 -- 172.21.15.7:0/4119626219 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 139) ==== 8370+0+0 (secure 0 0 0) 0x7f997401f7f0 con 0x563fc7b60200 2024-03-18T01:46:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.197+0000 7f9984986740 10 client.15984 fetch_fsmap finished waiting for FSMap version 139 2024-03-18T01:46:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.197+0000 7f9984986740 10 client.15984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:46:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.197+0000 7f9984986740 20 client.15984 populate_metadata read hostname 'smithi007' 2024-03-18T01:46:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.197+0000 7f9984986740 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563fc7b758a0 0x563fc7b77c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.197+0000 7f9984986740 4 client.15984 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:46:32.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.197+0000 7f9984986740 1 -- 172.21.15.7:0/4119626219 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fc7ad7be0 con 0x563fc7b758a0 2024-03-18T01:46:32.199 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.198+0000 7f997a000640 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563fc7b758a0 0x563fc7b77c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.200+0000 7f997a000640 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563fc7b758a0 0x563fc7b77c90 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b9da1c9055cf9db server_cookie=4679b96729109ba9 in_seq=0 out_seq=0 2024-03-18T01:46:32.201 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.200+0000 7f9968c00640 10 client.15984 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:46:32.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.205+0000 7f9968c00640 1 -- 172.21.15.7:0/4119626219 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563fc7ad7be0 con 0x563fc7b758a0 2024-03-18T01:46:32.205 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.205+0000 7f9968c00640 10 client.15984 handle_command_reply: tid=0 2024-03-18T01:46:32.207 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.205+0000 7f9984986740 1 client.15984 shutdown 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 -- 172.21.15.7:0/4119626219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 msgr2=0x563fc7b6a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6a2c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x563fc7b60600 tx=0x7f9974015590 comp rx=0 tx=0).stop 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 -- 172.21.15.7:0/4119626219 shutdown_connections 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563fc7b758a0 0x563fc7b77c90 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 0x563fc7b712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6a2c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 --2- 172.21.15.7:0/4119626219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b69d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 -- 172.21.15.7:0/4119626219 >> 172.21.15.7:0/4119626219 conn(0x563fc7b5ad80 msgr2=0x563fc7b59ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9968c00640 0 client.15984 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:46:32.208 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.207+0000 7f9984986740 1 -- 172.21.15.7:0/4119626219 shutdown_connections 2024-03-18T01:46:32.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.208+0000 7f9984986740 1 -- 172.21.15.7:0/4119626219 wait complete. 2024-03-18T01:46:32.209 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.208+0000 7f9984986740 20 client.15984 trim_cache size 0 max 16384 2024-03-18T01:46:32.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.266+0000 7f9984986740 1 Processor -- start 2024-03-18T01:46:32.267 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.266+0000 7f9984986740 1 -- start start 2024-03-18T01:46:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f9984986740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b59630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.268 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f9984986740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b71190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f9984986740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b65370 0x563fc7b59b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f9984986740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1b9200 con 0x563fc7b60200 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f9984986740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1bada0 con 0x563fc7b5f810 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f9984986740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1b98d0 con 0x563fc7b65370 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f997a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b59630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f997a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b59630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:43440/0 (socket says 172.21.15.7:43440) 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f997a000640 1 -- 172.21.15.7:0/2665809458 learned_addr learned my addr 172.21.15.7:0/2665809458 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.267+0000 7f997a000640 1 --2- 172.21.15.7:0/2665809458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b65370 0x563fc7b59b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.268+0000 7f997a000640 1 -- 172.21.15.7:0/2665809458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b65370 msgr2=0x563fc7b59b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.268+0000 7f997a000640 1 --2- 172.21.15.7:0/2665809458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b65370 0x563fc7b59b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.268+0000 7f997a000640 1 -- 172.21.15.7:0/2665809458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 msgr2=0x563fc7b71190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:32.269 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.268+0000 7f997a000640 1 --2- 172.21.15.7:0/2665809458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b71190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.268+0000 7f997a000640 1 -- 172.21.15.7:0/2665809458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fc7b61210 con 0x563fc7b5f810 2024-03-18T01:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.269+0000 7f997a000640 1 --2- 172.21.15.7:0/2665809458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b59630 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f99640095a0 tx=0x7f99640047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1ea8d84d43a6913 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:32.270 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.269+0000 7f996b400640 1 -- 172.21.15.7:0/2665809458 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99640222a0 con 0x563fc7b5f810 2024-03-18T01:46:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.269+0000 7f996b400640 1 -- 172.21.15.7:0/2665809458 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9964016800 con 0x563fc7b5f810 2024-03-18T01:46:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.269+0000 7f996b400640 1 -- 172.21.15.7:0/2665809458 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9964015620 con 0x563fc7b5f810 2024-03-18T01:46:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.270+0000 7f9984986740 1 -- 172.21.15.7:0/2665809458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 msgr2=0x563fc7b59630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.270+0000 7f9984986740 1 --2- 172.21.15.7:0/2665809458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b59630 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f99640095a0 tx=0x7f99640047d0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.271+0000 7f9984986740 1 -- 172.21.15.7:0/2665809458 shutdown_connections 2024-03-18T01:46:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.271+0000 7f9984986740 1 --2- 172.21.15.7:0/2665809458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b65370 0x563fc7b59b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.271 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.271+0000 7f9984986740 1 --2- 172.21.15.7:0/2665809458 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b71190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.271+0000 7f9984986740 1 --2- 172.21.15.7:0/2665809458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b5f810 0x563fc7b59630 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.271+0000 7f9984986740 1 -- 172.21.15.7:0/2665809458 >> 172.21.15.7:0/2665809458 conn(0x563fc7b5ad80 msgr2=0x563fc7b61740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.271+0000 7f9984986740 1 -- 172.21.15.7:0/2665809458 shutdown_connections 2024-03-18T01:46:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.271+0000 7f9984986740 1 -- 172.21.15.7:0/2665809458 wait complete. 2024-03-18T01:46:32.272 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.271+0000 7f9984986740 1 -- start start 2024-03-18T01:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.272+0000 7f9984986740 1 Processor -- start 2024-03-18T01:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.272+0000 7f9984986740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b6e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.272+0000 7f9984986740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.272+0000 7f9984986740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 0x563fc7b6fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.272+0000 7f9984986740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1ba6d0 con 0x563fc7b60200 2024-03-18T01:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.272+0000 7f9984986740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1b9fa0 con 0x563fc7b65370 2024-03-18T01:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.272+0000 7f9984986740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f997c1b7180 con 0x563fc7b5f810 2024-03-18T01:46:32.273 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.272+0000 7f9984986740 10 client.0 _ll_register_callbacks cb 0x7f9974018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.273+0000 7f997a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b6e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.273+0000 7f997a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b6e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:49988/0 (socket says 172.21.15.7:49988) 2024-03-18T01:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.273+0000 7f997a000640 1 -- 172.21.15.7:0/2910684385 learned_addr learned my addr 172.21.15.7:0/2910684385 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:46:32.276 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.273+0000 7f997a000640 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 0x563fc7b6fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.275+0000 7f997aa00640 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.276+0000 7f997aa00640 1 -- 172.21.15.7:0/2910684385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 msgr2=0x563fc7b6e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.276+0000 7f997aa00640 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b6e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.276+0000 7f997aa00640 1 -- 172.21.15.7:0/2910684385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 msgr2=0x563fc7b6fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.276+0000 7f997aa00640 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 0x563fc7b6fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.276+0000 7f997aa00640 1 -- 172.21.15.7:0/2910684385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fc7b700a0 con 0x563fc7b60200 2024-03-18T01:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.277+0000 7f997aa00640 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6eaf0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f997401fb60 tx=0x7f9974007020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5299cb2e136ea09f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:46:32.277 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.277+0000 7f9968c00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.278+0000 7f9968c00640 1 -- 172.21.15.7:0/2910684385 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997401a040 con 0x563fc7b60200 2024-03-18T01:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.278+0000 7f9968c00640 1 -- 172.21.15.7:0/2910684385 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9974007620 con 0x563fc7b60200 2024-03-18T01:46:32.278 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.278+0000 7f9984986740 1 -- 172.21.15.7:0/2910684385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9964016800 con 0x563fc7b60200 2024-03-18T01:46:32.279 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.279+0000 7f9968c00640 1 -- 172.21.15.7:0/2910684385 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) ==== 24+0+0 (secure 0 0 0) 0x7f9974003a30 con 0x563fc7b60200 2024-03-18T01:46:32.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.280+0000 7f9984986740 10 client.15987 fetch_fsmap learned FSMap version 139 2024-03-18T01:46:32.280 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.280+0000 7f9984986740 1 -- 172.21.15.7:0/2910684385 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=4+}) -- 0x563fc7b713b0 con 0x563fc7b60200 2024-03-18T01:46:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.280+0000 7f9968c00640 1 -- 172.21.15.7:0/2910684385 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9964016800 con 0x563fc7b60200 2024-03-18T01:46:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.280+0000 7f9968c00640 1 -- 172.21.15.7:0/2910684385 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 139) ==== 8370+0+0 (secure 0 0 0) 0x7f9974040070 con 0x563fc7b60200 2024-03-18T01:46:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.281+0000 7f9984986740 10 client.15987 fetch_fsmap finished waiting for FSMap version 139 2024-03-18T01:46:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.281+0000 7f9984986740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:46:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.281+0000 7f9984986740 20 client.15987 populate_metadata read hostname 'smithi007' 2024-03-18T01:46:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.281+0000 7f9984986740 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563fc7bdc0f0 0x563fc7bde510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:46:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.281+0000 7f9984986740 4 client.15987 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:46:32.281 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.281+0000 7f9984986740 1 -- 172.21.15.7:0/2910684385 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563fc7ad1730 con 0x563fc7bdc0f0 2024-03-18T01:46:32.282 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.281+0000 7f997a000640 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563fc7bdc0f0 0x563fc7bde510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:46:32.283 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.282+0000 7f997a000640 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563fc7bdc0f0 0x563fc7bde510 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=405b78bae9399e05 server_cookie=25102240f831d31b in_seq=0 out_seq=0 2024-03-18T01:46:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.284+0000 7f9968c00640 10 client.15987 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:46:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.284+0000 7f9968c00640 1 -- 172.21.15.7:0/2910684385 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x563fc7ad1730 con 0x563fc7bdc0f0 2024-03-18T01:46:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.284+0000 7f9968c00640 10 client.15987 handle_command_reply: tid=0 2024-03-18T01:46:32.284 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.284+0000 7f9984986740 1 client.15987 shutdown 2024-03-18T01:46:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.284+0000 7f9984986740 1 -- 172.21.15.7:0/2910684385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 msgr2=0x563fc7b6eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.284+0000 7f9984986740 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6eaf0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f997401fb60 tx=0x7f9974007020 comp rx=0 tx=0).stop 2024-03-18T01:46:32.285 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.285+0000 7f9984986740 1 -- 172.21.15.7:0/2910684385 shutdown_connections 2024-03-18T01:46:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.285+0000 7f9984986740 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x563fc7bdc0f0 0x563fc7bde510 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.285+0000 7f9984986740 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563fc7b65370 0x563fc7b6fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.285+0000 7f9984986740 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x563fc7b60200 0x563fc7b6eaf0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.285+0000 7f9984986740 1 --2- 172.21.15.7:0/2910684385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563fc7b5f810 0x563fc7b6e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.285+0000 7f9984986740 1 -- 172.21.15.7:0/2910684385 >> 172.21.15.7:0/2910684385 conn(0x563fc7b5ad80 msgr2=0x563fc7b59630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:32.286 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.285+0000 7f9968c00640 0 client.15987 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:46:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.286+0000 7f9984986740 1 -- 172.21.15.7:0/2910684385 shutdown_connections 2024-03-18T01:46:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.286+0000 7f9984986740 1 -- 172.21.15.7:0/2910684385 wait complete. 2024-03-18T01:46:32.287 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.286+0000 7f9984986740 20 client.15987 trim_cache size 0 max 16384 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "72a5620a-281f-4fd3-aa20-76f6ea39c1f2": { 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "72a5620a-281f-4fd3-aa20-76f6ea39c1f2", 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:46:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 -- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f99600a2840 msgr2=0x7f99600a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.289 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f99600a2840 0x7f99600a4d00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f996c009620 tx=0x7f996c005d50 comp rx=0 tx=0).stop 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 -- 172.21.15.7:0/4216100938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 msgr2=0x7f997c1adee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 0x7f997c1adee0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f996c036040 tx=0x7f996c004270 comp rx=0 tx=0).stop 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 -- 172.21.15.7:0/4216100938 shutdown_connections 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f997c10db00 0x7f997c1adee0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f997c10b150 0x7f997c1a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f99600a2840 0x7f99600a4d00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 --2- 172.21.15.7:0/4216100938 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f997c1087e0 0x7f997c1a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 -- 172.21.15.7:0/4216100938 >> 172.21.15.7:0/4216100938 conn(0x7f997c06e780 msgr2=0x7f997c10a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 -- 172.21.15.7:0/4216100938 shutdown_connections 2024-03-18T01:46:32.290 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:46:32.288+0000 7f9981c00640 1 -- 172.21.15.7:0/4216100938 wait complete. 2024-03-18T01:46:32.321 INFO:tasks.cephfs.filesystem:scrub status for tag:72a5620a-281f-4fd3-aa20-76f6ea39c1f2 - {'path': '/', 'tag': '72a5620a-281f-4fd3-aa20-76f6ea39c1f2', 'options': 'recursive,force'} 2024-03-18T01:46:33.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:32 smithi046 ceph-mon[36108]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 20 MiB/s wr, 4.38k op/s 2024-03-18T01:46:33.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:32 smithi007 ceph-mon[31996]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 20 MiB/s wr, 4.38k op/s 2024-03-18T01:46:33.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:32 smithi203 ceph-mon[36345]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 20 MiB/s wr, 4.38k op/s 2024-03-18T01:46:34.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:33 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 237.613771 secs 2024-03-18T01:46:34.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:33 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 237.613771 secs 2024-03-18T01:46:34.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:33 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 237.613771 secs 2024-03-18T01:46:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:34 smithi046 ceph-mon[36108]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 4.09k op/s 2024-03-18T01:46:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:34 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:34 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:35.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:34 smithi007 ceph-mon[31996]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 4.09k op/s 2024-03-18T01:46:35.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:34 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:35.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:34 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:34 smithi203 ceph-mon[36345]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 4.09k op/s 2024-03-18T01:46:35.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:34 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:35.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:34 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:36.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:35 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 237.159851 secs 2024-03-18T01:46:36.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:35 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 237.159851 secs 2024-03-18T01:46:36.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:35 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 237.159851 secs 2024-03-18T01:46:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:36 smithi046 ceph-mon[36108]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 15 MiB/s wr, 3.38k op/s 2024-03-18T01:46:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:36 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 329 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:37.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:36 smithi007 ceph-mon[31996]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 15 MiB/s wr, 3.38k op/s 2024-03-18T01:46:37.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:36 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 329 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:36 smithi203 ceph-mon[36345]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 15 MiB/s wr, 3.38k op/s 2024-03-18T01:46:37.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:36 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 329 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:38 smithi046 ceph-mon[36108]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 14 MiB/s wr, 3.12k op/s 2024-03-18T01:46:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:38 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:38 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:39.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:38 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 242.613834 secs 2024-03-18T01:46:39.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:38 smithi007 ceph-mon[31996]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 14 MiB/s wr, 3.12k op/s 2024-03-18T01:46:39.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:38 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:38 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:39.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:38 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 242.613834 secs 2024-03-18T01:46:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:38 smithi203 ceph-mon[36345]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 14 MiB/s wr, 3.12k op/s 2024-03-18T01:46:39.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:38 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:38 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:39.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:38 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 242.613834 secs 2024-03-18T01:46:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:40 smithi046 ceph-mon[36108]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.35k op/s 2024-03-18T01:46:41.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:40 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 242.159912 secs 2024-03-18T01:46:41.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:40 smithi007 ceph-mon[31996]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.35k op/s 2024-03-18T01:46:41.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:40 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 242.159912 secs 2024-03-18T01:46:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:40 smithi203 ceph-mon[36345]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 16 MiB/s wr, 3.35k op/s 2024-03-18T01:46:41.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:40 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 242.159912 secs 2024-03-18T01:46:42.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:41 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 334 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:42.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:41 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 334 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:42.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:41 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 334 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:43.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:42 smithi046 ceph-mon[36108]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 14 MiB/s wr, 2.98k op/s 2024-03-18T01:46:43.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:42 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:43.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:42 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:43.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:42 smithi007 ceph-mon[31996]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 14 MiB/s wr, 2.98k op/s 2024-03-18T01:46:43.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:42 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:43.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:42 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:42 smithi203 ceph-mon[36345]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 14 MiB/s wr, 2.98k op/s 2024-03-18T01:46:43.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:42 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:43.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:42 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:44.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:43 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 247.613896 secs 2024-03-18T01:46:44.051 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:43 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 247.613896 secs 2024-03-18T01:46:44.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:43 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 247.613896 secs 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout:Syncing()...61 sec 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout:FFSB benchmark finished at: Mon Mar 18 01:46:44 2024 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout:Results: 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout:Benchmark took 380.88 sec 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout:Total Results 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout:=============== 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout: ======= ============ ========= ======= =========== ========== 2024-03-18T01:46:44.895 INFO:tasks.workunit.client.0.smithi007.stdout: write : 5368320 14094.44 100.000% 100.000% 55.1MB/sec 2024-03-18T01:46:44.896 INFO:tasks.workunit.client.0.smithi007.stdout:- 2024-03-18T01:46:44.896 INFO:tasks.workunit.client.0.smithi007.stdout:14094.44 Transactions per Second 2024-03-18T01:46:44.896 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:44.896 INFO:tasks.workunit.client.0.smithi007.stdout:Throughput Results 2024-03-18T01:46:44.896 INFO:tasks.workunit.client.0.smithi007.stdout:=================== 2024-03-18T01:46:44.896 INFO:tasks.workunit.client.0.smithi007.stdout:Write Throughput: 55.1MB/sec 2024-03-18T01:46:45.005 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:45.005 INFO:tasks.workunit.client.0.smithi007.stdout:System Call Latency statistics in millisecs 2024-03-18T01:46:45.005 INFO:tasks.workunit.client.0.smithi007.stdout:===== 2024-03-18T01:46:45.005 INFO:tasks.workunit.client.0.smithi007.stdout: Min Avg Max Total Calls 2024-03-18T01:46:45.005 INFO:tasks.workunit.client.0.smithi007.stdout: ======== ======== ======== ============ 2024-03-18T01:46:45.005 INFO:tasks.workunit.client.0.smithi007.stdout:[ open] 0.019000 0.776917 106.672997 4194 2024-03-18T01:46:45.005 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[5] 100.000000 - 200.000000 : 1 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[6] 50.000000 - 100.000000 : 7 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[7] 20.000000 - 50.000000 : 16 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[8] 10.000000 - 20.000000 : 28 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[9] 5.000000 - 10.000000 : 73 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[10] 2.000000 - 5.000000 : 208 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[11] 1.000000 - 2.000000 : 231 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[12] 0.500000 - 1.000000 : 101 2024-03-18T01:46:45.006 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[13] 0.200000 - 0.500000 : 120 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[14] 0.100000 - 0.200000 : 558 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[15] 0.050000 - 0.100000 : 220 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[16] 0.020000 - 0.050000 : 2629 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[17] 0.010000 - 0.020000 : 2 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout:[ write] 0.003000 1.840022 38899.558594 5368320 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[0] 5000.000000 - 10000.000000 : 186 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[1] 2000.000000 - 5000.000000 : 238 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[2] 1000.000000 - 2000.000000 : 166 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[3] 500.000000 - 1000.000000 : 175 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[4] 200.000000 - 500.000000 : 369 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[5] 100.000000 - 200.000000 : 226 2024-03-18T01:46:45.007 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[6] 50.000000 - 100.000000 : 275 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[7] 20.000000 - 50.000000 : 1144 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[8] 10.000000 - 20.000000 : 2176 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[9] 5.000000 - 10.000000 : 2287 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[10] 2.000000 - 5.000000 : 3991 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[11] 1.000000 - 2.000000 : 2157 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[12] 0.500000 - 1.000000 : 1762 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[13] 0.200000 - 0.500000 : 3164 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[14] 0.100000 - 0.200000 : 5513 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[15] 0.050000 - 0.100000 : 12003 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[16] 0.020000 - 0.050000 : 141847 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[17] 0.010000 - 0.020000 : 2417575 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[18] 0.000000 - 0.010000 : 2772660 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout:[ lseek] 0.000000 0.002563 130.022995 5368320 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T01:46:45.008 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[5] 100.000000 - 200.000000 : 1 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[6] 50.000000 - 100.000000 : 5 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[7] 20.000000 - 50.000000 : 75 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[8] 10.000000 - 20.000000 : 185 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[9] 5.000000 - 10.000000 : 145 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[10] 2.000000 - 5.000000 : 357 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[11] 1.000000 - 2.000000 : 181 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[12] 0.500000 - 1.000000 : 116 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[13] 0.200000 - 0.500000 : 248 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[14] 0.100000 - 0.200000 : 257 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[15] 0.050000 - 0.100000 : 554 2024-03-18T01:46:45.009 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[16] 0.020000 - 0.050000 : 1907 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[17] 0.010000 - 0.020000 : 1417 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[18] 0.000000 - 0.010000 : 5362872 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout:[ close] 0.003000 0.012662 4.886000 4194 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-18T01:46:45.010 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[10] 2.000000 - 5.000000 : 3 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[12] 0.500000 - 1.000000 : 1 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[13] 0.200000 - 0.500000 : 6 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[14] 0.100000 - 0.200000 : 3 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[15] 0.050000 - 0.100000 : 8 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[16] 0.020000 - 0.050000 : 16 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[17] 0.010000 - 0.020000 : 1677 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: msec_range[18] 0.000000 - 0.010000 : 2480 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout: 2024-03-18T01:46:45.011 INFO:tasks.workunit.client.0.smithi007.stdout:1.0% User Time 2024-03-18T01:46:45.012 INFO:tasks.workunit.client.0.smithi007.stdout:15.9% System Time 2024-03-18T01:46:45.012 INFO:tasks.workunit.client.0.smithi007.stdout:16.9% CPU Utilization 2024-03-18T01:46:45.019 INFO:tasks.workunit.client.0.smithi007.stderr:+ cd .. 2024-03-18T01:46:45.020 INFO:tasks.workunit.client.0.smithi007.stderr:+ rm -r tmp ffsb 2024-03-18T01:46:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 11 MiB/s wr, 2.34k op/s 2024-03-18T01:46:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.056 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:44 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 11 MiB/s wr, 2.34k op/s 2024-03-18T01:46:45.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:45.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.144 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:46:45.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.145 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:44 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 11 MiB/s wr, 2.34k op/s 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.271 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:45.272 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:44 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:46.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:45 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:46.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:45 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 247.159975 secs 2024-03-18T01:46:46.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:45 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:46.143 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:45 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 247.159975 secs 2024-03-18T01:46:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:45 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:46:46.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:45 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 247.159975 secs 2024-03-18T01:46:47.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:46 smithi007 ceph-mon[31996]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 MiB/s wr, 2.54k op/s 2024-03-18T01:46:47.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:46 smithi007 ceph-mon[31996]: Health check update: 0 slow ops, oldest one blocked for 339 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:47.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:46 smithi203 ceph-mon[36345]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 MiB/s wr, 2.54k op/s 2024-03-18T01:46:47.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:46 smithi203 ceph-mon[36345]: Health check update: 0 slow ops, oldest one blocked for 339 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:47.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:46 smithi046 ceph-mon[36108]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 12 MiB/s wr, 2.54k op/s 2024-03-18T01:46:47.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:46 smithi046 ceph-mon[36108]: Health check update: 0 slow ops, oldest one blocked for 339 sec, osd.9 has slow ops (SLOW_OPS) 2024-03-18T01:46:49.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:48 smithi203 ceph-mon[36345]: pgmap v676: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 11 MiB/s wr, 2.24k op/s 2024-03-18T01:46:49.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:48 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 252.613919 secs 2024-03-18T01:46:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:48 smithi046 ceph-mon[36108]: pgmap v676: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 11 MiB/s wr, 2.24k op/s 2024-03-18T01:46:49.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:48 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 252.613919 secs 2024-03-18T01:46:49.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:48 smithi007 ceph-mon[31996]: pgmap v676: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 11 MiB/s wr, 2.24k op/s 2024-03-18T01:46:49.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:48 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 252.613919 secs 2024-03-18T01:46:50.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:49 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:50.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:49 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 302 secs 2024-03-18T01:46:50.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:49 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:50.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:49 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 302 secs 2024-03-18T01:46:50.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:49 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:50.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:49 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 302 secs 2024-03-18T01:46:51.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:50 smithi203 ceph-mon[36345]: pgmap v677: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 9.3 MiB/s wr, 1.93k op/s 2024-03-18T01:46:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:50 smithi203 ceph-mon[36345]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 339 sec, osd.9 has slow ops) 2024-03-18T01:46:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:50 smithi203 ceph-mon[36345]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-18T01:46:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:50 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:51.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:50 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:50 smithi046 ceph-mon[36108]: pgmap v677: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 9.3 MiB/s wr, 1.93k op/s 2024-03-18T01:46:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:50 smithi046 ceph-mon[36108]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 339 sec, osd.9 has slow ops) 2024-03-18T01:46:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:50 smithi046 ceph-mon[36108]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-18T01:46:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:50 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:51.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:50 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:50 smithi007 ceph-mon[31996]: pgmap v677: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 9.3 MiB/s wr, 1.93k op/s 2024-03-18T01:46:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:50 smithi007 ceph-mon[31996]: Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 339 sec, osd.9 has slow ops) 2024-03-18T01:46:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:50 smithi007 ceph-mon[31996]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-03-18T01:46:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:50 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:51.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:50 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:52.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:51 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:52.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:51 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:52.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:51 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:53.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:52 smithi203 ceph-mon[36345]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.8 MiB/s wr, 1.08k op/s 2024-03-18T01:46:53.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:52 smithi046 ceph-mon[36108]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.8 MiB/s wr, 1.08k op/s 2024-03-18T01:46:53.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:52 smithi007 ceph-mon[31996]: pgmap v678: 129 pgs: 129 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 4.8 MiB/s wr, 1.08k op/s 2024-03-18T01:46:54.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:53 smithi203 ceph-mon[36345]: 2 slow requests, 0 included below; oldest blocked for > 257.613982 secs 2024-03-18T01:46:54.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:53 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2024-03-18T01:46:54.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:53 smithi046 ceph-mon[36108]: 2 slow requests, 0 included below; oldest blocked for > 257.613982 secs 2024-03-18T01:46:54.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:53 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2024-03-18T01:46:54.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:53 smithi007 ceph-mon[31996]: 2 slow requests, 0 included below; oldest blocked for > 257.613982 secs 2024-03-18T01:46:54.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:53 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2024-03-18T01:46:55.141 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:54 smithi203 ceph-mon[36345]: pgmap v679: 129 pgs: 129 active+clean; 8.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 4.8 MiB/s wr, 1.13k op/s 2024-03-18T01:46:55.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:54 smithi203 ceph-mon[36345]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:46:55.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:54 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:55.142 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:54 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:54 smithi046 ceph-mon[36108]: pgmap v679: 129 pgs: 129 active+clean; 8.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 4.8 MiB/s wr, 1.13k op/s 2024-03-18T01:46:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:54 smithi046 ceph-mon[36108]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:46:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:54 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:55.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:54 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:54 smithi007 ceph-mon[31996]: pgmap v679: 129 pgs: 129 active+clean; 8.2 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 4.8 MiB/s wr, 1.13k op/s 2024-03-18T01:46:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:54 smithi007 ceph-mon[31996]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-18T01:46:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:54 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] up:active 2024-03-18T01:46:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:54 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:55 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:46:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:55 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:46:56.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:55 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:46:56.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:55 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:46:56.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:55 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:46:56.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:55 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:46:56.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:55 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:46:56.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:55 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:46:56.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:55 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:46:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:56 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:46:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:56 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:46:57.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:56 smithi046 ceph-mon[36108]: pgmap v680: 129 pgs: 129 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 4.8 MiB/s wr, 1.19k op/s 2024-03-18T01:46:57.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:56 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:46:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:56 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:46:57.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:56 smithi007 ceph-mon[31996]: pgmap v680: 129 pgs: 129 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 4.8 MiB/s wr, 1.19k op/s 2024-03-18T01:46:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:56 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:46:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:56 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:46:57.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:56 smithi203 ceph-mon[36345]: pgmap v680: 129 pgs: 129 active+clean; 5.6 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 4.8 MiB/s wr, 1.19k op/s 2024-03-18T01:46:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:58 smithi046 ceph-mon[36108]: pgmap v681: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 445 KiB/s wr, 284 op/s 2024-03-18T01:46:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:58 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:59.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:58 smithi046 ceph-mon[36108]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2024-03-18T01:46:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:58 smithi007 ceph-mon[31996]: pgmap v681: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 445 KiB/s wr, 284 op/s 2024-03-18T01:46:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:58 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:59.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:58 smithi007 ceph-mon[31996]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2024-03-18T01:46:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:58 smithi203 ceph-mon[36345]: pgmap v681: 129 pgs: 129 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 445 KiB/s wr, 284 op/s 2024-03-18T01:46:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:58 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:46:59.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:58 smithi203 ceph-mon[36345]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2024-03-18T01:47:00.211 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-18T01:47:00.212 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T01:47:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:59 smithi046 ceph-mon[36108]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-18T01:47:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:59 smithi046 ceph-mon[36108]: Cluster is now healthy 2024-03-18T01:47:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:59 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:47:00.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:46:59 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:47:00.290 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:59 smithi007 ceph-mon[31996]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-18T01:47:00.290 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:59 smithi007 ceph-mon[31996]: Cluster is now healthy 2024-03-18T01:47:00.290 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:59 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:47:00.290 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:46:59 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:47:00.317 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-03-18T01:47:00.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T01:47:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:59 smithi203 ceph-mon[36345]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-18T01:47:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:59 smithi203 ceph-mon[36345]: Cluster is now healthy 2024-03-18T01:47:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:59 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] up:active 2024-03-18T01:47:00.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:46:59 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:47:01.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:00 smithi046 ceph-mon[36108]: pgmap v682: 129 pgs: 129 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 436 KiB/s wr, 310 op/s 2024-03-18T01:47:01.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:00 smithi007 ceph-mon[31996]: pgmap v682: 129 pgs: 129 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 436 KiB/s wr, 310 op/s 2024-03-18T01:47:01.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:00 smithi203 ceph-mon[36345]: pgmap v682: 129 pgs: 129 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 436 KiB/s wr, 310 op/s 2024-03-18T01:47:02.269 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:01 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:47:02.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:01 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:47:02.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:47:02.322 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:47:02.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:01 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:47:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.545+0000 7f1476000640 1 -- 172.21.15.7:0/508683246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 msgr2=0x7f1470104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.545+0000 7f1476000640 1 --2- 172.21.15.7:0/508683246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 0x7f1470104520 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1464009900 tx=0x7f146402bc40 comp rx=0 tx=0).stop 2024-03-18T01:47:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.545+0000 7f1476000640 1 -- 172.21.15.7:0/508683246 shutdown_connections 2024-03-18T01:47:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.545+0000 7f1476000640 1 --2- 172.21.15.7:0/508683246 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1470107390 0x7f1470109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.545+0000 7f1476000640 1 --2- 172.21.15.7:0/508683246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1470104a60 0x7f1470106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.545+0000 7f1476000640 1 --2- 172.21.15.7:0/508683246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 0x7f1470104520 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.545+0000 7f1476000640 1 -- 172.21.15.7:0/508683246 >> 172.21.15.7:0/508683246 conn(0x7f14700fbac0 msgr2=0x7f14700fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.545+0000 7f1476000640 1 -- 172.21.15.7:0/508683246 shutdown_connections 2024-03-18T01:47:02.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.546+0000 7f1476000640 1 -- 172.21.15.7:0/508683246 wait complete. 2024-03-18T01:47:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1476000640 1 Processor -- start 2024-03-18T01:47:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1476000640 1 -- start start 2024-03-18T01:47:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1476000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 0x7f1470199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1476000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1470104a60 0x7f147019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1476000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1470107390 0x7f14701a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1476000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f147010d320 con 0x7f1470104a60 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1476000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f147010d1a0 con 0x7f1470107390 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1476000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f147010d4a0 con 0x7f1470102130 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f146fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1470104a60 0x7f147019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1474c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 0x7f1470199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f146fe00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1470104a60 0x7f147019a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43602/0 (socket says 172.21.15.7:43602) 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1474c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 0x7f1470199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51142/0 (socket says 172.21.15.7:51142) 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f146fe00640 1 -- 172.21.15.7:0/3125476514 learned_addr learned my addr 172.21.15.7:0/3125476514 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.547+0000 7f1475600640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1470107390 0x7f14701a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f1474c00640 1 -- 172.21.15.7:0/3125476514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1470107390 msgr2=0x7f14701a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f1474c00640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1470107390 0x7f14701a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f1474c00640 1 -- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1470104a60 msgr2=0x7f147019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.549 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f1474c00640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1470104a60 0x7f147019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f1474c00640 1 -- 172.21.15.7:0/3125476514 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14701a1790 con 0x7f1470102130 2024-03-18T01:47:02.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f1474c00640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 0x7f1470199b70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1464036040 tx=0x7f1464005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:02.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f146d600640 1 -- 172.21.15.7:0/3125476514 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464041070 con 0x7f1470102130 2024-03-18T01:47:02.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f146d600640 1 -- 172.21.15.7:0/3125476514 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14640044d0 con 0x7f1470102130 2024-03-18T01:47:02.550 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.548+0000 7f1476000640 1 -- 172.21.15.7:0/3125476514 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14701a1a20 con 0x7f1470102130 2024-03-18T01:47:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.549+0000 7f146d600640 1 -- 172.21.15.7:0/3125476514 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1464034440 con 0x7f1470102130 2024-03-18T01:47:02.551 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.549+0000 7f1476000640 1 -- 172.21.15.7:0/3125476514 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14701a1f30 con 0x7f1470102130 2024-03-18T01:47:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f147a3e9740 1 Processor -- start 2024-03-18T01:47:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f147a3e9740 1 -- start start 2024-03-18T01:47:02.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f146d600640 1 -- 172.21.15.7:0/3125476514 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f1464044070 con 0x7f1470102130 2024-03-18T01:47:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb427e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb428400 0x556ddb42cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb42fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f147a3e9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556ddb38c1f0 con 0x556ddb428400 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f147a3e9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556ddb334a80 con 0x556ddb42d570 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.553+0000 7f147a3e9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556ddb388290 con 0x556ddb427a10 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1475600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb427e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1474c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb428400 0x556ddb42cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1475600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb427e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51156/0 (socket says 172.21.15.7:51156) 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1475600640 1 -- 172.21.15.7:0/1574280410 learned_addr learned my addr 172.21.15.7:0/1574280410 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f146fe00640 1 --2- 172.21.15.7:0/1574280410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb42fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1475600640 1 -- 172.21.15.7:0/1574280410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 msgr2=0x556ddb42fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1475600640 1 --2- 172.21.15.7:0/1574280410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb42fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1475600640 1 -- 172.21.15.7:0/1574280410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb428400 msgr2=0x556ddb42cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1475600640 1 --2- 172.21.15.7:0/1574280410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb428400 0x556ddb42cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.556 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.554+0000 7f1475600640 1 -- 172.21.15.7:0/1574280410 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddb16a7d0 con 0x556ddb427a10 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f1475600640 1 --2- 172.21.15.7:0/1574280410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb427e30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1468010370 tx=0x7f1468010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6d8d6817075a264 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f145dc00640 1 -- 172.21.15.7:0/1574280410 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468016070 con 0x556ddb427a10 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f145dc00640 1 -- 172.21.15.7:0/1574280410 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1468004d20 con 0x556ddb427a10 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f145dc00640 1 -- 172.21.15.7:0/1574280410 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468005480 con 0x556ddb427a10 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f146d600640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14580a2880 0x7f14580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f146d600640 1 -- 172.21.15.7:0/3125476514 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14640f0810 con 0x7f1470102130 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f1474c00640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14580a2880 0x7f14580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f147a3e9740 1 -- 172.21.15.7:0/1574280410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 msgr2=0x556ddb427e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.557 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.555+0000 7f147a3e9740 1 --2- 172.21.15.7:0/1574280410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb427e30 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1468010370 tx=0x7f1468010830 comp rx=0 tx=0).stop 2024-03-18T01:47:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f147a3e9740 1 -- 172.21.15.7:0/1574280410 shutdown_connections 2024-03-18T01:47:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f147a3e9740 1 --2- 172.21.15.7:0/1574280410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb42fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f147a3e9740 1 --2- 172.21.15.7:0/1574280410 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb428400 0x556ddb42cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f147a3e9740 1 --2- 172.21.15.7:0/1574280410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb427e30 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f147a3e9740 1 -- 172.21.15.7:0/1574280410 >> 172.21.15.7:0/1574280410 conn(0x556ddb422f80 msgr2=0x556ddb423380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f147a3e9740 1 -- 172.21.15.7:0/1574280410 shutdown_connections 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f147a3e9740 1 -- 172.21.15.7:0/1574280410 wait complete. 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f1474c00640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14580a2880 0x7f14580a4d40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f14640ee320 tx=0x7f14640f6550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.556+0000 7f147a3e9740 1 -- start start 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f147a3e9740 1 Processor -- start 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb431f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb432490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4394d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f147a3e9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x556ddb430520 con 0x556ddb427a10 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f147a3e9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x556ddb347ed0 con 0x556ddb42d570 2024-03-18T01:47:02.561 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f147a3e9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x556ddb4306a0 con 0x556ddb428400 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f147a3e9740 10 client.0 _ll_register_callbacks cb 0x556ddb383220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f146fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb432490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f146fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb432490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51168/0 (socket says 172.21.15.7:51168) 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f146fe00640 1 -- 172.21.15.7:0/2335082438 learned_addr learned my addr 172.21.15.7:0/2335082438 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.557+0000 7f1475600640 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb431f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f1475600640 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f146fe00640 1 -- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 msgr2=0x556ddb4394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f146fe00640 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4394d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f146fe00640 1 -- 172.21.15.7:0/2335082438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 msgr2=0x556ddb431f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f146fe00640 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb431f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f146fe00640 1 -- 172.21.15.7:0/2335082438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ddb0a03f0 con 0x556ddb428400 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f1475600640 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:47:02.562 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f146fe00640 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb432490 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1460012740 tx=0x7f1460012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c1696a60d9fd2a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f1452000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f1452000640 1 -- 172.21.15.7:0/2335082438 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460004020 con 0x556ddb428400 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f1452000640 1 -- 172.21.15.7:0/2335082438 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14600041c0 con 0x556ddb428400 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.558+0000 7f147a3e9740 1 -- 172.21.15.7:0/2335082438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1468004d20 con 0x556ddb428400 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f1452000640 1 -- 172.21.15.7:0/2335082438 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f1460025050 con 0x556ddb428400 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f147a3e9740 10 client.26302 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f147a3e9740 1 -- 172.21.15.7:0/2335082438 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x556ddb0600f0 con 0x556ddb428400 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f1452000640 1 -- 172.21.15.7:0/2335082438 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1468004d20 con 0x556ddb428400 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f1452000640 1 -- 172.21.15.7:0/2335082438 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f1460003a60 con 0x556ddb428400 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f147a3e9740 10 client.26302 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f147a3e9740 10 client.26302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f147a3e9740 20 client.26302 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x556ddb43ddc0 0x556ddb440240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.563 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f147a3e9740 4 client.26302 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.559+0000 7f147a3e9740 1 -- 172.21.15.7:0/2335082438 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ddb39fbe0 con 0x556ddb43ddc0 2024-03-18T01:47:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.560+0000 7f1475600640 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x556ddb43ddc0 0x556ddb440240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.560+0000 7f1475600640 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x556ddb43ddc0 0x556ddb440240 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de55ad689c844fe7 server_cookie=5ae0aabe3c523a62 in_seq=0 out_seq=0 2024-03-18T01:47:02.564 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.560+0000 7f1452000640 10 client.26302 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f1452000640 1 -- 172.21.15.7:0/2335082438 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ddb39fbe0 con 0x556ddb43ddc0 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f1452000640 10 client.26302 handle_command_reply: tid=0 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 client.26302 shutdown 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 -- 172.21.15.7:0/2335082438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 msgr2=0x556ddb432490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb432490 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1460012740 tx=0x7f1460012c00 comp rx=0 tx=0).stop 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 -- 172.21.15.7:0/2335082438 shutdown_connections 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x556ddb43ddc0 0x556ddb440240 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.565 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb432490 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2335082438 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb431f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f1452000640 0 client.26302 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 -- 172.21.15.7:0/2335082438 >> 172.21.15.7:0/2335082438 conn(0x556ddb422f80 msgr2=0x556ddb422110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 -- 172.21.15.7:0/2335082438 shutdown_connections 2024-03-18T01:47:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.561+0000 7f147a3e9740 1 -- 172.21.15.7:0/2335082438 wait complete. 2024-03-18T01:47:02.566 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.562+0000 7f147a3e9740 20 client.26302 trim_cache size 0 max 16384 2024-03-18T01:47:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.581+0000 7f147a3e9740 1 Processor -- start 2024-03-18T01:47:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.581+0000 7f147a3e9740 1 -- start start 2024-03-18T01:47:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.581+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb429900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.581+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb428400 0x556ddb429e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.581+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb42d570 0x556ddb439490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.581+0000 7f147a3e9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f14701ac3b0 con 0x556ddb42d570 2024-03-18T01:47:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.581+0000 7f147a3e9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f14701ae010 con 0x556ddb428400 2024-03-18T01:47:02.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.581+0000 7f147a3e9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f14701acab0 con 0x556ddb427a10 2024-03-18T01:47:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb428400 0x556ddb429e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb428400 0x556ddb429e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59628/0 (socket says 172.21.15.7:59628) 2024-03-18T01:47:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 -- 172.21.15.7:0/2500082405 learned_addr learned my addr 172.21.15.7:0/2500082405 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f1475600640 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb429900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f1475600640 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb42d570 0x556ddb439490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 -- 172.21.15.7:0/2500082405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 msgr2=0x556ddb429900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb429900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 -- 172.21.15.7:0/2500082405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb42d570 msgr2=0x556ddb439490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb42d570 0x556ddb439490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 -- 172.21.15.7:0/2500082405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddb42d2c0 con 0x556ddb428400 2024-03-18T01:47:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.582+0000 7f146fe00640 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb428400 0x556ddb429e40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1460009f90 tx=0x7f1460009fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1a4b63e22e12b54 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.583+0000 7f145d200640 1 -- 172.21.15.7:0/2500082405 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146002d020 con 0x556ddb428400 2024-03-18T01:47:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.583+0000 7f145d200640 1 -- 172.21.15.7:0/2500082405 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1460012e50 con 0x556ddb428400 2024-03-18T01:47:02.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.583+0000 7f145d200640 1 -- 172.21.15.7:0/2500082405 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146001cc10 con 0x556ddb428400 2024-03-18T01:47:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.583+0000 7f147a3e9740 1 -- 172.21.15.7:0/2500082405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb428400 msgr2=0x556ddb429e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.583+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb428400 0x556ddb429e40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1460009f90 tx=0x7f1460009fc0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 -- 172.21.15.7:0/2500082405 shutdown_connections 2024-03-18T01:47:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb42d570 0x556ddb439490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb428400 0x556ddb429e40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 --2- 172.21.15.7:0/2500082405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb427a10 0x556ddb429900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 -- 172.21.15.7:0/2500082405 >> 172.21.15.7:0/2500082405 conn(0x556ddb422f80 msgr2=0x556ddb431f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 -- 172.21.15.7:0/2500082405 shutdown_connections 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 -- 172.21.15.7:0/2500082405 wait complete. 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 -- start start 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.584+0000 7f147a3e9740 1 Processor -- start 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb436b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb4370d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f147a3e9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4380e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f147a3e9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f14701ad8e0 con 0x556ddb427a10 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f147a3e9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f14701ad1b0 con 0x556ddb42d570 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f147a3e9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f14701aa390 con 0x556ddb428400 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f147a3e9740 10 client.0 _ll_register_callbacks cb 0x7f14600098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:02.587 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f146fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb4370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f1475600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb436b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f146fe00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb4370d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51180/0 (socket says 172.21.15.7:51180) 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f146fe00640 1 -- 172.21.15.7:0/3506728934 learned_addr learned my addr 172.21.15.7:0/3506728934 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.585+0000 7f1475600640 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4380e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f146fe00640 1 -- 172.21.15.7:0/3506728934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 msgr2=0x556ddb4380e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f146fe00640 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4380e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f146fe00640 1 -- 172.21.15.7:0/3506728934 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 msgr2=0x556ddb436b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f146fe00640 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb436b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f146fe00640 1 -- 172.21.15.7:0/3506728934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ddb4386d0 con 0x556ddb428400 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f1475600640 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb436b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f146fe00640 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb4370d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1460003f50 tx=0x7f146001c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b977b077dbe9361f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:02.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f1452000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f1452000640 1 -- 172.21.15.7:0/3506728934 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460036060 con 0x556ddb428400 2024-03-18T01:47:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f147a3e9740 1 -- 172.21.15.7:0/3506728934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1460012e50 con 0x556ddb428400 2024-03-18T01:47:02.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.586+0000 7f1452000640 1 -- 172.21.15.7:0/3506728934 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1460022040 con 0x556ddb428400 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f1452000640 1 -- 172.21.15.7:0/3506728934 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f146003d050 con 0x556ddb428400 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f147a3e9740 10 client.26308 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f147a3e9740 1 -- 172.21.15.7:0/3506728934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x556ddb4a4140 con 0x556ddb428400 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f1452000640 1 -- 172.21.15.7:0/3506728934 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1460012e50 con 0x556ddb428400 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f1452000640 1 -- 172.21.15.7:0/3506728934 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f146000f790 con 0x556ddb428400 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f147a3e9740 10 client.26308 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f147a3e9740 10 client.26308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f147a3e9740 20 client.26308 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f147a3e9740 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x556ddb4a4a80 0x556ddb4a6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f147a3e9740 4 client.26308 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.587+0000 7f147a3e9740 1 -- 172.21.15.7:0/3506728934 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ddb399730 con 0x556ddb4a4a80 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.588+0000 7f1475600640 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x556ddb4a4a80 0x556ddb4a6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.588+0000 7f1475600640 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x556ddb4a4a80 0x556ddb4a6ea0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8c73704bac0f5b5 server_cookie=3f707b27e589d5ef in_seq=0 out_seq=0 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.588+0000 7f1452000640 10 client.26308 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:02.591 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.588+0000 7f1452000640 1 -- 172.21.15.7:0/3506728934 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556ddb399730 con 0x556ddb4a4a80 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.588+0000 7f1452000640 10 client.26308 handle_command_reply: tid=0 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.588+0000 7f147a3e9740 1 client.26308 shutdown 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 -- 172.21.15.7:0/3506728934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 msgr2=0x556ddb4370d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb4370d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1460003f50 tx=0x7f146001c6a0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 -- 172.21.15.7:0/3506728934 shutdown_connections 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x556ddb4a4a80 0x556ddb4a6ea0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556ddb42d570 0x556ddb4380e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x556ddb428400 0x556ddb4370d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f1452000640 0 client.26308 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 --2- 172.21.15.7:0/3506728934 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x556ddb427a10 0x556ddb436b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 -- 172.21.15.7:0/3506728934 >> 172.21.15.7:0/3506728934 conn(0x556ddb422f80 msgr2=0x556ddb431f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 -- 172.21.15.7:0/3506728934 shutdown_connections 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 1 -- 172.21.15.7:0/3506728934 wait complete. 2024-03-18T01:47:02.592 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.589+0000 7f147a3e9740 20 client.26308 trim_cache size 0 max 16384 2024-03-18T01:47:02.593 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:02.593 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-03-18T01:47:02.593 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-03-18T01:47:02.593 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 -- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14580a2880 msgr2=0x7f14580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14580a2880 0x7f14580a4d40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f14640ee320 tx=0x7f14640f6550 comp rx=0 tx=0).stop 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 -- 172.21.15.7:0/3125476514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 msgr2=0x7f1470199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 0x7f1470199b70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1464036040 tx=0x7f1464005d50 comp rx=0 tx=0).stop 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 -- 172.21.15.7:0/3125476514 shutdown_connections 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1470107390 0x7f14701a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f14580a2880 0x7f14580a4d40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1470104a60 0x7f147019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.594 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 --2- 172.21.15.7:0/3125476514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1470102130 0x7f1470199b70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 -- 172.21.15.7:0/3125476514 >> 172.21.15.7:0/3125476514 conn(0x7f14700fbac0 msgr2=0x7f14701017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 -- 172.21.15.7:0/3125476514 shutdown_connections 2024-03-18T01:47:02.595 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.590+0000 7f1476000640 1 -- 172.21.15.7:0/3125476514 wait complete. 2024-03-18T01:47:02.608 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T01:47:02.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:47:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.819+0000 7f015b000640 1 -- 172.21.15.7:0/667367822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0154074240 msgr2=0x7f01540746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.819+0000 7f015b000640 1 --2- 172.21.15.7:0/667367822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0154074240 0x7f01540746c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0140009900 tx=0x7f014002bc40 comp rx=0 tx=0).stop 2024-03-18T01:47:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.820+0000 7f015b000640 1 -- 172.21.15.7:0/667367822 shutdown_connections 2024-03-18T01:47:02.820 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.820+0000 7f015b000640 1 --2- 172.21.15.7:0/667367822 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015410dbf0 0x7f015410ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.820+0000 7f015b000640 1 --2- 172.21.15.7:0/667367822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0154074240 0x7f01540746c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.820+0000 7f015b000640 1 --2- 172.21.15.7:0/667367822 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0154075820 0x7f0154073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.820+0000 7f015b000640 1 -- 172.21.15.7:0/667367822 >> 172.21.15.7:0/667367822 conn(0x7f01540fcb90 msgr2=0x7f01540fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:02.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.820+0000 7f015b000640 1 -- 172.21.15.7:0/667367822 shutdown_connections 2024-03-18T01:47:02.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.820+0000 7f015b000640 1 -- 172.21.15.7:0/667367822 wait complete. 2024-03-18T01:47:02.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.821+0000 7f015b000640 1 Processor -- start 2024-03-18T01:47:02.821 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.821+0000 7f015b000640 1 -- start start 2024-03-18T01:47:02.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f015b000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0154074240 0x7f015419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f015b000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0154075820 0x7f015419e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f015b000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015410dbf0 0x7f01541a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f015b000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f01541138c0 con 0x7f015410dbf0 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f015b000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0154113740 con 0x7f0154075820 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f015b000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0154113a40 con 0x7f0154074240 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f0159200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0154075820 0x7f015419e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f0159c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0154074240 0x7f015419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f0159200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0154075820 0x7f015419e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59652/0 (socket says 172.21.15.7:59652) 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f0159200640 1 -- 172.21.15.7:0/1355654152 learned_addr learned my addr 172.21.15.7:0/1355654152 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.822+0000 7f015a600640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015410dbf0 0x7f01541a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0159200640 1 -- 172.21.15.7:0/1355654152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0154074240 msgr2=0x7f015419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0159200640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0154074240 0x7f015419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0159200640 1 -- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015410dbf0 msgr2=0x7f01541a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0159200640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015410dbf0 0x7f01541a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0159200640 1 -- 172.21.15.7:0/1355654152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01541a5a20 con 0x7f0154075820 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f015a600640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015410dbf0 0x7f01541a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0159200640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0154075820 0x7f015419e340 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f014002c920 tx=0x7f014002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0152000640 1 -- 172.21.15.7:0/1355654152 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140041070 con 0x7f0154075820 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01541a5cb0 con 0x7f0154075820 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0152000640 1 -- 172.21.15.7:0/1355654152 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0140004450 con 0x7f0154075820 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f0152000640 1 -- 172.21.15.7:0/1355654152 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0140003d90 con 0x7f0154075820 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.823+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01541a61a0 con 0x7f0154075820 2024-03-18T01:47:02.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.825+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01340052c0 con 0x7f0154075820 2024-03-18T01:47:02.827 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.827+0000 7f0152000640 1 -- 172.21.15.7:0/1355654152 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f0140044070 con 0x7f0154075820 2024-03-18T01:47:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.828+0000 7f0152000640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f013c0a2920 0x7f013c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:02.828 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.828+0000 7f0152000640 1 -- 172.21.15.7:0/1355654152 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f01400f0820 con 0x7f0154075820 2024-03-18T01:47:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.828+0000 7f0159c00640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f013c0a2920 0x7f013c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:02.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.829+0000 7f0159c00640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f013c0a2920 0x7f013c0a4de0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0148009bd0 tx=0x7f0148005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:02.832 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:02.832+0000 7f0152000640 1 -- 172.21.15.7:0/1355654152 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f01400bd500 con 0x7f0154075820 2024-03-18T01:47:03.119 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.119+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01340055b0 con 0x7f0154075820 2024-03-18T01:47:03.120 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.120+0000 7f0152000640 1 -- 172.21.15.7:0/1355654152 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+11939 (secure 0 0 0) 0x7f01400bce00 con 0x7f0154075820 2024-03-18T01:47:03.120 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:47:03.120 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":150,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:47:00.268898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":164,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":160,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":188,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":160,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":158,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":157,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":182,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24596,24562,24542,24523,24544]},"id":1}]} 2024-03-18T01:47:03.121 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 150 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f013c0a2920 msgr2=0x7f013c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f013c0a2920 0x7f013c0a4de0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0148009bd0 tx=0x7f0148005f90 comp rx=0 tx=0).stop 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0154075820 msgr2=0x7f015419e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0154075820 0x7f015419e340 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f014002c920 tx=0x7f014002c660 comp rx=0 tx=0).stop 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 shutdown_connections 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f013c0a2920 0x7f013c0a4de0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f015410dbf0 0x7f01541a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0154075820 0x7f015419e340 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 --2- 172.21.15.7:0/1355654152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0154074240 0x7f015419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.123+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 >> 172.21.15.7:0/1355654152 conn(0x7f01540fcb90 msgr2=0x7f0154100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.124+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 shutdown_connections 2024-03-18T01:47:03.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.124+0000 7f015b000640 1 -- 172.21.15.7:0/1355654152 wait complete. 2024-03-18T01:47:03.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T01:47:03.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:03 smithi007 ceph-mon[31996]: pgmap v683: 129 pgs: 129 active+clean; 892 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 80 KiB/s wr, 281 op/s 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 -- 172.21.15.7:0/3842330017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a281094e0 msgr2=0x7f0a2810b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 --2- 172.21.15.7:0/3842330017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a281094e0 0x7f0a2810b970 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0a20009be0 tx=0x7f0a2002bef0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 -- 172.21.15.7:0/3842330017 shutdown_connections 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 --2- 172.21.15.7:0/3842330017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a281094e0 0x7f0a2810b970 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 --2- 172.21.15.7:0/3842330017 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a28074240 0x7f0a280746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 --2- 172.21.15.7:0/3842330017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a28075820 0x7f0a28073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 -- 172.21.15.7:0/3842330017 >> 172.21.15.7:0/3842330017 conn(0x7f0a280fcb90 msgr2=0x7f0a280fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 -- 172.21.15.7:0/3842330017 shutdown_connections 2024-03-18T01:47:03.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.364+0000 7f0a2d400640 1 -- 172.21.15.7:0/3842330017 wait complete. 2024-03-18T01:47:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.365+0000 7f0a2d400640 1 Processor -- start 2024-03-18T01:47:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a2d400640 1 -- start start 2024-03-18T01:47:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a2d400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a28074240 0x7f0a28199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a2d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a28075820 0x7f0a28199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a2d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281094e0 0x7f0a281a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a2d400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2810f4b0 con 0x7f0a28074240 2024-03-18T01:47:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a2d400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2810f330 con 0x7f0a281094e0 2024-03-18T01:47:03.367 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a2d400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2810f630 con 0x7f0a28075820 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a28074240 0x7f0a28199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a27400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a28075820 0x7f0a28199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a2ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281094e0 0x7f0a281a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a27400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a28075820 0x7f0a28199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51226/0 (socket says 172.21.15.7:51226) 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a28074240 0x7f0a28199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43664/0 (socket says 172.21.15.7:43664) 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.366+0000 7f0a27400640 1 -- 172.21.15.7:0/971760429 learned_addr learned my addr 172.21.15.7:0/971760429 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a27400640 1 -- 172.21.15.7:0/971760429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281094e0 msgr2=0x7f0a281a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a27400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281094e0 0x7f0a281a0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a27400640 1 -- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a28074240 msgr2=0x7f0a28199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a27400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a28074240 0x7f0a28199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.368 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a27400640 1 -- 172.21.15.7:0/971760429 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a281a1650 con 0x7f0a28075820 2024-03-18T01:47:03.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a27400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a28075820 0x7f0a28199f70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400e9f0 tx=0x7f0a1400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a24c00640 1 -- 172.21.15.7:0/971760429 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400ccc0 con 0x7f0a28075820 2024-03-18T01:47:03.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a24c00640 1 -- 172.21.15.7:0/971760429 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a14010910 con 0x7f0a28075820 2024-03-18T01:47:03.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a2d400640 1 -- 172.21.15.7:0/971760429 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a281a1940 con 0x7f0a28075820 2024-03-18T01:47:03.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.367+0000 7f0a24c00640 1 -- 172.21.15.7:0/971760429 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a140188a0 con 0x7f0a28075820 2024-03-18T01:47:03.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.368+0000 7f0a2d400640 1 -- 172.21.15.7:0/971760429 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a281a1ee0 con 0x7f0a28075820 2024-03-18T01:47:03.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.371+0000 7f0a24c00640 1 -- 172.21.15.7:0/971760429 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f0a14010ac0 con 0x7f0a28075820 2024-03-18T01:47:03.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.372+0000 7f0a317e5740 1 Processor -- start 2024-03-18T01:47:03.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a317e5740 1 -- start start 2024-03-18T01:47:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17a6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17abf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a24c00640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a0c0a2870 0x7f0a0c0a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17aea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a317e5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5628e176e3b0 con 0x5628e17a6a90 2024-03-18T01:47:03.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a317e5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5628e170b450 con 0x5628e17ac5f0 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a317e5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5628e16b3cb0 con 0x5628e17a7480 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a24c00640 1 -- 172.21.15.7:0/971760429 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0a140cf1c0 con 0x7f0a28075820 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a27400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17abf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a27e00640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a0c0a2870 0x7f0a0c0a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a27400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17abf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51240/0 (socket says 172.21.15.7:51240) 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a2ca00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17a6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.373+0000 7f0a27400640 1 -- 172.21.15.7:0/575847353 learned_addr learned my addr 172.21.15.7:0/575847353 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a2ca00640 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a27400640 1 -- 172.21.15.7:0/575847353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 msgr2=0x5628e17aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a27400640 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17aea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a27400640 1 -- 172.21.15.7:0/575847353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 msgr2=0x5628e17a6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a27400640 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17a6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a27400640 1 -- 172.21.15.7:0/575847353 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628e16d51f0 con 0x5628e17a7480 2024-03-18T01:47:03.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a2ca00640 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:47:03.379 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a27400640 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17abf80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0a140a3a10 tx=0x7f0a1400cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94043d4274952018 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a27e00640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a0c0a2870 0x7f0a0c0a4d30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0a18005fd0 tx=0x7f0a18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a1d200640 1 -- 172.21.15.7:0/575847353 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1409b7a0 con 0x5628e17a7480 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a1d200640 1 -- 172.21.15.7:0/575847353 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a1409f8f0 con 0x5628e17a7480 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.374+0000 7f0a1d200640 1 -- 172.21.15.7:0/575847353 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a140974f0 con 0x5628e17a7480 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 -- 172.21.15.7:0/575847353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 msgr2=0x5628e17abf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17abf80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0a140a3a10 tx=0x7f0a1400cef0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 -- 172.21.15.7:0/575847353 shutdown_connections 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17abf80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 --2- 172.21.15.7:0/575847353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17a6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 -- 172.21.15.7:0/575847353 >> 172.21.15.7:0/575847353 conn(0x5628e17a2000 msgr2=0x5628e17a2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 -- 172.21.15.7:0/575847353 shutdown_connections 2024-03-18T01:47:03.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.375+0000 7f0a317e5740 1 -- 172.21.15.7:0/575847353 wait complete. 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.376+0000 7f0a317e5740 1 -- start start 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.376+0000 7f0a317e5740 1 Processor -- start 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17ac5f0 0x5628e17b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a317e5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5628e16c7110 con 0x5628e17a6a90 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a317e5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5628e17074f0 con 0x5628e17a7480 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a317e5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5628e17af4e0 con 0x5628e17ac5f0 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a317e5740 10 client.0 _ll_register_callbacks cb 0x5628e16b4b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b11d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59686/0 (socket says 172.21.15.7:59686) 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27400640 1 -- 172.21.15.7:0/2502011945 learned_addr learned my addr 172.21.15.7:0/2502011945 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a2ca00640 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17ac5f0 0x5628e17b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27400640 1 -- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17ac5f0 msgr2=0x5628e17b81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27400640 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17ac5f0 0x5628e17b81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27400640 1 -- 172.21.15.7:0/2502011945 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 msgr2=0x5628e17b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27400640 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a27400640 1 -- 172.21.15.7:0/2502011945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628e141f3f0 con 0x5628e17a7480 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.377+0000 7f0a2ca00640 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17ac5f0 0x5628e17b81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a27400640 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b11d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0a14097c00 tx=0x7f0a140a4010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba75f5bdc2dc0f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a11600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a11600640 1 -- 172.21.15.7:0/2502011945 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a140b9060 con 0x5628e17a7480 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a11600640 1 -- 172.21.15.7:0/2502011945 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0a140afb80 con 0x5628e17a7480 2024-03-18T01:47:03.382 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a317e5740 1 -- 172.21.15.7:0/2502011945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a1409f8f0 con 0x5628e17a7480 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a11600640 1 -- 172.21.15.7:0/2502011945 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f0a14081050 con 0x5628e17a7480 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a317e5740 10 client.26345 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a317e5740 1 -- 172.21.15.7:0/2502011945 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x5628e14e97d0 con 0x5628e17a7480 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.378+0000 7f0a11600640 1 -- 172.21.15.7:0/2502011945 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a1409f8f0 con 0x5628e17a7480 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.379+0000 7f0a11600640 1 -- 172.21.15.7:0/2502011945 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f0a1409a660 con 0x5628e17a7480 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.379+0000 7f0a317e5740 10 client.26345 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.379+0000 7f0a317e5740 10 client.26345 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.379+0000 7f0a317e5740 20 client.26345 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.379+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5628e17bc650 0x5628e17bea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.379+0000 7f0a317e5740 4 client.26345 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.379+0000 7f0a317e5740 1 -- 172.21.15.7:0/2502011945 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628e1718990 con 0x5628e17bc650 2024-03-18T01:47:03.383 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.379+0000 7f0a27e00640 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5628e17bc650 0x5628e17bea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.384 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.380+0000 7f0a27e00640 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5628e17bc650 0x5628e17bea40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8aa570e4bc99e94f server_cookie=ce5abb4f941737a3 in_seq=0 out_seq=0 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.380+0000 7f0a11600640 10 client.26345 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a11600640 1 -- 172.21.15.7:0/2502011945 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628e1718990 con 0x5628e17bc650 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a11600640 10 client.26345 handle_command_reply: tid=0 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 client.26345 shutdown 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 -- 172.21.15.7:0/2502011945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 msgr2=0x5628e17b11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b11d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0a14097c00 tx=0x7f0a140a4010 comp rx=0 tx=0).stop 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 -- 172.21.15.7:0/2502011945 shutdown_connections 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5628e17bc650 0x5628e17bea40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17ac5f0 0x5628e17b81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b11d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2502011945 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 -- 172.21.15.7:0/2502011945 >> 172.21.15.7:0/2502011945 conn(0x5628e17a2000 msgr2=0x5628e17a1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a11600640 0 client.26345 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 -- 172.21.15.7:0/2502011945 shutdown_connections 2024-03-18T01:47:03.385 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 1 -- 172.21.15.7:0/2502011945 wait complete. 2024-03-18T01:47:03.386 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.381+0000 7f0a317e5740 20 client.26345 trim_cache size 0 max 16384 2024-03-18T01:47:03.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:03 smithi203 ceph-mon[36345]: pgmap v683: 129 pgs: 129 active+clean; 892 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 80 KiB/s wr, 281 op/s 2024-03-18T01:47:03.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.396+0000 7f0a317e5740 1 Processor -- start 2024-03-18T01:47:03.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.396+0000 7f0a317e5740 1 -- start start 2024-03-18T01:47:03.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a6a90 0x5628e17a0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17ac5f0 0x5628e17a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a317e5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281ac250 con 0x5628e17ac5f0 2024-03-18T01:47:03.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a317e5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281adeb0 con 0x5628e17a7480 2024-03-18T01:47:03.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a317e5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281ac950 con 0x5628e17a6a90 2024-03-18T01:47:03.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a6a90 0x5628e17a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a2ca00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17ac5f0 0x5628e17a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.398 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a6a90 0x5628e17a0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51260/0 (socket says 172.21.15.7:51260) 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27e00640 1 -- 172.21.15.7:0/1177485688 learned_addr learned my addr 172.21.15.7:0/1177485688 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27e00640 1 -- 172.21.15.7:0/1177485688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 msgr2=0x5628e17b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27e00640 1 --2- 172.21.15.7:0/1177485688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27e00640 1 -- 172.21.15.7:0/1177485688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17ac5f0 msgr2=0x5628e17a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27e00640 1 --2- 172.21.15.7:0/1177485688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17ac5f0 0x5628e17a0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27e00640 1 -- 172.21.15.7:0/1177485688 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628e17a7860 con 0x5628e17a6a90 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.397+0000 7f0a27400640 1 --2- 172.21.15.7:0/1177485688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.398+0000 7f0a27e00640 1 --2- 172.21.15.7:0/1177485688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a6a90 0x5628e17a0990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x5628e17a2f70 tx=0x7f0a18014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eec56cd40bc5b856 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.398+0000 7f0a13e00640 1 -- 172.21.15.7:0/1177485688 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1802d070 con 0x5628e17a6a90 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.398+0000 7f0a13e00640 1 -- 172.21.15.7:0/1177485688 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a180277b0 con 0x5628e17a6a90 2024-03-18T01:47:03.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.398+0000 7f0a13e00640 1 -- 172.21.15.7:0/1177485688 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1801e710 con 0x5628e17a6a90 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.398+0000 7f0a317e5740 1 -- 172.21.15.7:0/1177485688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a6a90 msgr2=0x5628e17a0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.398+0000 7f0a317e5740 1 --2- 172.21.15.7:0/1177485688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a6a90 0x5628e17a0990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x5628e17a2f70 tx=0x7f0a18014000 comp rx=0 tx=0).stop 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 -- 172.21.15.7:0/1177485688 shutdown_connections 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 --2- 172.21.15.7:0/1177485688 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17ac5f0 0x5628e17a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 --2- 172.21.15.7:0/1177485688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17a7480 0x5628e17b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 --2- 172.21.15.7:0/1177485688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a6a90 0x5628e17a0990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 -- 172.21.15.7:0/1177485688 >> 172.21.15.7:0/1177485688 conn(0x5628e17a2000 msgr2=0x5628e17a23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 -- 172.21.15.7:0/1177485688 shutdown_connections 2024-03-18T01:47:03.404 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 -- 172.21.15.7:0/1177485688 wait complete. 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 -- start start 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.399+0000 7f0a317e5740 1 Processor -- start 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a317e5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17b74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a317e5740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281ad780 con 0x5628e17a6a90 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a317e5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281ad050 con 0x5628e17ac5f0 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a317e5740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a281aa230 con 0x5628e17a7480 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a317e5740 10 client.0 _ll_register_callbacks cb 0x7f0a1409b570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43686/0 (socket says 172.21.15.7:43686) 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27e00640 1 -- 172.21.15.7:0/2421606585 learned_addr learned my addr 172.21.15.7:0/2421606585 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.405 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a2ca00640 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17b74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27e00640 1 -- 172.21.15.7:0/2421606585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 msgr2=0x5628e17b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27e00640 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17b6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27e00640 1 -- 172.21.15.7:0/2421606585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 msgr2=0x5628e17b74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27e00640 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17b74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27e00640 1 -- 172.21.15.7:0/2421606585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628e17b7ab0 con 0x5628e17a6a90 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.400+0000 7f0a27400640 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.401+0000 7f0a27e00640 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b5fc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009310 tx=0x7f0a18004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c18a397bad67069 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.401+0000 7f0a11600640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.401+0000 7f0a11600640 1 -- 172.21.15.7:0/2421606585 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18036060 con 0x5628e17a6a90 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.401+0000 7f0a11600640 1 -- 172.21.15.7:0/2421606585 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0a180272d0 con 0x5628e17a6a90 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.401+0000 7f0a317e5740 1 -- 172.21.15.7:0/2421606585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a180277b0 con 0x5628e17a6a90 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.401+0000 7f0a11600640 1 -- 172.21.15.7:0/2421606585 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f0a1802d070 con 0x5628e17a6a90 2024-03-18T01:47:03.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.401+0000 7f0a317e5740 10 client.16029 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.401+0000 7f0a317e5740 1 -- 172.21.15.7:0/2421606585 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x5628e1822ee0 con 0x5628e17a6a90 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a11600640 1 -- 172.21.15.7:0/2421606585 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a180277b0 con 0x5628e17a6a90 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a11600640 1 -- 172.21.15.7:0/2421606585 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f0a18004950 con 0x5628e17a6a90 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a317e5740 10 client.16029 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a317e5740 10 client.16029 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a317e5740 20 client.16029 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5628e18238c0 0x5628e1825ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a317e5740 4 client.16029 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a317e5740 1 -- 172.21.15.7:0/2421606585 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628e1712ce0 con 0x5628e18238c0 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.402+0000 7f0a27400640 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5628e18238c0 0x5628e1825ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a27400640 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5628e18238c0 0x5628e1825ce0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=be56e2557af5532 server_cookie=5619397dcbb77756 in_seq=0 out_seq=0 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a11600640 10 client.16029 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a11600640 1 -- 172.21.15.7:0/2421606585 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628e1712ce0 con 0x5628e18238c0 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a11600640 10 client.16029 handle_command_reply: tid=0 2024-03-18T01:47:03.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 client.16029 shutdown 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 -- 172.21.15.7:0/2421606585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 msgr2=0x5628e17b5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b5fc0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009310 tx=0x7f0a18004280 comp rx=0 tx=0).stop 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 -- 172.21.15.7:0/2421606585 shutdown_connections 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x5628e18238c0 0x5628e1825ce0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5628e17ac5f0 0x5628e17b74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5628e17a7480 0x5628e17b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 --2- 172.21.15.7:0/2421606585 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5628e17a6a90 0x5628e17b5fc0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 -- 172.21.15.7:0/2421606585 >> 172.21.15.7:0/2421606585 conn(0x5628e17a2000 msgr2=0x5628e17a5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a11600640 0 client.16029 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 -- 172.21.15.7:0/2421606585 shutdown_connections 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 1 -- 172.21.15.7:0/2421606585 wait complete. 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.403+0000 7f0a317e5740 20 client.16029 trim_cache size 0 max 16384 2024-03-18T01:47:03.408 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 -- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a0c0a2870 msgr2=0x7f0a0c0a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a0c0a2870 0x7f0a0c0a4d30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0a18005fd0 tx=0x7f0a18009260 comp rx=0 tx=0).stop 2024-03-18T01:47:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 -- 172.21.15.7:0/971760429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a28075820 msgr2=0x7f0a28199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a28075820 0x7f0a28199f70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400e9f0 tx=0x7f0a1400edf0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 -- 172.21.15.7:0/971760429 shutdown_connections 2024-03-18T01:47:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a281094e0 0x7f0a281a0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a28075820 0x7f0a28199f70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0a0c0a2870 0x7f0a0c0a4d30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 --2- 172.21.15.7:0/971760429 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0a28074240 0x7f0a28199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 -- 172.21.15.7:0/971760429 >> 172.21.15.7:0/971760429 conn(0x7f0a280fcb90 msgr2=0x7f0a281053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.404+0000 7f0a2d400640 1 -- 172.21.15.7:0/971760429 shutdown_connections 2024-03-18T01:47:03.410 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.405+0000 7f0a2d400640 1 -- 172.21.15.7:0/971760429 wait complete. 2024-03-18T01:47:03.424 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T01:47:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:03 smithi046 ceph-mon[36108]: pgmap v683: 129 pgs: 129 active+clean; 892 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 80 KiB/s wr, 281 op/s 2024-03-18T01:47:03.630 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 -- 172.21.15.7:0/2726684091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da41096a0 msgr2=0x7f8da410bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 --2- 172.21.15.7:0/2726684091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da41096a0 0x7f8da410bb30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c009ca0 tx=0x7f8d9c02c100 comp rx=0 tx=0).stop 2024-03-18T01:47:03.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 -- 172.21.15.7:0/2726684091 shutdown_connections 2024-03-18T01:47:03.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 --2- 172.21.15.7:0/2726684091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da41096a0 0x7f8da410bb30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 --2- 172.21.15.7:0/2726684091 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da4074240 0x7f8da40746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 --2- 172.21.15.7:0/2726684091 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8da4075820 0x7f8da4073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 -- 172.21.15.7:0/2726684091 >> 172.21.15.7:0/2726684091 conn(0x7f8da40fcd10 msgr2=0x7f8da40ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 -- 172.21.15.7:0/2726684091 shutdown_connections 2024-03-18T01:47:03.631 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.630+0000 7f8da9c00640 1 -- 172.21.15.7:0/2726684091 wait complete. 2024-03-18T01:47:03.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.631+0000 7f8da9c00640 1 Processor -- start 2024-03-18T01:47:03.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.631+0000 7f8da9c00640 1 -- start start 2024-03-18T01:47:03.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da9c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da4074240 0x7f8da4199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da9c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da4075820 0x7f8da419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da9c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8da41096a0 0x7f8da41a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da9c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8da410f620 con 0x7f8da41096a0 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da9c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8da410f4a0 con 0x7f8da4075820 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da9c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8da410f7a0 con 0x7f8da4074240 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da4074240 0x7f8da4199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da4074240 0x7f8da4199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51284/0 (socket says 172.21.15.7:51284) 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2a00640 1 -- 172.21.15.7:0/1798110257 learned_addr learned my addr 172.21.15.7:0/1798110257 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2000640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da4075820 0x7f8da419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da3400640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8da41096a0 0x7f8da41a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2a00640 1 -- 172.21.15.7:0/1798110257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da4075820 msgr2=0x7f8da419a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2a00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da4075820 0x7f8da419a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2a00640 1 -- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8da41096a0 msgr2=0x7f8da41a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2a00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8da41096a0 0x7f8da41a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2a00640 1 -- 172.21.15.7:0/1798110257 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da41a1860 con 0x7f8da4074240 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.632+0000 7f8da2000640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da4075820 0x7f8da419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.633+0000 7f8da2a00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da4074240 0x7f8da4199c50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400ea60 tx=0x7f8d9400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.633+0000 7f8d9b400640 1 -- 172.21.15.7:0/1798110257 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9400cde0 con 0x7f8da4074240 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.633+0000 7f8d9b400640 1 -- 172.21.15.7:0/1798110257 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d940035f0 con 0x7f8da4074240 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.633+0000 7f8d9b400640 1 -- 172.21.15.7:0/1798110257 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94010670 con 0x7f8da4074240 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.633+0000 7f8da9c00640 1 -- 172.21.15.7:0/1798110257 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da41a1b50 con 0x7f8da4074240 2024-03-18T01:47:03.635 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.633+0000 7f8da9c00640 1 -- 172.21.15.7:0/1798110257 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da41a20f0 con 0x7f8da4074240 2024-03-18T01:47:03.636 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.636+0000 7f8dac9ff740 1 Processor -- start 2024-03-18T01:47:03.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.636+0000 7f8dac9ff740 1 -- start start 2024-03-18T01:47:03.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b17bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.637 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 0x555d3b180db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8d9b400640 1 -- 172.21.15.7:0/1798110257 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f8d940037a0 con 0x7f8da4074240 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b1838b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555d3b1433b0 con 0x555d3b17b8c0 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555d3b0e0450 con 0x555d3b17c2b0 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555d3b088cb0 con 0x555d3b181420 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8da3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b17bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8da2000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b1838b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8da3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b17bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43734/0 (socket says 172.21.15.7:43734) 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8da3400640 1 -- 172.21.15.7:0/860432597 learned_addr learned my addr 172.21.15.7:0/860432597 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.638 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.637+0000 7f8da2a00640 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 0x555d3b180db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8da2000640 1 -- 172.21.15.7:0/860432597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 msgr2=0x555d3b180db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.640 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8da2000640 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 0x555d3b180db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8da2000640 1 -- 172.21.15.7:0/860432597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 msgr2=0x555d3b17bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8da2000640 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b17bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8da2000640 1 -- 172.21.15.7:0/860432597 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d3b0aa1f0 con 0x555d3b181420 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8da3400640 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b17bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8da2000640 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b1838b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009fd0 tx=0x7f8d900118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=307e6544325bc6b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8d8fe00640 1 -- 172.21.15.7:0/860432597 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90004020 con 0x555d3b181420 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8d8fe00640 1 -- 172.21.15.7:0/860432597 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d90005500 con 0x555d3b181420 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8d8fe00640 1 -- 172.21.15.7:0/860432597 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90004d40 con 0x555d3b181420 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.638+0000 7f8d9b400640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d880a2880 0x7f8d880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8d9b400640 1 -- 172.21.15.7:0/1798110257 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8d940cf110 con 0x7f8da4074240 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8da3400640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d880a2880 0x7f8d880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8dac9ff740 1 -- 172.21.15.7:0/860432597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 msgr2=0x555d3b1838b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b1838b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009fd0 tx=0x7f8d900118d0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8dac9ff740 1 -- 172.21.15.7:0/860432597 shutdown_connections 2024-03-18T01:47:03.641 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b1838b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 0x555d3b180db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/860432597 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b17bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.639+0000 7f8dac9ff740 1 -- 172.21.15.7:0/860432597 >> 172.21.15.7:0/860432597 conn(0x555d3b176e30 msgr2=0x555d3b177230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.640+0000 7f8da3400640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d880a2880 0x7f8d880a4d40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c006010 tx=0x7f8d9c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.640+0000 7f8dac9ff740 1 -- 172.21.15.7:0/860432597 shutdown_connections 2024-03-18T01:47:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.640+0000 7f8dac9ff740 1 -- 172.21.15.7:0/860432597 wait complete. 2024-03-18T01:47:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.640+0000 7f8dac9ff740 1 -- start start 2024-03-18T01:47:03.642 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.640+0000 7f8dac9ff740 1 Processor -- start 2024-03-18T01:47:03.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b185b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 0x555d3b186050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b18d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x555d3b09c110 con 0x555d3b17b8c0 2024-03-18T01:47:03.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555d3b0dc4f0 con 0x555d3b17c2b0 2024-03-18T01:47:03.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x555d3b1843d0 con 0x555d3b181420 2024-03-18T01:47:03.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8dac9ff740 10 client.0 _ll_register_callbacks cb 0x555d3b089b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8da2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b185b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8da3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b18d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8da2a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b185b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43738/0 (socket says 172.21.15.7:43738) 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8da2a00640 1 -- 172.21.15.7:0/2420731539 learned_addr learned my addr 172.21.15.7:0/2420731539 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.641+0000 7f8da2000640 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 0x555d3b186050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8da3400640 1 -- 172.21.15.7:0/2420731539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 msgr2=0x555d3b186050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8da3400640 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 0x555d3b186050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8da3400640 1 -- 172.21.15.7:0/2420731539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 msgr2=0x555d3b185b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8da3400640 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b185b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8da3400640 1 -- 172.21.15.7:0/2420731539 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d3adf43f0 con 0x555d3b181420 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8da2a00640 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b185b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8da3400640 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b18d090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c032420 tx=0x7f8d9c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=502f5ef0eb481d83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8d8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8d8cc00640 1 -- 172.21.15.7:0/2420731539 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c050020 con 0x555d3b181420 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.642+0000 7f8d8cc00640 1 -- 172.21.15.7:0/2420731539 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8d9c003f80 con 0x555d3b181420 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2420731539 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d90005500 con 0x555d3b181420 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8d8cc00640 1 -- 172.21.15.7:0/2420731539 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f8d9c039040 con 0x555d3b181420 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 10 client.26344 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2420731539 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x555d3aebe7d0 con 0x555d3b181420 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8d8cc00640 1 -- 172.21.15.7:0/2420731539 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d90005500 con 0x555d3b181420 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8d8cc00640 1 -- 172.21.15.7:0/2420731539 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f8d9c04c930 con 0x555d3b181420 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 10 client.26344 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 10 client.26344 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 20 client.26344 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x555d3b1919b0 0x555d3b193e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 4 client.26344 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.643+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2420731539 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d3b0ed990 con 0x555d3b1919b0 2024-03-18T01:47:03.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.644+0000 7f8da2a00640 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x555d3b1919b0 0x555d3b193e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.644+0000 7f8da2a00640 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x555d3b1919b0 0x555d3b193e50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=64639d1ffe347a78 server_cookie=50b968fdb6536b2d in_seq=0 out_seq=0 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.644+0000 7f8d8cc00640 10 client.26344 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8d8cc00640 1 -- 172.21.15.7:0/2420731539 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d3b0ed990 con 0x555d3b1919b0 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8d8cc00640 10 client.26344 handle_command_reply: tid=0 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 client.26344 shutdown 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2420731539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 msgr2=0x555d3b18d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b18d090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c032420 tx=0x7f8d9c0095c0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2420731539 shutdown_connections 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x555d3b1919b0 0x555d3b193e50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b181420 0x555d3b18d090 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b17c2b0 0x555d3b186050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2420731539 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17b8c0 0x555d3b185b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2420731539 >> 172.21.15.7:0/2420731539 conn(0x555d3b176e30 msgr2=0x555d3b1761f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8d8cc00640 0 client.26344 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:47:03.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2420731539 shutdown_connections 2024-03-18T01:47:03.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2420731539 wait complete. 2024-03-18T01:47:03.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.645+0000 7f8dac9ff740 20 client.26344 trim_cache size 0 max 16384 2024-03-18T01:47:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8dac9ff740 1 Processor -- start 2024-03-18T01:47:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8dac9ff740 1 -- start start 2024-03-18T01:47:03.663 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b177210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b17d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41ac510 con 0x555d3b17c2b0 2024-03-18T01:47:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41ae170 con 0x555d3b181420 2024-03-18T01:47:03.664 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41acc10 con 0x555d3b17b8c0 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8da2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b177210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8da2000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8da3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b17d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8da2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b177210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51322/0 (socket says 172.21.15.7:51322) 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.663+0000 7f8da2a00640 1 -- 172.21.15.7:0/2447473833 learned_addr learned my addr 172.21.15.7:0/2447473833 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8da2a00640 1 -- 172.21.15.7:0/2447473833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 msgr2=0x555d3b17d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8da2a00640 1 --2- 172.21.15.7:0/2447473833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b17d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8da2a00640 1 -- 172.21.15.7:0/2447473833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 msgr2=0x555d3b18d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8da2a00640 1 --2- 172.21.15.7:0/2447473833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8da2a00640 1 -- 172.21.15.7:0/2447473833 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d3b172c10 con 0x555d3b17b8c0 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8da2a00640 1 --2- 172.21.15.7:0/2447473833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b177210 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400d010 tx=0x7f8d940cc720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c6c4e910385447f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.666 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8d8f400640 1 -- 172.21.15.7:0/2447473833 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9409b230 con 0x555d3b17b8c0 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8d8f400640 1 -- 172.21.15.7:0/2447473833 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d9409bdf0 con 0x555d3b17b8c0 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.664+0000 7f8d8f400640 1 -- 172.21.15.7:0/2447473833 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d940860b0 con 0x555d3b17b8c0 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2447473833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 msgr2=0x555d3b177210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2447473833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b177210 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400d010 tx=0x7f8d940cc720 comp rx=0 tx=0).stop 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2447473833 shutdown_connections 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2447473833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b17d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2447473833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/2447473833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b177210 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2447473833 >> 172.21.15.7:0/2447473833 conn(0x555d3b176e30 msgr2=0x555d3b172080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2447473833 shutdown_connections 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.665+0000 7f8dac9ff740 1 -- 172.21.15.7:0/2447473833 wait complete. 2024-03-18T01:47:03.667 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 1 -- start start 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 1 Processor -- start 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b18a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b18b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41ada40 con 0x555d3b17c2b0 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41ad310 con 0x555d3b181420 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8da41aa4f0 con 0x555d3b17b8c0 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.666+0000 7f8dac9ff740 10 client.0 _ll_register_callbacks cb 0x7f8d9c044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:03.671 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b18a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b18b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b18a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51326/0 (socket says 172.21.15.7:51326) 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43754/0 (socket says 172.21.15.7:43754) 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 -- 172.21.15.7:0/1405472739 learned_addr learned my addr 172.21.15.7:0/1405472739 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b18b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59764/0 (socket says 172.21.15.7:59764) 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 -- 172.21.15.7:0/1405472739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 msgr2=0x555d3b18b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b18b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 -- 172.21.15.7:0/1405472739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 msgr2=0x555d3b18a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 -- 172.21.15.7:0/1405472739 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d3b18bdf0 con 0x555d3b17b8c0 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2000640 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da3400640 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b18b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8da2a00640 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b18a2b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009230 tx=0x7f8d94097960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23c20f4b23ec74a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.667+0000 7f8d8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8d8cc00640 1 -- 172.21.15.7:0/1405472739 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94099830 con 0x555d3b17b8c0 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8d8cc00640 1 -- 172.21.15.7:0/1405472739 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8d940b4510 con 0x555d3b17b8c0 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 1 -- 172.21.15.7:0/1405472739 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d9409bdf0 con 0x555d3b17b8c0 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8d8cc00640 1 -- 172.21.15.7:0/1405472739 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f8d9409b050 con 0x555d3b17b8c0 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 10 client.26350 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 1 -- 172.21.15.7:0/1405472739 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x555d3b18d160 con 0x555d3b17b8c0 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8d8cc00640 1 -- 172.21.15.7:0/1405472739 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d9409bdf0 con 0x555d3b17b8c0 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8d8cc00640 1 -- 172.21.15.7:0/1405472739 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f8d94097cd0 con 0x555d3b17b8c0 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 10 client.26350 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 10 client.26350 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 20 client.26350 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x555d3b1f8260 0x555d3b1fa680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 4 client.26350 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:03.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.668+0000 7f8dac9ff740 1 -- 172.21.15.7:0/1405472739 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d3b0e7ce0 con 0x555d3b1f8260 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.669+0000 7f8da2000640 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x555d3b1f8260 0x555d3b1fa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.669+0000 7f8da2000640 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x555d3b1f8260 0x555d3b1fa680 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ff009eff878ecdb5 server_cookie=1b590f68faa18d0e in_seq=0 out_seq=0 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.669+0000 7f8d8cc00640 10 client.26350 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.669+0000 7f8d8cc00640 1 -- 172.21.15.7:0/1405472739 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d3b0e7ce0 con 0x555d3b1f8260 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.669+0000 7f8d8cc00640 10 client.26350 handle_command_reply: tid=0 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.669+0000 7f8dac9ff740 1 client.26350 shutdown 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 -- 172.21.15.7:0/1405472739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 msgr2=0x555d3b18a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b18a2b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009230 tx=0x7f8d94097960 comp rx=0 tx=0).stop 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 -- 172.21.15.7:0/1405472739 shutdown_connections 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x555d3b1f8260 0x555d3b1fa680 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555d3b181420 0x555d3b18b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x555d3b17c2b0 0x555d3b18a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8d8cc00640 0 client.26350 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:47:03.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 --2- 172.21.15.7:0/1405472739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x555d3b17b8c0 0x555d3b18a2b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 -- 172.21.15.7:0/1405472739 >> 172.21.15.7:0/1405472739 conn(0x555d3b176e30 msgr2=0x555d3b177210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 -- 172.21.15.7:0/1405472739 shutdown_connections 2024-03-18T01:47:03.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 1 -- 172.21.15.7:0/1405472739 wait complete. 2024-03-18T01:47:03.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.670+0000 7f8dac9ff740 20 client.26350 trim_cache size 0 max 16384 2024-03-18T01:47:03.675 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 -- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d880a2880 msgr2=0x7f8d880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d880a2880 0x7f8d880a4d40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c006010 tx=0x7f8d9c032590 comp rx=0 tx=0).stop 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 -- 172.21.15.7:0/1798110257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da4074240 msgr2=0x7f8da4199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da4074240 0x7f8da4199c50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400ea60 tx=0x7f8d9400ef20 comp rx=0 tx=0).stop 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 -- 172.21.15.7:0/1798110257 shutdown_connections 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f8d880a2880 0x7f8d880a4d40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8da41096a0 0x7f8da41a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da4075820 0x7f8da419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 --2- 172.21.15.7:0/1798110257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da4074240 0x7f8da4199c50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 -- 172.21.15.7:0/1798110257 >> 172.21.15.7:0/1798110257 conn(0x7f8da40fcd10 msgr2=0x7f8da41055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 -- 172.21.15.7:0/1798110257 shutdown_connections 2024-03-18T01:47:03.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.671+0000 7f8da9c00640 1 -- 172.21.15.7:0/1798110257 wait complete. 2024-03-18T01:47:03.689 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T01:47:03.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 -- 172.21.15.7:0/3584012603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 msgr2=0x7f7c48108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 --2- 172.21.15.7:0/3584012603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 0x7f7c48108240 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009900 tx=0x7f7c3c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:47:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 -- 172.21.15.7:0/3584012603 shutdown_connections 2024-03-18T01:47:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 --2- 172.21.15.7:0/3584012603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c48109930 0x7f7c48110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 --2- 172.21.15.7:0/3584012603 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c48108f70 0x7f7c481093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 --2- 172.21.15.7:0/3584012603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 0x7f7c48108240 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 -- 172.21.15.7:0/3584012603 >> 172.21.15.7:0/3584012603 conn(0x7f7c48076de0 msgr2=0x7f7c48079200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 -- 172.21.15.7:0/3584012603 shutdown_connections 2024-03-18T01:47:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.901+0000 7f7c4f800640 1 -- 172.21.15.7:0/3584012603 wait complete. 2024-03-18T01:47:03.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.902+0000 7f7c4f800640 1 Processor -- start 2024-03-18T01:47:03.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.902+0000 7f7c4f800640 1 -- start start 2024-03-18T01:47:03.904 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.902+0000 7f7c4f800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 0x7f7c4819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.902+0000 7f7c4f800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c48108f70 0x7f7c4819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.902+0000 7f7c4f800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c48109930 0x7f7c481a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.902+0000 7f7c4f800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48113af0 con 0x7f7c48108f70 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.902+0000 7f7c4f800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48113970 con 0x7f7c48107e20 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.902+0000 7f7c4f800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48113c70 con 0x7f7c48109930 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 0x7f7c4819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 0x7f7c4819e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59778/0 (socket says 172.21.15.7:59778) 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4ce00640 1 -- 172.21.15.7:0/706986347 learned_addr learned my addr 172.21.15.7:0/706986347 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c48108f70 0x7f7c4819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4d800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c48109930 0x7f7c481a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4ce00640 1 -- 172.21.15.7:0/706986347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c48109930 msgr2=0x7f7c481a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c48109930 0x7f7c481a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4ce00640 1 -- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c48108f70 msgr2=0x7f7c4819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c48108f70 0x7f7c4819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4ce00640 1 -- 172.21.15.7:0/706986347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c481a5d70 con 0x7f7c48107e20 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c4d800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c48109930 0x7f7c481a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.903+0000 7f7c47e00640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c48108f70 0x7f7c4819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.904+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 0x7f7c4819e150 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c02c8e0 tx=0x7f7c3c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.904+0000 7f7c45600640 1 -- 172.21.15.7:0/706986347 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c043020 con 0x7f7c48107e20 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.904+0000 7f7c45600640 1 -- 172.21.15.7:0/706986347 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c3c0054e0 con 0x7f7c48107e20 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.904+0000 7f7c45600640 1 -- 172.21.15.7:0/706986347 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c003c20 con 0x7f7c48107e20 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.904+0000 7f7c4f800640 1 -- 172.21.15.7:0/706986347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c481a6000 con 0x7f7c48107e20 2024-03-18T01:47:03.906 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.904+0000 7f7c4f800640 1 -- 172.21.15.7:0/706986347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c481a65a0 con 0x7f7c48107e20 2024-03-18T01:47:03.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.908+0000 7f7c52579740 1 Processor -- start 2024-03-18T01:47:03.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.908+0000 7f7c52579740 1 -- start start 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c52579740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860ba8c0 0x558f860bace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c52579740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860bb2b0 0x558f860bfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c52579740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860c0420 0x558f860c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c52579740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558f860823b0 con 0x558f860bb2b0 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c52579740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558f8601f450 con 0x558f860c0420 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c52579740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558f85fc7cb0 con 0x558f860ba8c0 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c45600640 1 -- 172.21.15.7:0/706986347 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f7c3c046020 con 0x7f7c48107e20 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860c0420 0x558f860c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c4d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860ba8c0 0x558f860bace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860c0420 0x558f860c28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59782/0 (socket says 172.21.15.7:59782) 2024-03-18T01:47:03.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c47e00640 1 -- 172.21.15.7:0/1915389219 learned_addr learned my addr 172.21.15.7:0/1915389219 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c4d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860ba8c0 0x558f860bace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51350/0 (socket says 172.21.15.7:51350) 2024-03-18T01:47:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/1915389219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860bb2b0 0x558f860bfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c47e00640 1 -- 172.21.15.7:0/1915389219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860ba8c0 msgr2=0x558f860bace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c47e00640 1 --2- 172.21.15.7:0/1915389219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860ba8c0 0x558f860bace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.909+0000 7f7c47e00640 1 -- 172.21.15.7:0/1915389219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860bb2b0 msgr2=0x558f860bfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.910+0000 7f7c47e00640 1 --2- 172.21.15.7:0/1915389219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860bb2b0 0x558f860bfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.910+0000 7f7c47e00640 1 -- 172.21.15.7:0/1915389219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f85fe91f0 con 0x558f860c0420 2024-03-18T01:47:03.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.910+0000 7f7c47e00640 1 --2- 172.21.15.7:0/1915389219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860c0420 0x558f860c28b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7c340114f0 tx=0x7f7c340119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4407e810360a9101 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.910+0000 7f7c3a000640 1 -- 172.21.15.7:0/1915389219 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34005020 con 0x558f860c0420 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.910+0000 7f7c3a000640 1 -- 172.21.15.7:0/1915389219 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c340047a0 con 0x558f860c0420 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.910+0000 7f7c3a000640 1 -- 172.21.15.7:0/1915389219 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400f990 con 0x558f860c0420 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c45600640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7c300a2880 0x7f7c300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 -- 172.21.15.7:0/1915389219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860c0420 msgr2=0x558f860c28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 --2- 172.21.15.7:0/1915389219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860c0420 0x558f860c28b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7c340114f0 tx=0x7f7c340119b0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c45600640 1 -- 172.21.15.7:0/706986347 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7c3c0f0db0 con 0x7f7c48107e20 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c4d800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7c300a2880 0x7f7c300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 -- 172.21.15.7:0/1915389219 shutdown_connections 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 --2- 172.21.15.7:0/1915389219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860c0420 0x558f860c28b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 --2- 172.21.15.7:0/1915389219 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860bb2b0 0x558f860bfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 --2- 172.21.15.7:0/1915389219 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860ba8c0 0x558f860bace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 -- 172.21.15.7:0/1915389219 >> 172.21.15.7:0/1915389219 conn(0x558f860b5e30 msgr2=0x558f860b6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 -- 172.21.15.7:0/1915389219 shutdown_connections 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.911+0000 7f7c52579740 1 -- 172.21.15.7:0/1915389219 wait complete. 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c4d800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7c300a2880 0x7f7c300a4d40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x558f860bbb10 tx=0x7f7c40009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c52579740 1 -- start start 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c52579740 1 Processor -- start 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c52579740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c52579740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c52579740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 0x558f860cc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c52579740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558f85fdb110 con 0x558f860c0420 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c52579740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558f8601b4f0 con 0x558f860ba8c0 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.912+0000 7f7c52579740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558f860c33d0 con 0x558f860bb2b0 2024-03-18T01:47:03.915 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c52579740 10 client.0 _ll_register_callbacks cb 0x558f85fc8b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c4ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51354/0 (socket says 172.21.15.7:51354) 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c47e00640 1 -- 172.21.15.7:0/3092757448 learned_addr learned my addr 172.21.15.7:0/3092757448 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c4ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59794/0 (socket says 172.21.15.7:59794) 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c4d800640 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 0x558f860cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c47e00640 1 -- 172.21.15.7:0/3092757448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 msgr2=0x558f860c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c47e00640 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c47e00640 1 -- 172.21.15.7:0/3092757448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 msgr2=0x558f860cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c47e00640 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 0x558f860cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c47e00640 1 -- 172.21.15.7:0/3092757448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f85d333f0 con 0x558f860bb2b0 2024-03-18T01:47:03.916 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.913+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c47e00640 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c50d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7c34000c00 tx=0x7f7c34004e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1949ef3c24cb7895 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:03.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c2ea00640 1 -- 172.21.15.7:0/3092757448 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34028070 con 0x558f860bb2b0 2024-03-18T01:47:03.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c2ea00640 1 -- 172.21.15.7:0/3092757448 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7c34004020 con 0x558f860bb2b0 2024-03-18T01:47:03.919 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c52579740 1 -- 172.21.15.7:0/3092757448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c340047a0 con 0x558f860bb2b0 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c2ea00640 1 -- 172.21.15.7:0/3092757448 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f7c34033050 con 0x558f860bb2b0 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c52579740 10 client.26362 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c52579740 1 -- 172.21.15.7:0/3092757448 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x558f85dfd7d0 con 0x558f860bb2b0 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.914+0000 7f7c2ea00640 1 -- 172.21.15.7:0/3092757448 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c340047a0 con 0x558f860bb2b0 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.915+0000 7f7c2ea00640 1 -- 172.21.15.7:0/3092757448 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f7c3400fc70 con 0x558f860bb2b0 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.915+0000 7f7c52579740 10 client.26362 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.915+0000 7f7c52579740 10 client.26362 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.915+0000 7f7c52579740 20 client.26362 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.915+0000 7f7c52579740 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x558f860d07c0 0x558f860d2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.915+0000 7f7c52579740 4 client.26362 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.915+0000 7f7c52579740 1 -- 172.21.15.7:0/3092757448 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f8602c990 con 0x558f860d07c0 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.915+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x558f860d07c0 0x558f860d2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.920 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.916+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x558f860d07c0 0x558f860d2bd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3ea65cb63d88f3d1 server_cookie=22c7cf52ccd9cc52 in_seq=0 out_seq=0 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.916+0000 7f7c2ea00640 10 client.26362 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.916+0000 7f7c2ea00640 1 -- 172.21.15.7:0/3092757448 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558f8602c990 con 0x558f860d07c0 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.916+0000 7f7c2ea00640 10 client.26362 handle_command_reply: tid=0 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.916+0000 7f7c52579740 1 client.26362 shutdown 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 -- 172.21.15.7:0/3092757448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 msgr2=0x558f860c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c50d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7c34000c00 tx=0x7f7c34004e30 comp rx=0 tx=0).stop 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 -- 172.21.15.7:0/3092757448 shutdown_connections 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 0x558f860cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x558f860d07c0 0x558f860d2bd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c50d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 --2- 172.21.15.7:0/3092757448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c2ea00640 0 client.26362 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:47:03.921 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 -- 172.21.15.7:0/3092757448 >> 172.21.15.7:0/3092757448 conn(0x558f860b5e30 msgr2=0x558f860b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 -- 172.21.15.7:0/3092757448 shutdown_connections 2024-03-18T01:47:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 1 -- 172.21.15.7:0/3092757448 wait complete. 2024-03-18T01:47:03.922 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.917+0000 7f7c52579740 20 client.26362 trim_cache size 0 max 16384 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.932+0000 7f7c52579740 1 Processor -- start 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.932+0000 7f7c52579740 1 -- start start 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.932+0000 7f7c52579740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860ba8c0 0x558f860bc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.932+0000 7f7c52579740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860bb2b0 0x558f860bcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.932+0000 7f7c52579740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860c0420 0x558f860cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.932+0000 7f7c52579740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7c481b0990 con 0x558f860ba8c0 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.932+0000 7f7c52579740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c481b25f0 con 0x558f860bb2b0 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.932+0000 7f7c52579740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c481b1090 con 0x558f860c0420 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860bb2b0 0x558f860bcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860bb2b0 0x558f860bcc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59796/0 (socket says 172.21.15.7:59796) 2024-03-18T01:47:03.933 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 -- 172.21.15.7:0/3202619961 learned_addr learned my addr 172.21.15.7:0/3202619961 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c4d800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860c0420 0x558f860cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/3202619961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860ba8c0 0x558f860bc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 -- 172.21.15.7:0/3202619961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860c0420 msgr2=0x558f860cc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 --2- 172.21.15.7:0/3202619961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860c0420 0x558f860cc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 -- 172.21.15.7:0/3202619961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860ba8c0 msgr2=0x558f860bc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 --2- 172.21.15.7:0/3202619961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860ba8c0 0x558f860bc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 -- 172.21.15.7:0/3202619961 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f860b1c10 con 0x558f860bb2b0 2024-03-18T01:47:03.935 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c47e00640 1 --2- 172.21.15.7:0/3202619961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860bb2b0 0x558f860bcc50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7c340119f0 tx=0x7f7c34005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b10d547d01dbf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.933+0000 7f7c39600640 1 -- 172.21.15.7:0/3202619961 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34030020 con 0x558f860bb2b0 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c39600640 1 -- 172.21.15.7:0/3202619961 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c34003860 con 0x558f860bb2b0 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c39600640 1 -- 172.21.15.7:0/3202619961 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34028040 con 0x558f860bb2b0 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c52579740 1 -- 172.21.15.7:0/3202619961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860bb2b0 msgr2=0x558f860bcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c52579740 1 --2- 172.21.15.7:0/3202619961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860bb2b0 0x558f860bcc50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7c340119f0 tx=0x7f7c34005970 comp rx=0 tx=0).stop 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c52579740 1 -- 172.21.15.7:0/3202619961 shutdown_connections 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c52579740 1 --2- 172.21.15.7:0/3202619961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860c0420 0x558f860cc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c52579740 1 --2- 172.21.15.7:0/3202619961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860bb2b0 0x558f860bcc50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c52579740 1 --2- 172.21.15.7:0/3202619961 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860ba8c0 0x558f860bc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.934+0000 7f7c52579740 1 -- 172.21.15.7:0/3202619961 >> 172.21.15.7:0/3202619961 conn(0x558f860b5e30 msgr2=0x558f860b6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.935+0000 7f7c52579740 1 -- 172.21.15.7:0/3202619961 shutdown_connections 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.935+0000 7f7c52579740 1 -- 172.21.15.7:0/3202619961 wait complete. 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.935+0000 7f7c52579740 1 -- start start 2024-03-18T01:47:03.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.935+0000 7f7c52579740 1 Processor -- start 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c52579740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c52579740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c52579740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 0x558f86134b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c52579740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7c481b1ec0 con 0x558f860c0420 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c52579740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c481b1790 con 0x558f860ba8c0 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c52579740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c481ae970 con 0x558f860bb2b0 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c52579740 10 client.0 _ll_register_callbacks cb 0x7f7c40009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59810/0 (socket says 172.21.15.7:59810) 2024-03-18T01:47:03.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4ce00640 1 -- 172.21.15.7:0/1978208770 learned_addr learned my addr 172.21.15.7:0/1978208770 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c47e00640 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4d800640 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 0x558f86134b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4ce00640 1 -- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 msgr2=0x558f860c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4ce00640 1 -- 172.21.15.7:0/1978208770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 msgr2=0x558f86134b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 0x558f86134b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c4ce00640 1 -- 172.21.15.7:0/1978208770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f860b6ba0 con 0x558f860ba8c0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.936+0000 7f7c47e00640 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c4ce00640 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c91a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0ee8c0 tx=0x7f7c3c0f32c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24d41c7b2fe5e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c2ea00640 1 -- 172.21.15.7:0/1978208770 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c005690 con 0x558f860ba8c0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c2ea00640 1 -- 172.21.15.7:0/1978208770 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7c3c003dc0 con 0x558f860ba8c0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c52579740 1 -- 172.21.15.7:0/1978208770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c34003860 con 0x558f860ba8c0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c2ea00640 1 -- 172.21.15.7:0/1978208770 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f7c3c0bdaf0 con 0x558f860ba8c0 2024-03-18T01:47:03.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c52579740 10 client.26387 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c52579740 1 -- 172.21.15.7:0/1978208770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x558f86136290 con 0x558f860ba8c0 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.937+0000 7f7c2ea00640 1 -- 172.21.15.7:0/1978208770 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c34003860 con 0x558f860ba8c0 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.938+0000 7f7c2ea00640 1 -- 172.21.15.7:0/1978208770 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f7c3c0c1170 con 0x558f860ba8c0 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.938+0000 7f7c52579740 10 client.26387 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.938+0000 7f7c52579740 10 client.26387 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.938+0000 7f7c52579740 20 client.26387 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.938+0000 7f7c52579740 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x558f86136c30 0x558f86139050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.938+0000 7f7c52579740 4 client.26387 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.938+0000 7f7c52579740 1 -- 172.21.15.7:0/1978208770 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f86026ce0 con 0x558f86136c30 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.938+0000 7f7c47e00640 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x558f86136c30 0x558f86139050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c47e00640 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x558f86136c30 0x558f86139050 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3e25b37841bd4d61 server_cookie=59a31be5606de362 in_seq=0 out_seq=0 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c2ea00640 10 client.26387 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:47:03.942 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c2ea00640 1 -- 172.21.15.7:0/1978208770 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f86026ce0 con 0x558f86136c30 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c2ea00640 10 client.26387 handle_command_reply: tid=0 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 client.26387 shutdown 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 -- 172.21.15.7:0/1978208770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 msgr2=0x558f860c91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c91a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c0ee8c0 tx=0x7f7c3c0f32c0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 -- 172.21.15.7:0/1978208770 shutdown_connections 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558f860c0420 0x558f86134b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x558f86136c30 0x558f86139050 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558f860bb2b0 0x558f860c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c2ea00640 0 client.26387 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 --2- 172.21.15.7:0/1978208770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558f860ba8c0 0x558f860c91a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 -- 172.21.15.7:0/1978208770 >> 172.21.15.7:0/1978208770 conn(0x558f860b5e30 msgr2=0x558f860c4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 -- 172.21.15.7:0/1978208770 shutdown_connections 2024-03-18T01:47:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 1 -- 172.21.15.7:0/1978208770 wait complete. 2024-03-18T01:47:03.944 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.939+0000 7f7c52579740 20 client.26387 trim_cache size 0 max 16384 2024-03-18T01:47:03.944 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 -- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7c300a2880 msgr2=0x7f7c300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7c300a2880 0x7f7c300a4d40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x558f860bbb10 tx=0x7f7c40009500 comp rx=0 tx=0).stop 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 -- 172.21.15.7:0/706986347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 msgr2=0x7f7c4819e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 0x7f7c4819e150 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c02c8e0 tx=0x7f7c3c02c650 comp rx=0 tx=0).stop 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 -- 172.21.15.7:0/706986347 shutdown_connections 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c48109930 0x7f7c481a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7c300a2880 0x7f7c300a4d40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7c48108f70 0x7f7c4819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 --2- 172.21.15.7:0/706986347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c48107e20 0x7f7c4819e150 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 -- 172.21.15.7:0/706986347 >> 172.21.15.7:0/706986347 conn(0x7f7c48076de0 msgr2=0x7f7c4810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:03.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.940+0000 7f7c4f800640 1 -- 172.21.15.7:0/706986347 shutdown_connections 2024-03-18T01:47:03.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:03.941+0000 7f7c4f800640 1 -- 172.21.15.7:0/706986347 wait complete. 2024-03-18T01:47:03.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 -- 172.21.15.7:0/1841497098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 msgr2=0x7fd570108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 --2- 172.21.15.7:0/1841497098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 0x7fd570108240 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd560009900 tx=0x7fd56002bc30 comp rx=0 tx=0).stop 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 -- 172.21.15.7:0/1841497098 shutdown_connections 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 --2- 172.21.15.7:0/1841497098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570109930 0x7fd570110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 --2- 172.21.15.7:0/1841497098 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd570108f70 0x7fd5701093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 --2- 172.21.15.7:0/1841497098 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 0x7fd570108240 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 -- 172.21.15.7:0/1841497098 >> 172.21.15.7:0/1841497098 conn(0x7fd570076de0 msgr2=0x7fd570079200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 -- 172.21.15.7:0/1841497098 shutdown_connections 2024-03-18T01:47:04.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.178+0000 7fd575200640 1 -- 172.21.15.7:0/1841497098 wait complete. 2024-03-18T01:47:04.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.179+0000 7fd575200640 1 Processor -- start 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.179+0000 7fd575200640 1 -- start start 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.179+0000 7fd575200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 0x7fd57019e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.179+0000 7fd575200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570108f70 0x7fd57019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.179+0000 7fd575200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd570109930 0x7fd5701a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd575200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd570113aa0 con 0x7fd570109930 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd575200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd570113920 con 0x7fd570107e20 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd575200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd570113c20 con 0x7fd570108f70 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 0x7fd57019e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 0x7fd57019e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59834/0 (socket says 172.21.15.7:59834) 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56de00640 1 -- 172.21.15.7:0/3744238764 learned_addr learned my addr 172.21.15.7:0/3744238764 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd565400640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570108f70 0x7fd57019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56e800640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd570109930 0x7fd5701a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56de00640 1 -- 172.21.15.7:0/3744238764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570108f70 msgr2=0x7fd57019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56de00640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570108f70 0x7fd57019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56de00640 1 -- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd570109930 msgr2=0x7fd5701a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56de00640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd570109930 0x7fd5701a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56de00640 1 -- 172.21.15.7:0/3744238764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5701a5d30 con 0x7fd570107e20 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd56e800640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd570109930 0x7fd5701a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.180+0000 7fd565400640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570108f70 0x7fd57019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.181+0000 7fd56de00640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 0x7fd57019e110 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd560009230 tx=0x7fd56002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.181+0000 7fd567400640 1 -- 172.21.15.7:0/3744238764 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560041070 con 0x7fd570107e20 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.181+0000 7fd567400640 1 -- 172.21.15.7:0/3744238764 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd560004450 con 0x7fd570107e20 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.181+0000 7fd575200640 1 -- 172.21.15.7:0/3744238764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5701a5fc0 con 0x7fd570107e20 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.181+0000 7fd567400640 1 -- 172.21.15.7:0/3744238764 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd560003e20 con 0x7fd570107e20 2024-03-18T01:47:04.183 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.181+0000 7fd575200640 1 -- 172.21.15.7:0/3744238764 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5701a6420 con 0x7fd570107e20 2024-03-18T01:47:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.184+0000 7fd577fa1740 1 Processor -- start 2024-03-18T01:47:04.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.184+0000 7fd577fa1740 1 -- start start 2024-03-18T01:47:04.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.184+0000 7fd567400640 1 -- 172.21.15.7:0/3744238764 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fd560044070 con 0x7fd570107e20 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92ae8c0 0x5652c92aece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92b4420 0x5652c92b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd577fa1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5652c92763b0 con 0x5652c92b4420 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd577fa1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5652c9213450 con 0x5652c92ae8c0 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd577fa1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5652c91bbcb0 con 0x5652c92af2b0 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92ae8c0 0x5652c92aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd565400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92b4420 0x5652c92b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92ae8c0 0x5652c92aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59848/0 (socket says 172.21.15.7:59848) 2024-03-18T01:47:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56e800640 1 -- 172.21.15.7:0/433978723 learned_addr learned my addr 172.21.15.7:0/433978723 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56de00640 1 --2- 172.21.15.7:0/433978723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56e800640 1 -- 172.21.15.7:0/433978723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 msgr2=0x5652c92b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56e800640 1 --2- 172.21.15.7:0/433978723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56e800640 1 -- 172.21.15.7:0/433978723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92b4420 msgr2=0x5652c92b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56e800640 1 --2- 172.21.15.7:0/433978723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92b4420 0x5652c92b68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.185+0000 7fd56e800640 1 -- 172.21.15.7:0/433978723 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652c91dd1f0 con 0x5652c92ae8c0 2024-03-18T01:47:04.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.186+0000 7fd56e800640 1 --2- 172.21.15.7:0/433978723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92ae8c0 0x5652c92aece0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd568010450 tx=0x7fd568010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3ced148eee3f1cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.186+0000 7fd567400640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd5540a2880 0x7fd5540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.186+0000 7fd55f400640 1 -- 172.21.15.7:0/433978723 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd568017070 con 0x5652c92ae8c0 2024-03-18T01:47:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.186+0000 7fd55f400640 1 -- 172.21.15.7:0/433978723 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd568011c40 con 0x5652c92ae8c0 2024-03-18T01:47:04.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.186+0000 7fd55f400640 1 -- 172.21.15.7:0/433978723 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56801ad80 con 0x5652c92ae8c0 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.186+0000 7fd567400640 1 -- 172.21.15.7:0/3744238764 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd5600f09d0 con 0x7fd570107e20 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.186+0000 7fd56de00640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd5540a2880 0x7fd5540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 -- 172.21.15.7:0/433978723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92ae8c0 msgr2=0x5652c92aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 --2- 172.21.15.7:0/433978723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92ae8c0 0x5652c92aece0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd568010450 tx=0x7fd568010910 comp rx=0 tx=0).stop 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 -- 172.21.15.7:0/433978723 shutdown_connections 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 --2- 172.21.15.7:0/433978723 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92b4420 0x5652c92b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 --2- 172.21.15.7:0/433978723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 --2- 172.21.15.7:0/433978723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92ae8c0 0x5652c92aece0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 -- 172.21.15.7:0/433978723 >> 172.21.15.7:0/433978723 conn(0x5652c92a9e30 msgr2=0x5652c92aa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 -- 172.21.15.7:0/433978723 shutdown_connections 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd56de00640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd5540a2880 0x7fd5540a4d40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd56002cfd0 tx=0x7fd5600f6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.187+0000 7fd577fa1740 1 -- 172.21.15.7:0/433978723 wait complete. 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 1 -- start start 2024-03-18T01:47:04.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 1 Processor -- start 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92ae8c0 0x5652c92b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92af2b0 0x5652c92b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5652c91cf110 con 0x5652c92af2b0 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5652c920f4f0 con 0x5652c92b4420 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5652c92b73d0 con 0x5652c92ae8c0 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.188+0000 7fd577fa1740 10 client.0 _ll_register_callbacks cb 0x5652c91bcb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92af2b0 0x5652c92b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd56e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92ae8c0 0x5652c92b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92af2b0 0x5652c92b90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43850/0 (socket says 172.21.15.7:43850) 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 -- 172.21.15.7:0/2754870525 learned_addr learned my addr 172.21.15.7:0/2754870525 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd56e800640 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 -- 172.21.15.7:0/2754870525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92ae8c0 msgr2=0x5652c92b8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92ae8c0 0x5652c92b8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 -- 172.21.15.7:0/2754870525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 msgr2=0x5652c92c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 -- 172.21.15.7:0/2754870525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652c8f273f0 con 0x5652c92af2b0 2024-03-18T01:47:04.192 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd56e800640 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:04.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.189+0000 7fd565400640 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92af2b0 0x5652c92b90a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd558012d20 tx=0x7fd55800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20ef6186045148a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.190+0000 7fd553e00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:47:04.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.190+0000 7fd553e00640 1 -- 172.21.15.7:0/2754870525 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558012110 con 0x5652c92af2b0 2024-03-18T01:47:04.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.190+0000 7fd553e00640 1 -- 172.21.15.7:0/2754870525 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd558004030 con 0x5652c92af2b0 2024-03-18T01:47:04.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.190+0000 7fd577fa1740 1 -- 172.21.15.7:0/2754870525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd568011c40 con 0x5652c92af2b0 2024-03-18T01:47:04.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.190+0000 7fd553e00640 1 -- 172.21.15.7:0/2754870525 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7fd558025050 con 0x5652c92af2b0 2024-03-18T01:47:04.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.190+0000 7fd577fa1740 10 client.16077 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.190+0000 7fd577fa1740 1 -- 172.21.15.7:0/2754870525 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x5652c8ff17d0 con 0x5652c92af2b0 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.191+0000 7fd553e00640 1 -- 172.21.15.7:0/2754870525 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd568011c40 con 0x5652c92af2b0 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.191+0000 7fd553e00640 1 -- 172.21.15.7:0/2754870525 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7fd558010430 con 0x5652c92af2b0 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.191+0000 7fd577fa1740 10 client.16077 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.191+0000 7fd577fa1740 10 client.16077 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.191+0000 7fd577fa1740 20 client.16077 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.191+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5652c92c4b70 0x5652c92c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.191+0000 7fd577fa1740 4 client.16077 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.191+0000 7fd577fa1740 1 -- 172.21.15.7:0/2754870525 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652c9220990 con 0x5652c92c4b70 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.192+0000 7fd56e800640 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5652c92c4b70 0x5652c92c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.192+0000 7fd56e800640 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5652c92c4b70 0x5652c92c6fb0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=64803db22ca59ac9 server_cookie=bc9a4b1ef90d629c in_seq=0 out_seq=0 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.192+0000 7fd553e00640 10 client.16077 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd553e00640 1 -- 172.21.15.7:0/2754870525 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5652c9220990 con 0x5652c92c4b70 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd553e00640 10 client.16077 handle_command_reply: tid=0 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 client.16077 shutdown 2024-03-18T01:47:04.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 -- 172.21.15.7:0/2754870525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92af2b0 msgr2=0x5652c92b90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92af2b0 0x5652c92b90a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd558012d20 tx=0x7fd55800f790 comp rx=0 tx=0).stop 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 -- 172.21.15.7:0/2754870525 shutdown_connections 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5652c92c4b70 0x5652c92c6fb0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92af2b0 0x5652c92b90a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2754870525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92ae8c0 0x5652c92b8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd553e00640 0 client.16077 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.193+0000 7fd577fa1740 1 -- 172.21.15.7:0/2754870525 >> 172.21.15.7:0/2754870525 conn(0x5652c92a9e30 msgr2=0x5652c92a91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.194+0000 7fd577fa1740 1 -- 172.21.15.7:0/2754870525 shutdown_connections 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.194+0000 7fd577fa1740 1 -- 172.21.15.7:0/2754870525 wait complete. 2024-03-18T01:47:04.196 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.194+0000 7fd577fa1740 20 client.16077 trim_cache size 0 max 16384 2024-03-18T01:47:04.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.223+0000 7fd577fa1740 1 Processor -- start 2024-03-18T01:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.223+0000 7fd577fa1740 1 -- start start 2024-03-18T01:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92a9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92c00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd577fa1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5701b0950 con 0x5652c92ae8c0 2024-03-18T01:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd577fa1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5701b25b0 con 0x5652c92b4420 2024-03-18T01:47:04.224 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd577fa1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5701b1050 con 0x5652c92af2b0 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd56e800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd56e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd56e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59862/0 (socket says 172.21.15.7:59862) 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd56e800640 1 -- 172.21.15.7:0/1091322122 learned_addr learned my addr 172.21.15.7:0/1091322122 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.224+0000 7fd565400640 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92c00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd56e800640 1 -- 172.21.15.7:0/1091322122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 msgr2=0x5652c92c00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd56e800640 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92c00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd56e800640 1 -- 172.21.15.7:0/1091322122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 msgr2=0x5652c92a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd56e800640 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92a9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd56e800640 1 -- 172.21.15.7:0/1091322122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652c92aaba0 con 0x5652c92b4420 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd565400640 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92c00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:04.226 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd56e800640 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92cb870 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x5652c92b3f90 tx=0x7fd568010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=527ee3c1473b35a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd55ea00640 1 -- 172.21.15.7:0/1091322122 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd568017040 con 0x5652c92b4420 2024-03-18T01:47:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd55ea00640 1 -- 172.21.15.7:0/1091322122 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd568021040 con 0x5652c92b4420 2024-03-18T01:47:04.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.225+0000 7fd55ea00640 1 -- 172.21.15.7:0/1091322122 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd568011940 con 0x5652c92b4420 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.226+0000 7fd577fa1740 1 -- 172.21.15.7:0/1091322122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 msgr2=0x5652c92cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.226+0000 7fd577fa1740 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92cb870 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x5652c92b3f90 tx=0x7fd568010910 comp rx=0 tx=0).stop 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.226+0000 7fd577fa1740 1 -- 172.21.15.7:0/1091322122 shutdown_connections 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.226+0000 7fd577fa1740 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92b4420 0x5652c92cb870 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.226+0000 7fd577fa1740 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92af2b0 0x5652c92c00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.226+0000 7fd577fa1740 1 --2- 172.21.15.7:0/1091322122 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.226+0000 7fd577fa1740 1 -- 172.21.15.7:0/1091322122 >> 172.21.15.7:0/1091322122 conn(0x5652c92a9e30 msgr2=0x5652c92a8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.227+0000 7fd577fa1740 1 -- 172.21.15.7:0/1091322122 shutdown_connections 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.227+0000 7fd577fa1740 1 -- 172.21.15.7:0/1091322122 wait complete. 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.227+0000 7fd577fa1740 1 -- start start 2024-03-18T01:47:04.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.227+0000 7fd577fa1740 1 Processor -- start 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92bdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92af2b0 0x5652c92be270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd577fa1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92b4420 0x5652c92bf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd577fa1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd5701b1e80 con 0x5652c92ae8c0 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd577fa1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd5701b1750 con 0x5652c92af2b0 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd577fa1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5701ae930 con 0x5652c92b4420 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd577fa1740 10 client.0 _ll_register_callbacks cb 0x7fd558009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd565400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92af2b0 0x5652c92be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd565400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92af2b0 0x5652c92be270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59864/0 (socket says 172.21.15.7:59864) 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd56e800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92bdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd565400640 1 -- 172.21.15.7:0/2795549927 learned_addr learned my addr 172.21.15.7:0/2795549927 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.228+0000 7fd56e800640 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92b4420 0x5652c92bf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd565400640 1 -- 172.21.15.7:0/2795549927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92b4420 msgr2=0x5652c92bf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd565400640 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92b4420 0x5652c92bf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd565400640 1 -- 172.21.15.7:0/2795549927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 msgr2=0x5652c92bdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd565400640 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92bdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd565400640 1 -- 172.21.15.7:0/2795549927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652c92bf870 con 0x5652c92af2b0 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd56e800640 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92bdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd565400640 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92af2b0 0x5652c92be270 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd558010a10 tx=0x7fd55800fe80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=779b74b61faaa18a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd553e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd553e00640 1 -- 172.21.15.7:0/2795549927 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558010e60 con 0x5652c92af2b0 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd553e00640 1 -- 172.21.15.7:0/2795549927 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd558010430 con 0x5652c92af2b0 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd577fa1740 1 -- 172.21.15.7:0/2795549927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd568021040 con 0x5652c92af2b0 2024-03-18T01:47:04.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd553e00640 1 -- 172.21.15.7:0/2795549927 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7fd558036050 con 0x5652c92af2b0 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.229+0000 7fd577fa1740 10 client.26405 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd577fa1740 1 -- 172.21.15.7:0/2795549927 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x5652c932b490 con 0x5652c92af2b0 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd553e00640 1 -- 172.21.15.7:0/2795549927 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd568021040 con 0x5652c92af2b0 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd553e00640 1 -- 172.21.15.7:0/2795549927 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7fd558031030 con 0x5652c92af2b0 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd577fa1740 10 client.26405 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd577fa1740 10 client.26405 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd577fa1740 20 client.26405 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5652c932bec0 0x5652c932e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd577fa1740 4 client.26405 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd577fa1740 1 -- 172.21.15.7:0/2795549927 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5652c921ace0 con 0x5652c932bec0 2024-03-18T01:47:04.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.230+0000 7fd56e800640 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5652c932bec0 0x5652c932e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.231+0000 7fd56e800640 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5652c932bec0 0x5652c932e2e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4dac7658346a93b5 server_cookie=adcc90476af90aa2 in_seq=0 out_seq=0 2024-03-18T01:47:04.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.231+0000 7fd553e00640 10 client.26405 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.231+0000 7fd553e00640 1 -- 172.21.15.7:0/2795549927 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5652c921ace0 con 0x5652c932bec0 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.231+0000 7fd553e00640 10 client.26405 handle_command_reply: tid=0 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.231+0000 7fd577fa1740 1 client.26405 shutdown 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.231+0000 7fd577fa1740 1 -- 172.21.15.7:0/2795549927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92af2b0 msgr2=0x5652c92be270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.231+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92af2b0 0x5652c92be270 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd558010a10 tx=0x7fd55800fe80 comp rx=0 tx=0).stop 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 1 -- 172.21.15.7:0/2795549927 shutdown_connections 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x5652c932bec0 0x5652c932e2e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5652c92b4420 0x5652c92bf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5652c92af2b0 0x5652c92be270 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd553e00640 0 client.26405 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 1 --2- 172.21.15.7:0/2795549927 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5652c92ae8c0 0x5652c92bdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 1 -- 172.21.15.7:0/2795549927 >> 172.21.15.7:0/2795549927 conn(0x5652c92a9e30 msgr2=0x5652c92a91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 1 -- 172.21.15.7:0/2795549927 shutdown_connections 2024-03-18T01:47:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 1 -- 172.21.15.7:0/2795549927 wait complete. 2024-03-18T01:47:04.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd577fa1740 20 client.26405 trim_cache size 0 max 16384 2024-03-18T01:47:04.236 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd575200640 1 -- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd5540a2880 msgr2=0x7fd5540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.232+0000 7fd575200640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd5540a2880 0x7fd5540a4d40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd56002cfd0 tx=0x7fd5600f6530 comp rx=0 tx=0).stop 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 -- 172.21.15.7:0/3744238764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 msgr2=0x7fd57019e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 0x7fd57019e110 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd560009230 tx=0x7fd56002c650 comp rx=0 tx=0).stop 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 -- 172.21.15.7:0/3744238764 shutdown_connections 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd5540a2880 0x7fd5540a4d40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd570109930 0x7fd5701a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd570108f70 0x7fd57019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 --2- 172.21.15.7:0/3744238764 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd570107e20 0x7fd57019e110 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 -- 172.21.15.7:0/3744238764 >> 172.21.15.7:0/3744238764 conn(0x7fd570076de0 msgr2=0x7fd57010c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 -- 172.21.15.7:0/3744238764 shutdown_connections 2024-03-18T01:47:04.237 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.233+0000 7fd575200640 1 -- 172.21.15.7:0/3744238764 wait complete. 2024-03-18T01:47:04.255 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:04 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1355654152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:47:04.257 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:47:04.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:04 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1355654152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 -- 172.21.15.7:0/3419623024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 msgr2=0x7fefc80765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3419623024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 0x7fefc80765c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009900 tx=0x7fefb802bc40 comp rx=0 tx=0).stop 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 -- 172.21.15.7:0/3419623024 shutdown_connections 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3419623024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefc8077d80 0x7fefc8078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3419623024 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefc80773e0 0x7fefc8077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3419623024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 0x7fefc80765c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 -- 172.21.15.7:0/3419623024 >> 172.21.15.7:0/3419623024 conn(0x7fefc80fed80 msgr2=0x7fefc81011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 -- 172.21.15.7:0/3419623024 shutdown_connections 2024-03-18T01:47:04.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.495+0000 7fefcdc00640 1 -- 172.21.15.7:0/3419623024 wait complete. 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.496+0000 7fefcdc00640 1 Processor -- start 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.496+0000 7fefcdc00640 1 -- start start 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefcdc00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 0x7fefc819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefcdc00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefc80773e0 0x7fefc819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefcdc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefc8077d80 0x7fefc81a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefcdc00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81138c0 con 0x7fefc80773e0 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefcdc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8113740 con 0x7fefc8077d80 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefcdc00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc8113a40 con 0x7fefc80761a0 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefc7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 0x7fefc819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.498 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefc7400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefc80773e0 0x7fefc819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefcd200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefc8077d80 0x7fefc81a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefc7e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 0x7fefc819def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51446/0 (socket says 172.21.15.7:51446) 2024-03-18T01:47:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.497+0000 7fefc7e00640 1 -- 172.21.15.7:0/3748193670 learned_addr learned my addr 172.21.15.7:0/3748193670 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc7e00640 1 -- 172.21.15.7:0/3748193670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefc8077d80 msgr2=0x7fefc81a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc7e00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefc8077d80 0x7fefc81a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc7e00640 1 -- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefc80773e0 msgr2=0x7fefc819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc7e00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefc80773e0 0x7fefc819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc7e00640 1 -- 172.21.15.7:0/3748193670 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc81a5ac0 con 0x7fefc80761a0 2024-03-18T01:47:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc7e00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 0x7fefc819def0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fefb802c8f0 tx=0x7fefb8005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc4c00640 1 -- 172.21.15.7:0/3748193670 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8041070 con 0x7fefc80761a0 2024-03-18T01:47:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc4c00640 1 -- 172.21.15.7:0/3748193670 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb80044d0 con 0x7fefc80761a0 2024-03-18T01:47:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefcdc00640 1 -- 172.21.15.7:0/3748193670 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc81a5d50 con 0x7fefc80761a0 2024-03-18T01:47:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefc4c00640 1 -- 172.21.15.7:0/3748193670 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb8034440 con 0x7fefc80761a0 2024-03-18T01:47:04.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.498+0000 7fefcdc00640 1 -- 172.21.15.7:0/3748193670 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc81a61d0 con 0x7fefc80761a0 2024-03-18T01:47:04.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.503+0000 7fefd1e47740 1 Processor -- start 2024-03-18T01:47:04.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.503+0000 7fefd1e47740 1 -- start start 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.503+0000 7fefc4c00640 1 -- 172.21.15.7:0/3748193670 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fefb8044070 con 0x7fefc80761a0 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef84eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef89f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 0x55f31ef8ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefd1e47740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f31ef4c3b0 con 0x55f31ef85480 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefd1e47740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f31eee9450 con 0x55f31ef8a5f0 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefd1e47740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f31ee91cb0 con 0x55f31ef84a90 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefcd200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef84eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefcd200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef84eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51460/0 (socket says 172.21.15.7:51460) 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefcd200640 1 -- 172.21.15.7:0/1779972601 learned_addr learned my addr 172.21.15.7:0/1779972601 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefc7400640 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 0x55f31ef8ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefc7e00640 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef89f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefcd200640 1 -- 172.21.15.7:0/1779972601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 msgr2=0x55f31ef8ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefcd200640 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 0x55f31ef8ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefcd200640 1 -- 172.21.15.7:0/1779972601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 msgr2=0x55f31ef89f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefcd200640 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef89f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.506 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.504+0000 7fefcd200640 1 -- 172.21.15.7:0/1779972601 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f31eeb31f0 con 0x55f31ef84a90 2024-03-18T01:47:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.505+0000 7fefcd200640 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef84eb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009f50 tx=0x7fefc00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14008cba902c7394 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.505+0000 7fefc4c00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefac0a2870 0x7fefac0a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.505+0000 7fefbd600640 1 -- 172.21.15.7:0/1779972601 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0004d20 con 0x55f31ef84a90 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.505+0000 7fefbd600640 1 -- 172.21.15.7:0/1779972601 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc0004050 con 0x55f31ef84a90 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.505+0000 7fefbd600640 1 -- 172.21.15.7:0/1779972601 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0005280 con 0x55f31ef84a90 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.505+0000 7fefc4c00640 1 -- 172.21.15.7:0/3748193670 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fefb80f0900 con 0x7fefc80761a0 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.505+0000 7fefc7e00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefac0a2870 0x7fefac0a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 -- 172.21.15.7:0/1779972601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 msgr2=0x55f31ef84eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef84eb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fefc0009f50 tx=0x7fefc00118d0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 -- 172.21.15.7:0/1779972601 shutdown_connections 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 0x55f31ef8ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef89f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 --2- 172.21.15.7:0/1779972601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef84eb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 -- 172.21.15.7:0/1779972601 >> 172.21.15.7:0/1779972601 conn(0x55f31ef80000 msgr2=0x55f31ef80400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefc7e00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefac0a2870 0x7fefac0a4d30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fefb802c180 tx=0x7fefb802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 -- 172.21.15.7:0/1779972601 shutdown_connections 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 -- 172.21.15.7:0/1779972601 wait complete. 2024-03-18T01:47:04.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.506+0000 7fefd1e47740 1 -- start start 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefd1e47740 1 Processor -- start 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef8ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef8f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 0x55f31ef962f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefd1e47740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55f31eea5110 con 0x55f31ef85480 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefd1e47740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55f31eee54f0 con 0x55f31ef8a5f0 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefd1e47740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f31ef8d5a0 con 0x55f31ef84a90 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefd1e47740 10 client.0 _ll_register_callbacks cb 0x55f31ee92b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefc7400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef8f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefcd200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef8ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefc7400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef8f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:43902/0 (socket says 172.21.15.7:43902) 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.507+0000 7fefc7400640 1 -- 172.21.15.7:0/259594484 learned_addr learned my addr 172.21.15.7:0/259594484 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefcd200640 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 0x55f31ef962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.510 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefc7400640 1 -- 172.21.15.7:0/259594484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 msgr2=0x55f31ef8ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefc7400640 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef8ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefc7400640 1 -- 172.21.15.7:0/259594484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 msgr2=0x55f31ef962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefc7400640 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 0x55f31ef962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefc7400640 1 -- 172.21.15.7:0/259594484 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f31ebfd3f0 con 0x55f31ef85480 2024-03-18T01:47:04.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefc7400640 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef8f2b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fefb400bf80 tx=0x7fefb40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e98c44c04dad6e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.511 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefb2000640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:47:04.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.508+0000 7fefb2000640 1 -- 172.21.15.7:0/259594484 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4004020 con 0x55f31ef85480 2024-03-18T01:47:04.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefb2000640 1 -- 172.21.15.7:0/259594484 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fefb40041c0 con 0x55f31ef85480 2024-03-18T01:47:04.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 1 -- 172.21.15.7:0/259594484 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefc0004050 con 0x55f31ef85480 2024-03-18T01:47:04.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefb2000640 1 -- 172.21.15.7:0/259594484 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7fefb4024050 con 0x55f31ef85480 2024-03-18T01:47:04.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 10 client.16095 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:04.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 1 -- 172.21.15.7:0/259594484 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x55f31ecc77d0 con 0x55f31ef85480 2024-03-18T01:47:04.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefb2000640 1 -- 172.21.15.7:0/259594484 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc0004050 con 0x55f31ef85480 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefb2000640 1 -- 172.21.15.7:0/259594484 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7fefb40039c0 con 0x55f31ef85480 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 10 client.16095 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 20 client.16095 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f31ef9ad80 0x55f31ef9d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 4 client.16095 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.509+0000 7fefd1e47740 1 -- 172.21.15.7:0/259594484 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f31eef6990 con 0x55f31ef9ad80 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.510+0000 7fefcd200640 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f31ef9ad80 0x55f31ef9d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.510+0000 7fefcd200640 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f31ef9ad80 0x55f31ef9d1e0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bcf0bda7cea9ad6 server_cookie=3356569985464cd4 in_seq=0 out_seq=0 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.510+0000 7fefb2000640 10 client.16095 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.511+0000 7fefb2000640 1 -- 172.21.15.7:0/259594484 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f31eef6990 con 0x55f31ef9ad80 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.511+0000 7fefb2000640 10 client.16095 handle_command_reply: tid=0 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 client.16095 shutdown 2024-03-18T01:47:04.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 -- 172.21.15.7:0/259594484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 msgr2=0x55f31ef8f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef8f2b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fefb400bf80 tx=0x7fefb40118d0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 -- 172.21.15.7:0/259594484 shutdown_connections 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f31ef9ad80 0x55f31ef9d1e0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef8a5f0 0x55f31ef962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef85480 0x55f31ef8f2b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 --2- 172.21.15.7:0/259594484 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef8ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 -- 172.21.15.7:0/259594484 >> 172.21.15.7:0/259594484 conn(0x55f31ef80000 msgr2=0x55f31ef7f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefb2000640 0 client.16095 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 -- 172.21.15.7:0/259594484 shutdown_connections 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 1 -- 172.21.15.7:0/259594484 wait complete. 2024-03-18T01:47:04.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.512+0000 7fefd1e47740 20 client.16095 trim_cache size 0 max 16384 2024-03-18T01:47:04.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:04 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1355654152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:47:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.528+0000 7fefd1e47740 1 Processor -- start 2024-03-18T01:47:04.529 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.528+0000 7fefd1e47740 1 -- start start 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef868a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef86de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef8a5f0 0x55f31ef962b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefd1e47740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81b0670 con 0x55f31ef8a5f0 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefd1e47740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81b22d0 con 0x55f31ef85480 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefd1e47740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81b0d70 con 0x55f31ef84a90 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefc7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef86de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefc7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef86de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59900/0 (socket says 172.21.15.7:59900) 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefc7400640 1 -- 172.21.15.7:0/454542315 learned_addr learned my addr 172.21.15.7:0/454542315 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefcd200640 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef868a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefcd200640 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef8a5f0 0x55f31ef962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefc7400640 1 -- 172.21.15.7:0/454542315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 msgr2=0x55f31ef868a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefc7400640 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef868a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.530 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefc7400640 1 -- 172.21.15.7:0/454542315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef8a5f0 msgr2=0x55f31ef962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefc7400640 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef8a5f0 0x55f31ef962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.531 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.529+0000 7fefc7400640 1 -- 172.21.15.7:0/454542315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f31ef80d70 con 0x55f31ef85480 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.530+0000 7fefc7400640 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef86de0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fefb4001eb0 tx=0x7fefb400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc41b8738c653a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.530+0000 7fefbcc00640 1 -- 172.21.15.7:0/454542315 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb401a480 con 0x55f31ef85480 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.530+0000 7fefbcc00640 1 -- 172.21.15.7:0/454542315 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb40035f0 con 0x55f31ef85480 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.530+0000 7fefbcc00640 1 -- 172.21.15.7:0/454542315 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4005310 con 0x55f31ef85480 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 -- 172.21.15.7:0/454542315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 msgr2=0x55f31ef86de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef86de0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fefb4001eb0 tx=0x7fefb400b480 comp rx=0 tx=0).stop 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 -- 172.21.15.7:0/454542315 shutdown_connections 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef8a5f0 0x55f31ef962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef86de0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 --2- 172.21.15.7:0/454542315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef84a90 0x55f31ef868a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 -- 172.21.15.7:0/454542315 >> 172.21.15.7:0/454542315 conn(0x55f31ef80000 msgr2=0x55f31ef803e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.533 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 -- 172.21.15.7:0/454542315 shutdown_connections 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 -- 172.21.15.7:0/454542315 wait complete. 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 -- start start 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.531+0000 7fefd1e47740 1 Processor -- start 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef84a90 0x55f31ef93040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef93580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefd1e47740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef8a5f0 0x55f31efff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefd1e47740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81b1ba0 con 0x55f31ef84a90 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefd1e47740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81b1470 con 0x55f31ef85480 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefd1e47740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fefc81ae650 con 0x55f31ef8a5f0 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefd1e47740 10 client.0 _ll_register_callbacks cb 0x7fefb400bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefc7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef93580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefcd200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef84a90 0x55f31ef93040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefc7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef93580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59910/0 (socket says 172.21.15.7:59910) 2024-03-18T01:47:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefc7400640 1 -- 172.21.15.7:0/3337151146 learned_addr learned my addr 172.21.15.7:0/3337151146 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefcd200640 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef8a5f0 0x55f31efff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefc7400640 1 -- 172.21.15.7:0/3337151146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef8a5f0 msgr2=0x55f31efff0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefc7400640 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef8a5f0 0x55f31efff0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefc7400640 1 -- 172.21.15.7:0/3337151146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef84a90 msgr2=0x55f31ef93040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefc7400640 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef84a90 0x55f31ef93040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.535 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.532+0000 7fefc7400640 1 -- 172.21.15.7:0/3337151146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f31ef93bc0 con 0x55f31ef85480 2024-03-18T01:47:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.533+0000 7fefc7400640 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef93580 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fefb4005ca0 tx=0x7fefb40039d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1873559096200800 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.533+0000 7fefb2000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:47:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.533+0000 7fefb2000640 1 -- 172.21.15.7:0/3337151146 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb401a5c0 con 0x55f31ef85480 2024-03-18T01:47:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.533+0000 7fefd1e47740 1 -- 172.21.15.7:0/3337151146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefb40035f0 con 0x55f31ef85480 2024-03-18T01:47:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.533+0000 7fefb2000640 1 -- 172.21.15.7:0/3337151146 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fefb4004190 con 0x55f31ef85480 2024-03-18T01:47:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.533+0000 7fefb2000640 1 -- 172.21.15.7:0/3337151146 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7fefb40050b0 con 0x55f31ef85480 2024-03-18T01:47:04.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.533+0000 7fefd1e47740 10 client.26423 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.533+0000 7fefd1e47740 1 -- 172.21.15.7:0/3337151146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x55f31f0006c0 con 0x55f31ef85480 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefb2000640 1 -- 172.21.15.7:0/3337151146 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb40035f0 con 0x55f31ef85480 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefb2000640 1 -- 172.21.15.7:0/3337151146 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7fefb40107c0 con 0x55f31ef85480 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefd1e47740 10 client.26423 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefd1e47740 10 client.26423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefd1e47740 20 client.26423 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefd1e47740 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f31f001070 0x55f31f003490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefd1e47740 4 client.26423 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefd1e47740 1 -- 172.21.15.7:0/3337151146 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f31eef0ce0 con 0x55f31f001070 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.534+0000 7fefcd200640 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f31f001070 0x55f31f003490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefcd200640 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f31f001070 0x55f31f003490 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dea054d165c32de8 server_cookie=2acba9da96cce6c in_seq=0 out_seq=0 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefb2000640 10 client.26423 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:04.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefb2000640 1 -- 172.21.15.7:0/3337151146 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f31eef0ce0 con 0x55f31f001070 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefb2000640 10 client.26423 handle_command_reply: tid=0 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 client.26423 shutdown 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 -- 172.21.15.7:0/3337151146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 msgr2=0x55f31ef93580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef93580 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fefb4005ca0 tx=0x7fefb40039d0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 -- 172.21.15.7:0/3337151146 shutdown_connections 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x55f31f001070 0x55f31f003490 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f31ef8a5f0 0x55f31efff0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f31ef85480 0x55f31ef93580 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 --2- 172.21.15.7:0/3337151146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55f31ef84a90 0x55f31ef93040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 -- 172.21.15.7:0/3337151146 >> 172.21.15.7:0/3337151146 conn(0x55f31ef80000 msgr2=0x55f31ef83260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefb2000640 0 client.26423 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 -- 172.21.15.7:0/3337151146 shutdown_connections 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.535+0000 7fefd1e47740 1 -- 172.21.15.7:0/3337151146 wait complete. 2024-03-18T01:47:04.539 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefd1e47740 20 client.26423 trim_cache size 0 max 16384 2024-03-18T01:47:04.540 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:04.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 -- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefac0a2870 msgr2=0x7fefac0a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefac0a2870 0x7fefac0a4d30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fefb802c180 tx=0x7fefb802cf90 comp rx=0 tx=0).stop 2024-03-18T01:47:04.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 -- 172.21.15.7:0/3748193670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 msgr2=0x7fefc819def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 0x7fefc819def0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fefb802c8f0 tx=0x7fefb8005d50 comp rx=0 tx=0).stop 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 -- 172.21.15.7:0/3748193670 shutdown_connections 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fefc8077d80 0x7fefc81a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fefac0a2870 0x7fefac0a4d30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fefc80773e0 0x7fefc819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 --2- 172.21.15.7:0/3748193670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fefc80761a0 0x7fefc819def0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 -- 172.21.15.7:0/3748193670 >> 172.21.15.7:0/3748193670 conn(0x7fefc80fed80 msgr2=0x7fefc8100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.536+0000 7fefcdc00640 1 -- 172.21.15.7:0/3748193670 shutdown_connections 2024-03-18T01:47:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:04.537+0000 7fefcdc00640 1 -- 172.21.15.7:0/3748193670 wait complete. 2024-03-18T01:47:05.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:05 smithi007 ceph-mon[31996]: pgmap v684: 129 pgs: 129 active+clean; 63 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 71 KiB/s wr, 270 op/s 2024-03-18T01:47:05.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:05 smithi203 ceph-mon[36345]: pgmap v684: 129 pgs: 129 active+clean; 63 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 71 KiB/s wr, 270 op/s 2024-03-18T01:47:05.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:05 smithi046 ceph-mon[36108]: pgmap v684: 129 pgs: 129 active+clean; 63 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 71 KiB/s wr, 270 op/s 2024-03-18T01:47:05.555 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T01:47:05.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:47:05.555 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T01:47:05.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.786+0000 7f844c800640 1 -- 172.21.15.7:0/11054164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84441022e0 msgr2=0x7f84441046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.786+0000 7f844c800640 1 --2- 172.21.15.7:0/11054164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84441022e0 0x7f84441046d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8438009900 tx=0x7f843802bc30 comp rx=0 tx=0).stop 2024-03-18T01:47:05.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.786+0000 7f844c800640 1 -- 172.21.15.7:0/11054164 shutdown_connections 2024-03-18T01:47:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.786+0000 7f844c800640 1 --2- 172.21.15.7:0/11054164 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444107540 0x7f8444109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.786+0000 7f844c800640 1 --2- 172.21.15.7:0/11054164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8444104c10 0x7f8444107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.786+0000 7f844c800640 1 --2- 172.21.15.7:0/11054164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84441022e0 0x7f84441046d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.786+0000 7f844c800640 1 -- 172.21.15.7:0/11054164 >> 172.21.15.7:0/11054164 conn(0x7f84440fbc70 msgr2=0x7f84440fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.787+0000 7f844c800640 1 -- 172.21.15.7:0/11054164 shutdown_connections 2024-03-18T01:47:05.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.787+0000 7f844c800640 1 -- 172.21.15.7:0/11054164 wait complete. 2024-03-18T01:47:05.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f844c800640 1 Processor -- start 2024-03-18T01:47:05.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f844c800640 1 -- start start 2024-03-18T01:47:05.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f844c800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84441022e0 0x7f8444199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f844c800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444104c10 0x7f844419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f844c800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8444107540 0x7f84441a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f844c800640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f844410d4f0 con 0x7f8444104c10 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f844c800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f844410d370 con 0x7f84441022e0 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f844c800640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f844410d670 con 0x7f8444107540 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f8449e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84441022e0 0x7f8444199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f8449e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84441022e0 0x7f8444199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:59926/0 (socket says 172.21.15.7:59926) 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.788+0000 7f8449e00640 1 -- 172.21.15.7:0/1325394281 learned_addr learned my addr 172.21.15.7:0/1325394281 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f844a800640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8444107540 0x7f84441a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f8449400640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444104c10 0x7f844419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f8449e00640 1 -- 172.21.15.7:0/1325394281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8444107540 msgr2=0x7f84441a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f8449e00640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8444107540 0x7f84441a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f8449e00640 1 -- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444104c10 msgr2=0x7f844419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f8449e00640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444104c10 0x7f844419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f8449e00640 1 -- 172.21.15.7:0/1325394281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84441a1900 con 0x7f84441022e0 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f844a800640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8444107540 0x7f84441a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:05.791 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.789+0000 7f8449e00640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84441022e0 0x7f8444199cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8438005ab0 tx=0x7f843802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.790+0000 7f843ea00640 1 -- 172.21.15.7:0/1325394281 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438043020 con 0x7f84441022e0 2024-03-18T01:47:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.790+0000 7f843ea00640 1 -- 172.21.15.7:0/1325394281 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84380054e0 con 0x7f84441022e0 2024-03-18T01:47:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.790+0000 7f844c800640 1 -- 172.21.15.7:0/1325394281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84441a1b90 con 0x7f84441022e0 2024-03-18T01:47:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.790+0000 7f843ea00640 1 -- 172.21.15.7:0/1325394281 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438003ce0 con 0x7f84441022e0 2024-03-18T01:47:05.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.790+0000 7f844c800640 1 -- 172.21.15.7:0/1325394281 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84441a20e0 con 0x7f84441022e0 2024-03-18T01:47:05.793 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.793+0000 7f844f567740 1 Processor -- start 2024-03-18T01:47:05.794 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.793+0000 7f844f567740 1 -- start start 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844f567740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f11dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844f567740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f11e1d0 0x558c5f122cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844f567740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f123340 0x558c5f1257d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844f567740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558c5f080f80 con 0x558c5f123340 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844f567740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558c5f029ad0 con 0x558c5f11e1d0 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844f567740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c5f07d020 con 0x558c5f11d7e0 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f11dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f11dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51504/0 (socket says 172.21.15.7:51504) 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844a800640 1 -- 172.21.15.7:0/3975505833 learned_addr learned my addr 172.21.15.7:0/3975505833 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f8449400640 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f123340 0x558c5f1257d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f843ea00640 1 -- 172.21.15.7:0/1325394281 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f8438046020 con 0x7f84441022e0 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.794+0000 7f844a800640 1 -- 172.21.15.7:0/3975505833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f11e1d0 msgr2=0x558c5f122cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:47:05.796 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f844a800640 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f11e1d0 0x558c5f122cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f844a800640 1 -- 172.21.15.7:0/3975505833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f123340 msgr2=0x558c5f1257d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f844a800640 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f123340 0x558c5f1257d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f844a800640 1 -- 172.21.15.7:0/3975505833 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c5f0416f0 con 0x558c5f11d7e0 2024-03-18T01:47:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f8449400640 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f123340 0x558c5f1257d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f844a800640 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f11dc00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8440007f40 tx=0x7f844000e7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a227e5a4ca0e4292 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f8433000640 1 -- 172.21.15.7:0/3975505833 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440014070 con 0x558c5f11d7e0 2024-03-18T01:47:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f8433000640 1 -- 172.21.15.7:0/3975505833 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8440004d20 con 0x558c5f11d7e0 2024-03-18T01:47:05.797 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.795+0000 7f8433000640 1 -- 172.21.15.7:0/3975505833 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440005590 con 0x558c5f11d7e0 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.796+0000 7f843ea00640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f842c0a2800 0x7f842c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.796+0000 7f844f567740 1 -- 172.21.15.7:0/3975505833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 msgr2=0x558c5f11dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.796+0000 7f844f567740 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f11dc00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8440007f40 tx=0x7f844000e7e0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.796+0000 7f843ea00640 1 -- 172.21.15.7:0/1325394281 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f84380f10b0 con 0x7f84441022e0 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.796+0000 7f844f567740 1 -- 172.21.15.7:0/3975505833 shutdown_connections 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f844f567740 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f123340 0x558c5f1257d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f844f567740 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f11e1d0 0x558c5f122cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f844f567740 1 --2- 172.21.15.7:0/3975505833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f11dc00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f844f567740 1 -- 172.21.15.7:0/3975505833 >> 172.21.15.7:0/3975505833 conn(0x558c5f0e5930 msgr2=0x558c5f119150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f8449e00640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f842c0a2800 0x7f842c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f844f567740 1 -- 172.21.15.7:0/3975505833 shutdown_connections 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f844f567740 1 -- 172.21.15.7:0/3975505833 wait complete. 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f8449e00640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f842c0a2800 0x7f842c0a4cc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f84380bd390 tx=0x7f8438009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.797+0000 7f844f567740 1 -- start start 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.798+0000 7f844f567740 1 Processor -- start 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.798+0000 7f844f567740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f127b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.798+0000 7f844f567740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f1280c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.798+0000 7f844f567740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.798+0000 7f844f567740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558c5f1262f0 con 0x558c5f11d7e0 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.798+0000 7f844f567740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558c5f03cf30 con 0x558c5f123340 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.798+0000 7f844f567740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558c5f126470 con 0x558c5f11e1d0 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.798+0000 7f844f567740 10 client.0 _ll_register_callbacks cb 0x558c5f02abc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:05.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f1280c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f1280c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51506/0 (socket says 172.21.15.7:51506) 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 -- 172.21.15.7:0/3838318601 learned_addr learned my addr 172.21.15.7:0/3838318601 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f844a800640 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f844a800640 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f127b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 -- 172.21.15.7:0/3838318601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 msgr2=0x558c5f12f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 -- 172.21.15.7:0/3838318601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 msgr2=0x558c5f127b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f127b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 -- 172.21.15.7:0/3838318601 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c5ee5d7e0 con 0x558c5f11e1d0 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f844a800640 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f127b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8449400640 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f1280c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8434013790 tx=0x7f8434013c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ea784bbd5706a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.799+0000 7f8427400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.800+0000 7f8427400640 1 -- 172.21.15.7:0/3838318601 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8434011d70 con 0x558c5f11e1d0 2024-03-18T01:47:05.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.800+0000 7f8427400640 1 -- 172.21.15.7:0/3838318601 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8434004030 con 0x558c5f11e1d0 2024-03-18T01:47:05.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.800+0000 7f844f567740 1 -- 172.21.15.7:0/3838318601 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8440004d20 con 0x558c5f11e1d0 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.800+0000 7f8427400640 1 -- 172.21.15.7:0/3838318601 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f8434004230 con 0x558c5f11e1d0 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.800+0000 7f844f567740 10 client.26416 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.800+0000 7f844f567740 1 -- 172.21.15.7:0/3838318601 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x558c5eebdf70 con 0x558c5f11e1d0 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.800+0000 7f8427400640 1 -- 172.21.15.7:0/3838318601 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8440004d20 con 0x558c5f11e1d0 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.801+0000 7f8427400640 1 -- 172.21.15.7:0/3838318601 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f843401a070 con 0x558c5f11e1d0 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.801+0000 7f844f567740 10 client.26416 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.801+0000 7f844f567740 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.801+0000 7f844f567740 20 client.26416 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.801+0000 7f844f567740 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558c5f133b10 0x558c5f135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.801+0000 7f844f567740 4 client.26416 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.801+0000 7f844f567740 1 -- 172.21.15.7:0/3838318601 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c5f08e4c0 con 0x558c5f133b10 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.801+0000 7f844a800640 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558c5f133b10 0x558c5f135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.802+0000 7f844a800640 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558c5f133b10 0x558c5f135f70 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc3151e2632c0553 server_cookie=3ac3ba7224fd3c2d in_seq=0 out_seq=0 2024-03-18T01:47:05.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.802+0000 7f8427400640 10 client.26416 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.802+0000 7f8427400640 1 -- 172.21.15.7:0/3838318601 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558c5f08e4c0 con 0x558c5f133b10 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f8427400640 10 client.26416 handle_command_reply: tid=0 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 client.26416 shutdown 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 -- 172.21.15.7:0/3838318601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 msgr2=0x558c5f1280c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f1280c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8434013790 tx=0x7f8434013c50 comp rx=0 tx=0).stop 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 -- 172.21.15.7:0/3838318601 shutdown_connections 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558c5f133b10 0x558c5f135f70 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f8427400640 0 client.26416 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f1280c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 --2- 172.21.15.7:0/3838318601 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f127b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 -- 172.21.15.7:0/3838318601 >> 172.21.15.7:0/3838318601 conn(0x558c5f0e5930 msgr2=0x558c5f118350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 -- 172.21.15.7:0/3838318601 shutdown_connections 2024-03-18T01:47:05.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 1 -- 172.21.15.7:0/3838318601 wait complete. 2024-03-18T01:47:05.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.803+0000 7f844f567740 20 client.26416 trim_cache size 0 max 16384 2024-03-18T01:47:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.823+0000 7f844f567740 1 Processor -- start 2024-03-18T01:47:05.823 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.823+0000 7f844f567740 1 -- start start 2024-03-18T01:47:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.823+0000 7f844f567740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f1270d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.823+0000 7f844f567740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11e1d0 0x558c5f12f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.823+0000 7f844f567740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f11caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.823+0000 7f844f567740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f84441ac4e0 con 0x558c5f11e1d0 2024-03-18T01:47:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.823+0000 7f844f567740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84441ae140 con 0x558c5f123340 2024-03-18T01:47:05.824 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.823+0000 7f844f567740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f84441acbe0 con 0x558c5f11d7e0 2024-03-18T01:47:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f844a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f1270d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f8449400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11e1d0 0x558c5f12f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f844a800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f1270d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51522/0 (socket says 172.21.15.7:51522) 2024-03-18T01:47:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f844a800640 1 -- 172.21.15.7:0/1470131973 learned_addr learned my addr 172.21.15.7:0/1470131973 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f844a800640 1 --2- 172.21.15.7:0/1470131973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f11caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f8449400640 1 -- 172.21.15.7:0/1470131973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 msgr2=0x558c5f1270d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:05.825 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f8449400640 1 --2- 172.21.15.7:0/1470131973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f1270d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f8449400640 1 -- 172.21.15.7:0/1470131973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 msgr2=0x558c5f11caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f8449400640 1 --2- 172.21.15.7:0/1470131973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f11caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f8449400640 1 -- 172.21.15.7:0/1470131973 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c5ed55170 con 0x558c5f11e1d0 2024-03-18T01:47:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.824+0000 7f8449400640 1 --2- 172.21.15.7:0/1470131973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11e1d0 0x558c5f12f0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f843400f710 tx=0x7f8434005980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d431a804c209da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.825+0000 7f8432600640 1 -- 172.21.15.7:0/1470131973 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843402d020 con 0x558c5f11e1d0 2024-03-18T01:47:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.825+0000 7f8432600640 1 -- 172.21.15.7:0/1470131973 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f843400fcd0 con 0x558c5f11e1d0 2024-03-18T01:47:05.826 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.825+0000 7f8432600640 1 -- 172.21.15.7:0/1470131973 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8434011df0 con 0x558c5f11e1d0 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 -- 172.21.15.7:0/1470131973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11e1d0 msgr2=0x558c5f12f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 --2- 172.21.15.7:0/1470131973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11e1d0 0x558c5f12f0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f843400f710 tx=0x7f8434005980 comp rx=0 tx=0).stop 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 -- 172.21.15.7:0/1470131973 shutdown_connections 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 --2- 172.21.15.7:0/1470131973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f11caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 --2- 172.21.15.7:0/1470131973 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11e1d0 0x558c5f12f0c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 --2- 172.21.15.7:0/1470131973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11d7e0 0x558c5f1270d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 -- 172.21.15.7:0/1470131973 >> 172.21.15.7:0/1470131973 conn(0x558c5f0e5930 msgr2=0x558c5f11c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 -- 172.21.15.7:0/1470131973 shutdown_connections 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 -- 172.21.15.7:0/1470131973 wait complete. 2024-03-18T01:47:05.829 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 -- start start 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.826+0000 7f844f567740 1 Processor -- start 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844f567740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f12d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844f567740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f12d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844f567740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844f567740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f84441ada10 con 0x558c5f11d7e0 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844f567740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f84441ad2e0 con 0x558c5f123340 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844f567740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f84441aa4c0 con 0x558c5f11e1d0 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844f567740 10 client.0 _ll_register_callbacks cb 0x7f8434009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f8449400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f12d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f8449400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f12d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51524/0 (socket says 172.21.15.7:51524) 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f8449400640 1 -- 172.21.15.7:0/655016238 learned_addr learned my addr 172.21.15.7:0/655016238 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844a800640 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f12d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.827+0000 7f844a800640 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8449400640 1 -- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 msgr2=0x558c5f12e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:05.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8449400640 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8449400640 1 -- 172.21.15.7:0/655016238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 msgr2=0x558c5f12d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8449400640 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f12d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8449400640 1 -- 172.21.15.7:0/655016238 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c5f12ebe0 con 0x558c5f11e1d0 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f844a800640 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8449400640 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f12d5e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f843400f880 tx=0x7f8434005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de6a9299d3d6a326 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8427400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8427400640 1 -- 172.21.15.7:0/655016238 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8434038060 con 0x558c5f11e1d0 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8427400640 1 -- 172.21.15.7:0/655016238 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8434011d70 con 0x558c5f11e1d0 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f844f567740 1 -- 172.21.15.7:0/655016238 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f843400fcd0 con 0x558c5f11e1d0 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f8427400640 1 -- 172.21.15.7:0/655016238 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f843402d070 con 0x558c5f11e1d0 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f844f567740 10 client.26422 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.828+0000 7f844f567740 1 -- 172.21.15.7:0/655016238 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x558c5f19a790 con 0x558c5f11e1d0 2024-03-18T01:47:05.831 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f8427400640 1 -- 172.21.15.7:0/655016238 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f843400fcd0 con 0x558c5f11e1d0 2024-03-18T01:47:05.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f8427400640 1 -- 172.21.15.7:0/655016238 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f843401a040 con 0x558c5f11e1d0 2024-03-18T01:47:05.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f844f567740 10 client.26422 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:05.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f844f567740 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:05.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f844f567740 20 client.26422 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:05.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f844f567740 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558c5f19aff0 0x558c5f19d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:05.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f844f567740 4 client.26422 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T01:47:05.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f844f567740 1 -- 172.21.15.7:0/655016238 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558c5f088810 con 0x558c5f19aff0 2024-03-18T01:47:05.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.829+0000 7f844a800640 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558c5f19aff0 0x558c5f19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.830+0000 7f8427400640 10 client.26422 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.830+0000 7f844a800640 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558c5f19aff0 0x558c5f19d410 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6a628e072f60ea6 server_cookie=162644ecfc558be1 in_seq=0 out_seq=0 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.831+0000 7f8427400640 1 -- 172.21.15.7:0/655016238 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558c5f088810 con 0x558c5f19aff0 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.831+0000 7f8427400640 10 client.26422 handle_command_reply: tid=0 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.831+0000 7f844f567740 1 client.26422 shutdown 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.831+0000 7f844f567740 1 -- 172.21.15.7:0/655016238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 msgr2=0x558c5f12d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.831+0000 7f844f567740 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f12d5e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f843400f880 tx=0x7f8434005f80 comp rx=0 tx=0).stop 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.831+0000 7f844f567740 1 -- 172.21.15.7:0/655016238 shutdown_connections 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.831+0000 7f844f567740 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558c5f19aff0 0x558c5f19d410 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.832+0000 7f844f567740 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558c5f123340 0x558c5f12e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.832+0000 7f844f567740 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558c5f11e1d0 0x558c5f12d5e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.832+0000 7f844f567740 1 --2- 172.21.15.7:0/655016238 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558c5f11d7e0 0x558c5f12d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.832+0000 7f844f567740 1 -- 172.21.15.7:0/655016238 >> 172.21.15.7:0/655016238 conn(0x558c5f0e5930 msgr2=0x558c5f11cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.832+0000 7f8427400640 0 client.26422 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.832+0000 7f844f567740 1 -- 172.21.15.7:0/655016238 shutdown_connections 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.832+0000 7f844f567740 1 -- 172.21.15.7:0/655016238 wait complete. 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.832+0000 7f844f567740 20 client.26422 trim_cache size 0 max 16384 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stdout: "return_code": 0, 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_tag": "548120cb-361b-4627-9b61-9b6b5faf1c7d", 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": "asynchronous" 2024-03-18T01:47:05.835 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:05.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 -- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f842c0a2800 msgr2=0x7f842c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f842c0a2800 0x7f842c0a4cc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f84380bd390 tx=0x7f8438009260 comp rx=0 tx=0).stop 2024-03-18T01:47:05.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 -- 172.21.15.7:0/1325394281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84441022e0 msgr2=0x7f8444199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:05.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84441022e0 0x7f8444199cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8438005ab0 tx=0x7f843802c6c0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 -- 172.21.15.7:0/1325394281 shutdown_connections 2024-03-18T01:47:05.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8444107540 0x7f84441a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.836 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f842c0a2800 0x7f842c0a4cc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8444104c10 0x7f844419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 --2- 172.21.15.7:0/1325394281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84441022e0 0x7f8444199cc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 -- 172.21.15.7:0/1325394281 >> 172.21.15.7:0/1325394281 conn(0x7f84440fbc70 msgr2=0x7f8444101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 -- 172.21.15.7:0/1325394281 shutdown_connections 2024-03-18T01:47:05.837 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:05.833+0000 7f844c800640 1 -- 172.21.15.7:0/1325394281 wait complete. 2024-03-18T01:47:05.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:47:05.857 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 -- 172.21.15.7:0/1510849840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61f40761a0 msgr2=0x7f61f40765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 --2- 172.21.15.7:0/1510849840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61f40761a0 0x7f61f40765c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009900 tx=0x7f61e802bc40 comp rx=0 tx=0).stop 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 -- 172.21.15.7:0/1510849840 shutdown_connections 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 --2- 172.21.15.7:0/1510849840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f4077d80 0x7f61f4078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 --2- 172.21.15.7:0/1510849840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f40773e0 0x7f61f4077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 --2- 172.21.15.7:0/1510849840 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61f40761a0 0x7f61f40765c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 -- 172.21.15.7:0/1510849840 >> 172.21.15.7:0/1510849840 conn(0x7f61f40fef90 msgr2=0x7f61f41013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 -- 172.21.15.7:0/1510849840 shutdown_connections 2024-03-18T01:47:06.076 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.075+0000 7f61fb600640 1 -- 172.21.15.7:0/1510849840 wait complete. 2024-03-18T01:47:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.076+0000 7f61fb600640 1 Processor -- start 2024-03-18T01:47:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.076+0000 7f61fb600640 1 -- start start 2024-03-18T01:47:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61fb600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f40761a0 0x7f61f419e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61fb600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61f40773e0 0x7f61f419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61fb600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f4077d80 0x7f61f41a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61fb600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f61f4113ab0 con 0x7f61f40773e0 2024-03-18T01:47:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61fb600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61f4113930 con 0x7f61f4077d80 2024-03-18T01:47:06.078 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61fb600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61f4113c30 con 0x7f61f40761a0 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f40761a0 0x7f61f419e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f40761a0 0x7f61f419e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53378/0 (socket says 172.21.15.7:53378) 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f8c00640 1 -- 172.21.15.7:0/3977734889 learned_addr learned my addr 172.21.15.7:0/3977734889 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f3e00640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61f40773e0 0x7f61f419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f9600640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f4077d80 0x7f61f41a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f8c00640 1 -- 172.21.15.7:0/3977734889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f4077d80 msgr2=0x7f61f41a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f8c00640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f4077d80 0x7f61f41a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f8c00640 1 -- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61f40773e0 msgr2=0x7f61f419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f8c00640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61f40773e0 0x7f61f419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.077+0000 7f61f8c00640 1 -- 172.21.15.7:0/3977734889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f41a5d80 con 0x7f61f40761a0 2024-03-18T01:47:06.079 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.078+0000 7f61f8c00640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f40761a0 0x7f61f419e160 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009230 tx=0x7f61e802c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.078+0000 7f61f1600640 1 -- 172.21.15.7:0/3977734889 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8041070 con 0x7f61f40761a0 2024-03-18T01:47:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.078+0000 7f61f1600640 1 -- 172.21.15.7:0/3977734889 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e8004450 con 0x7f61f40761a0 2024-03-18T01:47:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.078+0000 7f61f1600640 1 -- 172.21.15.7:0/3977734889 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8003e00 con 0x7f61f40761a0 2024-03-18T01:47:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.078+0000 7f61fb600640 1 -- 172.21.15.7:0/3977734889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f41a6010 con 0x7f61f40761a0 2024-03-18T01:47:06.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.078+0000 7f61fb600640 1 -- 172.21.15.7:0/3977734889 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f41a6550 con 0x7f61f40761a0 2024-03-18T01:47:06.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.081+0000 7f61f1600640 1 -- 172.21.15.7:0/3977734889 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f61e8044070 con 0x7f61f40761a0 2024-03-18T01:47:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.082+0000 7f61f1600640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61dc0a2880 0x7f61dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.083+0000 7f61f1600640 1 -- 172.21.15.7:0/3977734889 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f61e80f0980 con 0x7f61f40761a0 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.083+0000 7f61f3e00640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61dc0a2880 0x7f61dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.083+0000 7f61fe3b2740 1 Processor -- start 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.083+0000 7f61fe3b2740 1 -- start start 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355abc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 0x558d355b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558d355101f0 con 0x558d355ac200 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d354b8a80 con 0x558d355b1370 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558d3550c290 con 0x558d355ab810 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f3e00640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61dc0a2880 0x7f61dc0a4d40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f61f419f670 tx=0x7f61e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:06.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355abc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53388/0 (socket says 172.21.15.7:53388) 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f8c00640 1 -- 172.21.15.7:0/664079173 learned_addr learned my addr 172.21.15.7:0/664079173 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f3e00640 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f9600640 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 0x558d355b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f8c00640 1 -- 172.21.15.7:0/664079173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 msgr2=0x558d355b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f8c00640 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 0x558d355b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f8c00640 1 -- 172.21.15.7:0/664079173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 msgr2=0x558d355b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f8c00640 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.084+0000 7f61f8c00640 1 -- 172.21.15.7:0/664079173 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d352ee7d0 con 0x558d355ab810 2024-03-18T01:47:06.086 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.085+0000 7f61f8c00640 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355abc30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f61e80bcd10 tx=0x7f61e802bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f583d74b6f4bac4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:06.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.085+0000 7f61e5c00640 1 -- 172.21.15.7:0/664079173 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e804b070 con 0x558d355ab810 2024-03-18T01:47:06.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.085+0000 7f61e5c00640 1 -- 172.21.15.7:0/664079173 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e80c10b0 con 0x558d355ab810 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.085+0000 7f61e5c00640 1 -- 172.21.15.7:0/664079173 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e80c5270 con 0x558d355ab810 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 -- 172.21.15.7:0/664079173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 msgr2=0x558d355abc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355abc30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f61e80bcd10 tx=0x7f61e802bc40 comp rx=0 tx=0).stop 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 -- 172.21.15.7:0/664079173 shutdown_connections 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 0x558d355b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/664079173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355abc30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 -- 172.21.15.7:0/664079173 >> 172.21.15.7:0/664079173 conn(0x558d355a6d80 msgr2=0x558d355a7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 -- 172.21.15.7:0/664079173 shutdown_connections 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.086+0000 7f61fe3b2740 1 -- 172.21.15.7:0/664079173 wait complete. 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.087+0000 7f61fe3b2740 1 -- start start 2024-03-18T01:47:06.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.087+0000 7f61fe3b2740 1 Processor -- start 2024-03-18T01:47:06.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ab810 0x558d355b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ac200 0x558d355b6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355bd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x558d355b40e0 con 0x558d355ab810 2024-03-18T01:47:06.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d354cbed0 con 0x558d355ac200 2024-03-18T01:47:06.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558d355b4260 con 0x558d355b1370 2024-03-18T01:47:06.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61fe3b2740 10 client.0 _ll_register_callbacks cb 0x558d35507220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:06.091 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61f9600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355bd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61f9600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355bd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53390/0 (socket says 172.21.15.7:53390) 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61f9600640 1 -- 172.21.15.7:0/3072317744 learned_addr learned my addr 172.21.15.7:0/3072317744 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61f3e00640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ac200 0x558d355b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.088+0000 7f61f8c00640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ab810 0x558d355b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61f9600640 1 -- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ac200 msgr2=0x558d355b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61f9600640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ac200 0x558d355b6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61f9600640 1 -- 172.21.15.7:0/3072317744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ab810 msgr2=0x558d355b5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61f9600640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ab810 0x558d355b5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61f9600640 1 -- 172.21.15.7:0/3072317744 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d352243f0 con 0x558d355b1370 2024-03-18T01:47:06.092 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61f8c00640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ab810 0x558d355b5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61f3e00640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ac200 0x558d355b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61f9600640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355bd080 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f61ec012d20 tx=0x7f61ec00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee07dccad44579fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61da000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61da000640 1 -- 172.21.15.7:0/3072317744 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec00fe00 con 0x558d355b1370 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61da000640 1 -- 172.21.15.7:0/3072317744 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f61ec012110 con 0x558d355b1370 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.089+0000 7f61fe3b2740 1 -- 172.21.15.7:0/3072317744 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61e80c10b0 con 0x558d355b1370 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61da000640 1 -- 172.21.15.7:0/3072317744 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f61ec024050 con 0x558d355b1370 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61fe3b2740 10 client.26434 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61fe3b2740 1 -- 172.21.15.7:0/3072317744 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x558d351e40f0 con 0x558d355b1370 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61da000640 1 -- 172.21.15.7:0/3072317744 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e80c10b0 con 0x558d355b1370 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61da000640 1 -- 172.21.15.7:0/3072317744 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f61ec010430 con 0x558d355b1370 2024-03-18T01:47:06.093 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61fe3b2740 10 client.26434 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:06.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61fe3b2740 10 client.26434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:06.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61fe3b2740 20 client.26434 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:06.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558d355c1a20 0x558d355c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61fe3b2740 4 client.26434 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:06.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.090+0000 7f61fe3b2740 1 -- 172.21.15.7:0/3072317744 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d35523be0 con 0x558d355c1a20 2024-03-18T01:47:06.094 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.091+0000 7f61f8c00640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558d355c1a20 0x558d355c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.091+0000 7f61f8c00640 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558d355c1a20 0x558d355c3ec0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5d61f2721d9c493 server_cookie=2df7361f4a28760e in_seq=0 out_seq=0 2024-03-18T01:47:06.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.091+0000 7f61da000640 10 client.26434 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:06.095 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61da000640 1 -- 172.21.15.7:0/3072317744 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558d35523be0 con 0x558d355c1a20 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61da000640 10 client.26434 handle_command_reply: tid=0 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 client.26434 shutdown 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 -- 172.21.15.7:0/3072317744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 msgr2=0x558d355bd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355bd080 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f61ec012d20 tx=0x7f61ec00f790 comp rx=0 tx=0).stop 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 -- 172.21.15.7:0/3072317744 shutdown_connections 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558d355c1a20 0x558d355c3ec0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355bd080 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ac200 0x558d355b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/3072317744 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ab810 0x558d355b5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61da000640 0 client.26434 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.092+0000 7f61fe3b2740 1 -- 172.21.15.7:0/3072317744 >> 172.21.15.7:0/3072317744 conn(0x558d355a6d80 msgr2=0x558d355a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.093+0000 7f61fe3b2740 1 -- 172.21.15.7:0/3072317744 shutdown_connections 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.093+0000 7f61fe3b2740 1 -- 172.21.15.7:0/3072317744 wait complete. 2024-03-18T01:47:06.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.093+0000 7f61fe3b2740 20 client.26434 trim_cache size 0 max 16384 2024-03-18T01:47:06.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.120+0000 7f61fe3b2740 1 Processor -- start 2024-03-18T01:47:06.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.120+0000 7f61fe3b2740 1 -- start start 2024-03-18T01:47:06.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.120+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ab810 0x558d355b4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.121 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355bd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355b5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b0910 con 0x558d355ac200 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b2570 con 0x558d355ab810 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b1010 con 0x558d355b1370 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f8c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ab810 0x558d355b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f9600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355b5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f9600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355b5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53400/0 (socket says 172.21.15.7:53400) 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f8c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ab810 0x558d355b4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:42360/0 (socket says 172.21.15.7:42360) 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f3e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355bd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.122 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f9600640 1 -- 172.21.15.7:0/1232272608 learned_addr learned my addr 172.21.15.7:0/1232272608 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f9600640 1 -- 172.21.15.7:0/1232272608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ab810 msgr2=0x558d355b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f9600640 1 --2- 172.21.15.7:0/1232272608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ab810 0x558d355b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f9600640 1 -- 172.21.15.7:0/1232272608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 msgr2=0x558d355bd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f9600640 1 --2- 172.21.15.7:0/1232272608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355bd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f9600640 1 -- 172.21.15.7:0/1232272608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d355a27f0 con 0x558d355b1370 2024-03-18T01:47:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.121+0000 7f61f8c00640 1 --2- 172.21.15.7:0/1232272608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ab810 0x558d355b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.122+0000 7f61f9600640 1 --2- 172.21.15.7:0/1232272608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355b5b00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f61ec002230 tx=0x7f61ec012910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=972f8765060838d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:06.124 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.122+0000 7f61e5200640 1 -- 172.21.15.7:0/1232272608 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec02d070 con 0x558d355b1370 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.122+0000 7f61e5200640 1 -- 172.21.15.7:0/1232272608 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ec021040 con 0x558d355b1370 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.122+0000 7f61e5200640 1 -- 172.21.15.7:0/1232272608 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec010430 con 0x558d355b1370 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1232272608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 msgr2=0x558d355b5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1232272608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355b5b00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f61ec002230 tx=0x7f61ec012910 comp rx=0 tx=0).stop 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1232272608 shutdown_connections 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1232272608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355b1370 0x558d355b5b00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1232272608 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355bd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1232272608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355ab810 0x558d355b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1232272608 >> 172.21.15.7:0/1232272608 conn(0x558d355a6d80 msgr2=0x558d355ad790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1232272608 shutdown_connections 2024-03-18T01:47:06.125 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.123+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1232272608 wait complete. 2024-03-18T01:47:06.126 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.124+0000 7f61fe3b2740 1 -- start start 2024-03-18T01:47:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.124+0000 7f61fe3b2740 1 Processor -- start 2024-03-18T01:47:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355ba4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355baa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61fe3b2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 0x558d355bba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b1e40 con 0x558d355ac200 2024-03-18T01:47:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41b1710 con 0x558d355b1370 2024-03-18T01:47:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61fe3b2740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f61f41ae8f0 con 0x558d355ab810 2024-03-18T01:47:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61fe3b2740 10 client.0 _ll_register_callbacks cb 0x7f61ec009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355ba4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f8c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355ba4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:53410/0 (socket says 172.21.15.7:53410) 2024-03-18T01:47:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f8c00640 1 -- 172.21.15.7:0/1257720169 learned_addr learned my addr 172.21.15.7:0/1257720169 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:06.130 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f3e00640 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355baa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f9600640 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 0x558d355bba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f8c00640 1 -- 172.21.15.7:0/1257720169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 msgr2=0x558d355bba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f8c00640 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 0x558d355bba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f8c00640 1 -- 172.21.15.7:0/1257720169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 msgr2=0x558d355baa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f8c00640 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355baa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f8c00640 1 -- 172.21.15.7:0/1257720169 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d355bc020 con 0x558d355ab810 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.125+0000 7f61f3e00640 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355baa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61f8c00640 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355ba4e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009430 tx=0x7f61e80b98d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a3f4c381a4e656a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61da000640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61da000640 1 -- 172.21.15.7:0/1257720169 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e804b040 con 0x558d355ab810 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61da000640 1 -- 172.21.15.7:0/1257720169 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f61e80bb310 con 0x558d355ab810 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1257720169 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61ec021040 con 0x558d355ab810 2024-03-18T01:47:06.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61da000640 1 -- 172.21.15.7:0/1257720169 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f61e80bd010 con 0x558d355ab810 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61fe3b2740 10 client.26440 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1257720169 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x558d355bd330 con 0x558d355ab810 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.126+0000 7f61da000640 1 -- 172.21.15.7:0/1257720169 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ec021040 con 0x558d355ab810 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61da000640 1 -- 172.21.15.7:0/1257720169 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f61e80c0070 con 0x558d355ab810 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61fe3b2740 10 client.26440 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61fe3b2740 10 client.26440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61fe3b2740 20 client.26440 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558d35628310 0x558d3562a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61fe3b2740 4 client.26440 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1257720169 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558d3551d730 con 0x558d35628310 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61f3e00640 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558d35628310 0x558d3562a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61f3e00640 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558d35628310 0x558d3562a730 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97d565501689a069 server_cookie=52e195a3e4c98905 in_seq=0 out_seq=0 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.127+0000 7f61da000640 10 client.26440 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:06.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61da000640 1 -- 172.21.15.7:0/1257720169 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558d3551d730 con 0x558d35628310 2024-03-18T01:47:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61da000640 10 client.26440 handle_command_reply: tid=0 2024-03-18T01:47:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 client.26440 shutdown 2024-03-18T01:47:06.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1257720169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 msgr2=0x558d355ba4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355ba4e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009430 tx=0x7f61e80b98d0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1257720169 shutdown_connections 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x558d35628310 0x558d3562a730 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d355b1370 0x558d355bba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x558d355ac200 0x558d355baa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 --2- 172.21.15.7:0/1257720169 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558d355ab810 0x558d355ba4e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61da000640 0 client.26440 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.128+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1257720169 >> 172.21.15.7:0/1257720169 conn(0x558d355a6d80 msgr2=0x558d355b5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:06.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.129+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1257720169 shutdown_connections 2024-03-18T01:47:06.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.129+0000 7f61fe3b2740 1 -- 172.21.15.7:0/1257720169 wait complete. 2024-03-18T01:47:06.135 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.129+0000 7f61fe3b2740 20 client.26440 trim_cache size 0 max 16384 2024-03-18T01:47:06.135 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T01:47:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": { 2024-03-18T01:47:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "548120cb-361b-4627-9b61-9b6b5faf1c7d": { 2024-03-18T01:47:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "path": "/", 2024-03-18T01:47:06.135 INFO:teuthology.orchestra.run.smithi007.stdout: "tag": "548120cb-361b-4627-9b61-9b6b5faf1c7d", 2024-03-18T01:47:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: "options": "recursive,force" 2024-03-18T01:47:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:06.136 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:06.136 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:06.136 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 -- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61dc0a2880 msgr2=0x7f61dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61dc0a2880 0x7f61dc0a4d40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f61f419f670 tx=0x7f61e0005f90 comp rx=0 tx=0).stop 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 -- 172.21.15.7:0/3977734889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f40761a0 msgr2=0x7f61f419e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f40761a0 0x7f61f419e160 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009230 tx=0x7f61e802c660 comp rx=0 tx=0).stop 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 -- 172.21.15.7:0/3977734889 shutdown_connections 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61f4077d80 0x7f61f41a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f61dc0a2880 0x7f61dc0a4d40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f61f40773e0 0x7f61f419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 --2- 172.21.15.7:0/3977734889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61f40761a0 0x7f61f419e160 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 -- 172.21.15.7:0/3977734889 >> 172.21.15.7:0/3977734889 conn(0x7f61f40fef90 msgr2=0x7f61f4100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 -- 172.21.15.7:0/3977734889 shutdown_connections 2024-03-18T01:47:06.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:06.130+0000 7f61fb600640 1 -- 172.21.15.7:0/3977734889 wait complete. 2024-03-18T01:47:06.157 INFO:tasks.cephfs.filesystem:scrub status for tag:548120cb-361b-4627-9b61-9b6b5faf1c7d - {'path': '/', 'tag': '548120cb-361b-4627-9b61-9b6b5faf1c7d', 'options': 'recursive,force'} 2024-03-18T01:47:06.687 DEBUG:teuthology.parallel:result is None 2024-03-18T01:47:06.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T01:47:06.812 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T01:47:06.813 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-18T01:47:06.825 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-18T01:47:07.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:07 smithi007 ceph-mon[31996]: pgmap v685: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 102 KiB/s wr, 239 op/s 2024-03-18T01:47:07.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:07 smithi007 ceph-mon[31996]: scrub queued for path: / 2024-03-18T01:47:07.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:07 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:47:07.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:07 smithi007 ceph-mon[31996]: scrub summary: active paths [/] 2024-03-18T01:47:07.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:07 smithi007 ceph-mon[31996]: scrub summary: idle+waiting paths [/] 2024-03-18T01:47:07.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:07 smithi203 ceph-mon[36345]: pgmap v685: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 102 KiB/s wr, 239 op/s 2024-03-18T01:47:07.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:07 smithi203 ceph-mon[36345]: scrub queued for path: / 2024-03-18T01:47:07.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:07 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:47:07.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:07 smithi203 ceph-mon[36345]: scrub summary: active paths [/] 2024-03-18T01:47:07.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:07 smithi203 ceph-mon[36345]: scrub summary: idle+waiting paths [/] 2024-03-18T01:47:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:07 smithi046 ceph-mon[36108]: pgmap v685: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 102 KiB/s wr, 239 op/s 2024-03-18T01:47:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:07 smithi046 ceph-mon[36108]: scrub queued for path: / 2024-03-18T01:47:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:07 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:47:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:07 smithi046 ceph-mon[36108]: scrub summary: active paths [/] 2024-03-18T01:47:07.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:07 smithi046 ceph-mon[36108]: scrub summary: idle+waiting paths [/] 2024-03-18T01:47:09.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:09 smithi007 ceph-mon[31996]: pgmap v686: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 86 KiB/s wr, 175 op/s 2024-03-18T01:47:09.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:09 smithi203 ceph-mon[36345]: pgmap v686: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 86 KiB/s wr, 175 op/s 2024-03-18T01:47:09.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:09 smithi046 ceph-mon[36108]: pgmap v686: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 86 KiB/s wr, 175 op/s 2024-03-18T01:47:11.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:11 smithi203 ceph-mon[36345]: pgmap v687: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 88 KiB/s wr, 130 op/s 2024-03-18T01:47:11.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:11 smithi203 ceph-mon[36345]: scrub summary: idle 2024-03-18T01:47:11.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:11 smithi046 ceph-mon[36108]: pgmap v687: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 88 KiB/s wr, 130 op/s 2024-03-18T01:47:11.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:11 smithi046 ceph-mon[36108]: scrub summary: idle 2024-03-18T01:47:11.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:11 smithi007 ceph-mon[31996]: pgmap v687: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 88 KiB/s wr, 130 op/s 2024-03-18T01:47:11.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:11 smithi007 ceph-mon[31996]: scrub summary: idle 2024-03-18T01:47:13.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:13 smithi203 ceph-mon[36345]: pgmap v688: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 96 op/s 2024-03-18T01:47:13.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:13 smithi046 ceph-mon[36108]: pgmap v688: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 96 op/s 2024-03-18T01:47:13.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:13 smithi007 ceph-mon[31996]: pgmap v688: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 96 op/s 2024-03-18T01:47:15.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:15 smithi203 ceph-mon[36345]: pgmap v689: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 42 KiB/s wr, 48 op/s 2024-03-18T01:47:15.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:15 smithi046 ceph-mon[36108]: pgmap v689: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 42 KiB/s wr, 48 op/s 2024-03-18T01:47:15.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:15 smithi007 ceph-mon[31996]: pgmap v689: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 42 KiB/s wr, 48 op/s 2024-03-18T01:47:17.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:17 smithi203 ceph-mon[36345]: pgmap v690: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-03-18T01:47:17.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:17 smithi046 ceph-mon[36108]: pgmap v690: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-03-18T01:47:17.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:17 smithi007 ceph-mon[31996]: pgmap v690: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 41 KiB/s wr, 26 op/s 2024-03-18T01:47:19.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:19 smithi203 ceph-mon[36345]: pgmap v691: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.5 KiB/s wr, 11 op/s 2024-03-18T01:47:19.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:19 smithi046 ceph-mon[36108]: pgmap v691: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.5 KiB/s wr, 11 op/s 2024-03-18T01:47:19.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:19 smithi007 ceph-mon[31996]: pgmap v691: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.5 KiB/s wr, 11 op/s 2024-03-18T01:47:21.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:21 smithi203 ceph-mon[36345]: pgmap v692: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-03-18T01:47:21.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:21 smithi046 ceph-mon[36108]: pgmap v692: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-03-18T01:47:21.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:21 smithi007 ceph-mon[31996]: pgmap v692: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.3 KiB/s wr, 11 op/s 2024-03-18T01:47:23.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:23 smithi203 ceph-mon[36345]: pgmap v693: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-03-18T01:47:23.521 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:23 smithi046 ceph-mon[36108]: pgmap v693: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-03-18T01:47:23.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:23 smithi007 ceph-mon[31996]: pgmap v693: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.9 KiB/s wr, 11 op/s 2024-03-18T01:47:24.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:24 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:47:24.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:24 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:47:24.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:24 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:47:25.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:25 smithi203 ceph-mon[36345]: pgmap v694: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-18T01:47:25.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:25 smithi046 ceph-mon[36108]: pgmap v694: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-18T01:47:25.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:25 smithi007 ceph-mon[31996]: pgmap v694: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-18T01:47:26.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:47:26.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:47:26.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:47:26.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:47:26.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:47:26.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:47:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:47:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:47:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:47:27.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:27 smithi046 ceph-mon[36108]: pgmap v695: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T01:47:27.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:27 smithi007 ceph-mon[31996]: pgmap v695: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T01:47:27.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:27 smithi203 ceph-mon[36345]: pgmap v695: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T01:47:29.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:29 smithi046 ceph-mon[36108]: pgmap v696: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:47:29.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:29 smithi007 ceph-mon[31996]: pgmap v696: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:47:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:29 smithi203 ceph-mon[36345]: pgmap v696: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T01:47:31.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:31 smithi046 ceph-mon[36108]: pgmap v697: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-18T01:47:31.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:31 smithi007 ceph-mon[31996]: pgmap v697: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-18T01:47:31.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:31 smithi203 ceph-mon[36345]: pgmap v697: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-18T01:47:33.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:33 smithi046 ceph-mon[36108]: pgmap v698: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:47:33.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:33 smithi007 ceph-mon[31996]: pgmap v698: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:47:33.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:33 smithi203 ceph-mon[36345]: pgmap v698: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T01:47:35.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:35 smithi046 ceph-mon[36108]: pgmap v699: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:35.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:35 smithi007 ceph-mon[31996]: pgmap v699: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:35.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:35 smithi203 ceph-mon[36345]: pgmap v699: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:36.158 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:47:36.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:47:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.391+0000 7f0b58c00640 1 -- 172.21.15.7:0/566070323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541000b0 msgr2=0x7f0b541004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.391+0000 7f0b58c00640 1 --2- 172.21.15.7:0/566070323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541000b0 0x7f0b541004d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0b44009900 tx=0x7f0b4402bc80 comp rx=0 tx=0).stop 2024-03-18T01:47:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.391+0000 7f0b58c00640 1 -- 172.21.15.7:0/566070323 shutdown_connections 2024-03-18T01:47:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.391+0000 7f0b58c00640 1 --2- 172.21.15.7:0/566070323 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b541013b0 0x7f0b54101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.392 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.391+0000 7f0b58c00640 1 --2- 172.21.15.7:0/566070323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b54100a10 0x7f0b54100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.391+0000 7f0b58c00640 1 --2- 172.21.15.7:0/566070323 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541000b0 0x7f0b541004d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.391+0000 7f0b58c00640 1 -- 172.21.15.7:0/566070323 >> 172.21.15.7:0/566070323 conn(0x7f0b540fbc70 msgr2=0x7f0b540fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.391+0000 7f0b58c00640 1 -- 172.21.15.7:0/566070323 shutdown_connections 2024-03-18T01:47:36.393 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.392+0000 7f0b58c00640 1 -- 172.21.15.7:0/566070323 wait complete. 2024-03-18T01:47:36.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.392+0000 7f0b58c00640 1 Processor -- start 2024-03-18T01:47:36.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b58c00640 1 -- start start 2024-03-18T01:47:36.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b58c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b541000b0 0x7f0b54199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b58c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b54100a10 0x7f0b5419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b58c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541013b0 0x7f0b541a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b58c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5410f6a0 con 0x7f0b541000b0 2024-03-18T01:47:36.395 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b58c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5410f520 con 0x7f0b541013b0 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b58c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5410f820 con 0x7f0b54100a10 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b51e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b541000b0 0x7f0b54199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b52800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541013b0 0x7f0b541a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b52800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541013b0 0x7f0b541a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52494/0 (socket says 172.21.15.7:52494) 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b52800640 1 -- 172.21.15.7:0/37441209 learned_addr learned my addr 172.21.15.7:0/37441209 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.393+0000 7f0b51400640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b54100a10 0x7f0b5419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b52800640 1 -- 172.21.15.7:0/37441209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b54100a10 msgr2=0x7f0b5419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b52800640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b54100a10 0x7f0b5419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b52800640 1 -- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b541000b0 msgr2=0x7f0b54199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b52800640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b541000b0 0x7f0b54199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b52800640 1 -- 172.21.15.7:0/37441209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b541a1960 con 0x7f0b541013b0 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b52800640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541013b0 0x7f0b541a12f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00eb10 tx=0x7f0b4c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b4aa00640 1 -- 172.21.15.7:0/37441209 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c004b20 con 0x7f0b541013b0 2024-03-18T01:47:36.396 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b4aa00640 1 -- 172.21.15.7:0/37441209 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4c004cc0 con 0x7f0b541013b0 2024-03-18T01:47:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b4aa00640 1 -- 172.21.15.7:0/37441209 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c010760 con 0x7f0b541013b0 2024-03-18T01:47:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.394+0000 7f0b58c00640 1 -- 172.21.15.7:0/37441209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b541a1c50 con 0x7f0b541013b0 2024-03-18T01:47:36.397 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.395+0000 7f0b58c00640 1 -- 172.21.15.7:0/37441209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b541a2150 con 0x7f0b541013b0 2024-03-18T01:47:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.399+0000 7f0b5b994740 1 Processor -- start 2024-03-18T01:47:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.399+0000 7f0b4aa00640 1 -- 172.21.15.7:0/37441209 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f0b4c005020 con 0x7f0b541013b0 2024-03-18T01:47:36.399 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.399+0000 7f0b5b994740 1 -- start start 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accdac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accdfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b5b994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5564acc3f1f0 con 0x5564accda810 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b5b994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5564acbe7a80 con 0x5564acce0370 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b5b994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5564acc3b290 con 0x5564accdb200 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accdfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accdfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58800/0 (socket says 172.21.15.7:58800) 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51e00640 1 -- 172.21.15.7:0/3072742741 learned_addr learned my addr 172.21.15.7:0/3072742741 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51400640 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accdac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.401 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51400640 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51e00640 1 -- 172.21.15.7:0/3072742741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 msgr2=0x5564acce2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51e00640 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51e00640 1 -- 172.21.15.7:0/3072742741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 msgr2=0x5564accdac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51e00640 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accdac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51e00640 1 -- 172.21.15.7:0/3072742741 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564aca1d7d0 con 0x5564accdb200 2024-03-18T01:47:36.402 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.400+0000 7f0b51400640 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accdac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.401+0000 7f0b4aa00640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0b380a2880 0x7f0b380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.401+0000 7f0b51e00640 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accdfd00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0b44000c00 tx=0x7f0b4402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a52520273c31f6dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.401+0000 7f0b4aa00640 1 -- 172.21.15.7:0/37441209 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0b4c0cf300 con 0x7f0b541013b0 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.401+0000 7f0b3f000640 1 -- 172.21.15.7:0/3072742741 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b44045070 con 0x5564accdb200 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.401+0000 7f0b51400640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0b380a2880 0x7f0b380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.401+0000 7f0b3f000640 1 -- 172.21.15.7:0/3072742741 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b44004040 con 0x5564accdb200 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.401+0000 7f0b3f000640 1 -- 172.21.15.7:0/3072742741 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b44004b80 con 0x5564accdb200 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 -- 172.21.15.7:0/3072742741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 msgr2=0x5564accdfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b51400640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0b380a2880 0x7f0b380a4d40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x5564accdba60 tx=0x7f0b400096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:36.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accdfd00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0b44000c00 tx=0x7f0b4402c650 comp rx=0 tx=0).stop 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 -- 172.21.15.7:0/3072742741 shutdown_connections 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accdfd00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3072742741 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accdac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 -- 172.21.15.7:0/3072742741 >> 172.21.15.7:0/3072742741 conn(0x5564accd5d80 msgr2=0x5564accd6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 -- 172.21.15.7:0/3072742741 shutdown_connections 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.402+0000 7f0b5b994740 1 -- 172.21.15.7:0/3072742741 wait complete. 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.403+0000 7f0b5b994740 1 -- start start 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.403+0000 7f0b5b994740 1 Processor -- start 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.403+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.403+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accdb200 0x5564acce5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564accec250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b5b994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x5564acce32c0 con 0x5564accdb200 2024-03-18T01:47:36.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b5b994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5564acbfaed0 con 0x5564acce0370 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b5b994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5564acce3440 con 0x5564accda810 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b5b994740 10 client.0 _ll_register_callbacks cb 0x5564acc36220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b52800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564accec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58814/0 (socket says 172.21.15.7:58814) 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b52800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564accec250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52512/0 (socket says 172.21.15.7:52512) 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51e00640 1 -- 172.21.15.7:0/696952700 learned_addr learned my addr 172.21.15.7:0/696952700 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51400640 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accdb200 0x5564acce5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51e00640 1 -- 172.21.15.7:0/696952700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 msgr2=0x5564accec250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51e00640 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564accec250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51e00640 1 -- 172.21.15.7:0/696952700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accdb200 msgr2=0x5564acce5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51e00640 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accdb200 0x5564acce5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b51e00640 1 -- 172.21.15.7:0/696952700 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564ac9533f0 con 0x5564accda810 2024-03-18T01:47:36.408 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.404+0000 7f0b52800640 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564accec250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.405+0000 7f0b51e00640 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce4ce0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0b4402c7a0 tx=0x7f0b4402cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b90b6a0c54af488 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.405+0000 7f0b33400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.405+0000 7f0b33400640 1 -- 172.21.15.7:0/696952700 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4404f060 con 0x5564accda810 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.405+0000 7f0b33400640 1 -- 172.21.15.7:0/696952700 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0b44004f10 con 0x5564accda810 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.405+0000 7f0b5b994740 1 -- 172.21.15.7:0/696952700 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b44004040 con 0x5564accda810 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.405+0000 7f0b33400640 1 -- 172.21.15.7:0/696952700 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f0b44056050 con 0x5564accda810 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.405+0000 7f0b5b994740 10 client.26452 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.405+0000 7f0b5b994740 1 -- 172.21.15.7:0/696952700 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x5564ac9130f0 con 0x5564accda810 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b33400640 1 -- 172.21.15.7:0/696952700 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b44004040 con 0x5564accda810 2024-03-18T01:47:36.409 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b33400640 1 -- 172.21.15.7:0/696952700 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f0b44003b90 con 0x5564accda810 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b5b994740 10 client.26452 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b5b994740 10 client.26452 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b5b994740 20 client.26452 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b5b994740 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5564accf0740 0x5564accf2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b5b994740 4 client.26452 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b5b994740 1 -- 172.21.15.7:0/696952700 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564acc52be0 con 0x5564accf0740 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.406+0000 7f0b51400640 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5564accf0740 0x5564accf2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.407+0000 7f0b51400640 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5564accf0740 0x5564accf2b90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2bede903713c058 server_cookie=8a0d805e50998e52 in_seq=0 out_seq=0 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.407+0000 7f0b33400640 10 client.26452 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:36.411 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.408+0000 7f0b33400640 1 -- 172.21.15.7:0/696952700 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5564acc52be0 con 0x5564accf0740 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.408+0000 7f0b33400640 10 client.26452 handle_command_reply: tid=0 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.408+0000 7f0b5b994740 1 client.26452 shutdown 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 -- 172.21.15.7:0/696952700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 msgr2=0x5564acce4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce4ce0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0b4402c7a0 tx=0x7f0b4402cc10 comp rx=0 tx=0).stop 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 -- 172.21.15.7:0/696952700 shutdown_connections 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5564accf0740 0x5564accf2b90 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564accec250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accdb200 0x5564acce5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 --2- 172.21.15.7:0/696952700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce4ce0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 -- 172.21.15.7:0/696952700 >> 172.21.15.7:0/696952700 conn(0x5564accd5d80 msgr2=0x5564accd4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b33400640 0 client.26452 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 -- 172.21.15.7:0/696952700 shutdown_connections 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 1 -- 172.21.15.7:0/696952700 wait complete. 2024-03-18T01:47:36.412 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.409+0000 7f0b5b994740 20 client.26452 trim_cache size 0 max 16384 2024-03-18T01:47:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.432+0000 7f0b5b994740 1 Processor -- start 2024-03-18T01:47:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.432+0000 7f0b5b994740 1 -- start start 2024-03-18T01:47:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.432+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accd4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.432+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accec210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.432+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.434 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.432+0000 7f0b5b994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541ac560 con 0x5564accda810 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.432+0000 7f0b5b994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541ae1c0 con 0x5564acce0370 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.432+0000 7f0b5b994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541acc60 con 0x5564accdb200 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b51400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accec210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52522/0 (socket says 172.21.15.7:52522) 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b51400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accec210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58824/0 (socket says 172.21.15.7:58824) 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 -- 172.21.15.7:0/1990690907 learned_addr learned my addr 172.21.15.7:0/1990690907 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b51e00640 1 --2- 172.21.15.7:0/1990690907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accd4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 -- 172.21.15.7:0/1990690907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 msgr2=0x5564accec210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 --2- 172.21.15.7:0/1990690907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accec210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 -- 172.21.15.7:0/1990690907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 msgr2=0x5564accd4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:36.435 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 --2- 172.21.15.7:0/1990690907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accd4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 -- 172.21.15.7:0/1990690907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564accf3210 con 0x5564acce0370 2024-03-18T01:47:36.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.433+0000 7f0b52800640 1 --2- 172.21.15.7:0/1990690907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce4300 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c09b5e0 tx=0x7f0b4c0058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce05ef64173ac2b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:36.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b3e600640 1 -- 172.21.15.7:0/1990690907 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c0b3070 con 0x5564acce0370 2024-03-18T01:47:36.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b3e600640 1 -- 172.21.15.7:0/1990690907 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4c09b8e0 con 0x5564acce0370 2024-03-18T01:47:36.436 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b3e600640 1 -- 172.21.15.7:0/1990690907 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c09bda0 con 0x5564acce0370 2024-03-18T01:47:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b5b994740 1 -- 172.21.15.7:0/1990690907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 msgr2=0x5564acce4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b5b994740 1 --2- 172.21.15.7:0/1990690907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce4300 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c09b5e0 tx=0x7f0b4c0058d0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b5b994740 1 -- 172.21.15.7:0/1990690907 shutdown_connections 2024-03-18T01:47:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b5b994740 1 --2- 172.21.15.7:0/1990690907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564acce0370 0x5564acce4300 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b5b994740 1 --2- 172.21.15.7:0/1990690907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accdb200 0x5564accec210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.440 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b5b994740 1 --2- 172.21.15.7:0/1990690907 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564accda810 0x5564accd4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.434+0000 7f0b5b994740 1 -- 172.21.15.7:0/1990690907 >> 172.21.15.7:0/1990690907 conn(0x5564accd5d80 msgr2=0x5564accd4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.435+0000 7f0b5b994740 1 -- 172.21.15.7:0/1990690907 shutdown_connections 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.435+0000 7f0b5b994740 1 -- 172.21.15.7:0/1990690907 wait complete. 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.435+0000 7f0b5b994740 1 -- start start 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.435+0000 7f0b5b994740 1 Processor -- start 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.435+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564accdb200 0x5564acce9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b5b994740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564acce0370 0x5564acceaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b5b994740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541ada90 con 0x5564acce0370 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b5b994740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541ad360 con 0x5564accdb200 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b5b994740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541aa540 con 0x5564accda810 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b5b994740 10 client.0 _ll_register_callbacks cb 0x7f0b44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:36.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564accdb200 0x5564acce9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564accdb200 0x5564acce9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52526/0 (socket says 172.21.15.7:52526) 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51400640 1 -- 172.21.15.7:0/3571796350 learned_addr learned my addr 172.21.15.7:0/3571796350 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b52800640 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564acce0370 0x5564acceaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51400640 1 -- 172.21.15.7:0/3571796350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 msgr2=0x5564acce94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51400640 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51400640 1 -- 172.21.15.7:0/3571796350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564acce0370 msgr2=0x5564acceaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51400640 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564acce0370 0x5564acceaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51400640 1 -- 172.21.15.7:0/3571796350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564acceb030 con 0x5564accdb200 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.436+0000 7f0b51e00640 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.437+0000 7f0b51400640 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564accdb200 0x5564acce9a30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x5564acce61b0 tx=0x7f0b40004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4acb0515824acc08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.437+0000 7f0b33400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.437+0000 7f0b33400640 1 -- 172.21.15.7:0/3571796350 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4002e480 con 0x5564accdb200 2024-03-18T01:47:36.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.437+0000 7f0b5b994740 1 -- 172.21.15.7:0/3571796350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b4c09b8e0 con 0x5564accdb200 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.437+0000 7f0b33400640 1 -- 172.21.15.7:0/3571796350 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0b4002e620 con 0x5564accdb200 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.437+0000 7f0b33400640 1 -- 172.21.15.7:0/3571796350 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f0b4003e050 con 0x5564accdb200 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.437+0000 7f0b5b994740 10 client.26474 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.437+0000 7f0b5b994740 1 -- 172.21.15.7:0/3571796350 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x5564accec3b0 con 0x5564accdb200 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b33400640 1 -- 172.21.15.7:0/3571796350 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4c09b8e0 con 0x5564accdb200 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b33400640 1 -- 172.21.15.7:0/3571796350 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f0b4002b050 con 0x5564accdb200 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b5b994740 10 client.26474 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b5b994740 10 client.26474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b5b994740 20 client.26474 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:36.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5564acd57170 0x5564acd59590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b5b994740 4 client.26474 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b5b994740 1 -- 172.21.15.7:0/3571796350 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5564acc4c730 con 0x5564acd57170 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.438+0000 7f0b51e00640 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5564acd57170 0x5564acd59590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b51e00640 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5564acd57170 0x5564acd59590 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=468a4e592a137fae server_cookie=15b60ea87020ce21 in_seq=0 out_seq=0 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b33400640 10 client.26474 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b33400640 1 -- 172.21.15.7:0/3571796350 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5564acc4c730 con 0x5564acd57170 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b33400640 10 client.26474 handle_command_reply: tid=0 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 client.26474 shutdown 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 -- 172.21.15.7:0/3571796350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564accdb200 msgr2=0x5564acce9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564accdb200 0x5564acce9a30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x5564acce61b0 tx=0x7f0b40004730 comp rx=0 tx=0).stop 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 -- 172.21.15.7:0/3571796350 shutdown_connections 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x5564acd57170 0x5564acd59590 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x5564acce0370 0x5564acceaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.444 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5564accdb200 0x5564acce9a30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 --2- 172.21.15.7:0/3571796350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5564accda810 0x5564acce94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b33400640 0 client.26474 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 -- 172.21.15.7:0/3571796350 >> 172.21.15.7:0/3571796350 conn(0x5564accd5d80 msgr2=0x5564accd4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 -- 172.21.15.7:0/3571796350 shutdown_connections 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 1 -- 172.21.15.7:0/3571796350 wait complete. 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.439+0000 7f0b5b994740 20 client.26474 trim_cache size 0 max 16384 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stdout: "status": "no active scrubs running", 2024-03-18T01:47:36.445 INFO:teuthology.orchestra.run.smithi007.stdout: "scrubs": {} 2024-03-18T01:47:36.446 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 -- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0b380a2880 msgr2=0x7f0b380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0b380a2880 0x7f0b380a4d40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x5564accdba60 tx=0x7f0b400096c0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.446 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 -- 172.21.15.7:0/37441209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541013b0 msgr2=0x7f0b541a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541013b0 0x7f0b541a12f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00eb10 tx=0x7f0b4c00c2d0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 -- 172.21.15.7:0/37441209 shutdown_connections 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b541013b0 0x7f0b541a12f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b54100a10 0x7f0b5419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f0b380a2880 0x7f0b380a4d40 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 --2- 172.21.15.7:0/37441209 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f0b541000b0 0x7f0b54199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 -- 172.21.15.7:0/37441209 >> 172.21.15.7:0/37441209 conn(0x7f0b540fbc70 msgr2=0x7f0b540fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 -- 172.21.15.7:0/37441209 shutdown_connections 2024-03-18T01:47:36.447 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.440+0000 7f0b58c00640 1 -- 172.21.15.7:0/37441209 wait complete. 2024-03-18T01:47:36.459 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T01:47:36.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:47:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 -- 172.21.15.7:0/774011778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6714107580 msgr2=0x7f6714109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 --2- 172.21.15.7:0/774011778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6714107580 0x7f6714109970 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f670c009ca0 tx=0x7f670c02c0f0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 -- 172.21.15.7:0/774011778 shutdown_connections 2024-03-18T01:47:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 --2- 172.21.15.7:0/774011778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6714107580 0x7f6714109970 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 --2- 172.21.15.7:0/774011778 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6714104c50 0x7f6714107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.672 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 --2- 172.21.15.7:0/774011778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6714102320 0x7f6714104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 -- 172.21.15.7:0/774011778 >> 172.21.15.7:0/774011778 conn(0x7f67140fbc70 msgr2=0x7f67140fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:36.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 -- 172.21.15.7:0/774011778 shutdown_connections 2024-03-18T01:47:36.673 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.671+0000 7f671a000640 1 -- 172.21.15.7:0/774011778 wait complete. 2024-03-18T01:47:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.672+0000 7f671a000640 1 Processor -- start 2024-03-18T01:47:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f671a000640 1 -- start start 2024-03-18T01:47:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f671a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6714102320 0x7f6714199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f671a000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6714104c50 0x7f671419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f671a000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6714107580 0x7f67141a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f671a000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f671410d490 con 0x7f6714104c50 2024-03-18T01:47:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f671a000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f671410d310 con 0x7f6714102320 2024-03-18T01:47:36.674 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f671a000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f671410d610 con 0x7f6714107580 2024-03-18T01:47:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f6713400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6714102320 0x7f6714199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f6713400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6714102320 0x7f6714199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52550/0 (socket says 172.21.15.7:52550) 2024-03-18T01:47:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f670aa00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6714104c50 0x7f671419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.673+0000 7f6713400640 1 -- 172.21.15.7:0/2357921234 learned_addr learned my addr 172.21.15.7:0/2357921234 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:36.675 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.674+0000 7f6713e00640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6714107580 0x7f67141a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.674+0000 7f670aa00640 1 -- 172.21.15.7:0/2357921234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6714107580 msgr2=0x7f67141a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.674+0000 7f670aa00640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6714107580 0x7f67141a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.674+0000 7f670aa00640 1 -- 172.21.15.7:0/2357921234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6714102320 msgr2=0x7f6714199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.674+0000 7f670aa00640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6714102320 0x7f6714199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.674+0000 7f670aa00640 1 -- 172.21.15.7:0/2357921234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67141a1830 con 0x7f6714104c50 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.674+0000 7f6713400640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6714102320 0x7f6714199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.674+0000 7f670aa00640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6714104c50 0x7f671419a160 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f66fc00c940 tx=0x7f66fc00ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.675+0000 7f6710c00640 1 -- 172.21.15.7:0/2357921234 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc007bb0 con 0x7f6714104c50 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.675+0000 7f6710c00640 1 -- 172.21.15.7:0/2357921234 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66fc007d50 con 0x7f6714104c50 2024-03-18T01:47:36.676 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.675+0000 7f6710c00640 1 -- 172.21.15.7:0/2357921234 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc0187b0 con 0x7f6714104c50 2024-03-18T01:47:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.675+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67141a1b20 con 0x7f6714104c50 2024-03-18T01:47:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.675+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67141a2020 con 0x7f6714104c50 2024-03-18T01:47:36.677 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.676+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6714109070 con 0x7f6714104c50 2024-03-18T01:47:36.679 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.679+0000 7f6710c00640 1 -- 172.21.15.7:0/2357921234 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f66fc00f450 con 0x7f6714104c50 2024-03-18T01:47:36.680 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.680+0000 7f6710c00640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f66f80a2800 0x7f66f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.680+0000 7f6710c00640 1 -- 172.21.15.7:0/2357921234 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f66fc0cf170 con 0x7f6714104c50 2024-03-18T01:47:36.681 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.681+0000 7f6713400640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f66f80a2800 0x7f66f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:36.682 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.681+0000 7f6713400640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f66f80a2800 0x7f66f80a4cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f671419b0f0 tx=0x7f6704005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:36.692 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.692+0000 7f6710c00640 1 -- 172.21.15.7:0/2357921234 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f66fc014030 con 0x7f6714104c50 2024-03-18T01:47:36.941 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.941+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f671402d1b0 con 0x7f6714104c50 2024-03-18T01:47:36.943 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.942+0000 7f6710c00640 1 -- 172.21.15.7:0/2357921234 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+11939 (secure 0 0 0) 0x7f671402d1b0 con 0x7f6714104c50 2024-03-18T01:47:36.943 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:47:36.943 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":150,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:47:00.268898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":164,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":160,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":188,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":160,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":158,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":157,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":182,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24596,24562,24542,24523,24544]},"id":1}]} 2024-03-18T01:47:36.943 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 150 2024-03-18T01:47:36.945 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f66f80a2800 msgr2=0x7f66f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f66f80a2800 0x7f66f80a4cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f671419b0f0 tx=0x7f6704005f90 comp rx=0 tx=0).stop 2024-03-18T01:47:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6714104c50 msgr2=0x7f671419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6714104c50 0x7f671419a160 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f66fc00c940 tx=0x7f66fc00ce00 comp rx=0 tx=0).stop 2024-03-18T01:47:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 shutdown_connections 2024-03-18T01:47:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6714107580 0x7f67141a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f66f80a2800 0x7f66f80a4cc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.946 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6714104c50 0x7f671419a160 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.945+0000 7f671a000640 1 --2- 172.21.15.7:0/2357921234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6714102320 0x7f6714199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:36.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.946+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 >> 172.21.15.7:0/2357921234 conn(0x7f67140fbc70 msgr2=0x7f6714101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:36.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.946+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 shutdown_connections 2024-03-18T01:47:36.947 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:36.946+0000 7f671a000640 1 -- 172.21.15.7:0/2357921234 wait complete. 2024-03-18T01:47:36.968 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T01:47:37.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.176+0000 7fd124a00640 1 -- 172.21.15.7:0/3741105418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd12010ddc0 msgr2=0x7fd1201101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.177 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.176+0000 7fd124a00640 1 --2- 172.21.15.7:0/3741105418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd12010ddc0 0x7fd1201101b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd118009ca0 tx=0x7fd11802c100 comp rx=0 tx=0).stop 2024-03-18T01:47:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.177+0000 7fd124a00640 1 -- 172.21.15.7:0/3741105418 shutdown_connections 2024-03-18T01:47:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.177+0000 7fd124a00640 1 --2- 172.21.15.7:0/3741105418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd12010ddc0 0x7fd1201101b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.177+0000 7fd124a00640 1 --2- 172.21.15.7:0/3741105418 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd120074240 0x7fd1200746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.177+0000 7fd124a00640 1 --2- 172.21.15.7:0/3741105418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120075820 0x7fd120073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.177+0000 7fd124a00640 1 -- 172.21.15.7:0/3741105418 >> 172.21.15.7:0/3741105418 conn(0x7fd1200fcd20 msgr2=0x7fd1200ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.177+0000 7fd124a00640 1 -- 172.21.15.7:0/3741105418 shutdown_connections 2024-03-18T01:47:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.177+0000 7fd124a00640 1 -- 172.21.15.7:0/3741105418 wait complete. 2024-03-18T01:47:37.178 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.178+0000 7fd124a00640 1 Processor -- start 2024-03-18T01:47:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.178+0000 7fd124a00640 1 -- start start 2024-03-18T01:47:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd124a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120074240 0x7fd12019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.179 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd124a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120075820 0x7fd12019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd124a00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd12010ddc0 0x7fd1201a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd124a00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd120113a40 con 0x7fd12010ddc0 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd124a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201138c0 con 0x7fd120074240 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd124a00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd120113bc0 con 0x7fd120075820 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd11d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120075820 0x7fd12019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd11de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120074240 0x7fd12019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd11d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120075820 0x7fd12019e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58874/0 (socket says 172.21.15.7:58874) 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd11d400640 1 -- 172.21.15.7:0/3101475779 learned_addr learned my addr 172.21.15.7:0/3101475779 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.180 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.179+0000 7fd11e800640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd12010ddc0 0x7fd1201a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd11de00640 1 -- 172.21.15.7:0/3101475779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120075820 msgr2=0x7fd12019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd11de00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120075820 0x7fd12019e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd11de00640 1 -- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd12010ddc0 msgr2=0x7fd1201a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd11de00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd12010ddc0 0x7fd1201a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.181 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd11de00640 1 -- 172.21.15.7:0/3101475779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1201a5c70 con 0x7fd120074240 2024-03-18T01:47:37.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd11e800640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd12010ddc0 0x7fd1201a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:37.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd11d400640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120075820 0x7fd12019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:37.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd11de00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120074240 0x7fd12019e050 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd11000e9f0 tx=0x7fd11000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd116a00640 1 -- 172.21.15.7:0/3101475779 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11000cd80 con 0x7fd120074240 2024-03-18T01:47:37.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd116a00640 1 -- 172.21.15.7:0/3101475779 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1100035f0 con 0x7fd120074240 2024-03-18T01:47:37.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd124a00640 1 -- 172.21.15.7:0/3101475779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1201a5f60 con 0x7fd120074240 2024-03-18T01:47:37.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd116a00640 1 -- 172.21.15.7:0/3101475779 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd110010690 con 0x7fd120074240 2024-03-18T01:47:37.182 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.180+0000 7fd124a00640 1 -- 172.21.15.7:0/3101475779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1201a6420 con 0x7fd120074240 2024-03-18T01:47:37.184 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.183+0000 7fd116a00640 1 -- 172.21.15.7:0/3101475779 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fd110003b30 con 0x7fd120074240 2024-03-18T01:47:37.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.185+0000 7fd116a00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1040a2880 0x7fd1040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.185 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.185+0000 7fd116a00640 1 -- 172.21.15.7:0/3101475779 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd1100cf110 con 0x7fd120074240 2024-03-18T01:47:37.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.185+0000 7fd1277c1740 1 Processor -- start 2024-03-18T01:47:37.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.185+0000 7fd11d400640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1040a2880 0x7fd1040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.186 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.185+0000 7fd1277c1740 1 -- start start 2024-03-18T01:47:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c628c0 0x55fe54c62ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c67db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c68420 0x55fe54c6a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd1277c1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55fe54c2a3b0 con 0x55fe54c68420 2024-03-18T01:47:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd1277c1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fe54bc7450 con 0x55fe54c632b0 2024-03-18T01:47:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd1277c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fe54b6fcb0 con 0x55fe54c628c0 2024-03-18T01:47:37.187 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd11d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c67db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd11d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c67db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52592/0 (socket says 172.21.15.7:52592) 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd11d400640 1 -- 172.21.15.7:0/1375172099 learned_addr learned my addr 172.21.15.7:0/1375172099 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd11de00640 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c628c0 0x55fe54c62ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd11d400640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1040a2880 0x7fd1040a4d40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd12019f560 tx=0x7fd10c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.186+0000 7fd11e800640 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c68420 0x55fe54c6a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd11d400640 1 -- 172.21.15.7:0/1375172099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c628c0 msgr2=0x55fe54c62ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd11d400640 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c628c0 0x55fe54c62ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd11d400640 1 -- 172.21.15.7:0/1375172099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c68420 msgr2=0x55fe54c6a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd11d400640 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c68420 0x55fe54c6a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd11d400640 1 -- 172.21.15.7:0/1375172099 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe54b911f0 con 0x55fe54c632b0 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd11e800640 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c68420 0x55fe54c6a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:37.188 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd11d400640 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c67db0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00f6f0 tx=0x7fd10c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3edbb0b816f0429c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd10b000640 1 -- 172.21.15.7:0/1375172099 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c005020 con 0x55fe54c632b0 2024-03-18T01:47:37.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd10b000640 1 -- 172.21.15.7:0/1375172099 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd10c0047a0 con 0x55fe54c632b0 2024-03-18T01:47:37.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.187+0000 7fd10b000640 1 -- 172.21.15.7:0/1375172099 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c01c870 con 0x55fe54c632b0 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.188+0000 7fd1277c1740 1 -- 172.21.15.7:0/1375172099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 msgr2=0x55fe54c67db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.188+0000 7fd1277c1740 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c67db0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00f6f0 tx=0x7fd10c014000 comp rx=0 tx=0).stop 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.188+0000 7fd1277c1740 1 -- 172.21.15.7:0/1375172099 shutdown_connections 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.188+0000 7fd1277c1740 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c68420 0x55fe54c6a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.188+0000 7fd1277c1740 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c67db0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.188+0000 7fd1277c1740 1 --2- 172.21.15.7:0/1375172099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c628c0 0x55fe54c62ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.188+0000 7fd1277c1740 1 -- 172.21.15.7:0/1375172099 >> 172.21.15.7:0/1375172099 conn(0x55fe54c5de30 msgr2=0x55fe54c5e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.188+0000 7fd1277c1740 1 -- 172.21.15.7:0/1375172099 shutdown_connections 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.189+0000 7fd1277c1740 1 -- 172.21.15.7:0/1375172099 wait complete. 2024-03-18T01:47:37.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.189+0000 7fd1277c1740 1 -- start start 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.189+0000 7fd1277c1740 1 Processor -- start 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c6cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 0x55fe54c6d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c74110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd1277c1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55fe54b83110 con 0x55fe54c632b0 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd1277c1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55fe54bc34f0 con 0x55fe54c628c0 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd1277c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fe54c6b3d0 con 0x55fe54c68420 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd1277c1740 10 client.0 _ll_register_callbacks cb 0x55fe54b70b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:37.193 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd11de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c6cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd11e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c74110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd11de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c6cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52594/0 (socket says 172.21.15.7:52594) 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.190+0000 7fd11de00640 1 -- 172.21.15.7:0/3065113050 learned_addr learned my addr 172.21.15.7:0/3065113050 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.191+0000 7fd11d400640 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 0x55fe54c6d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.191+0000 7fd11e800640 1 -- 172.21.15.7:0/3065113050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 msgr2=0x55fe54c6cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.191+0000 7fd11e800640 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c6cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.191+0000 7fd11e800640 1 -- 172.21.15.7:0/3065113050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 msgr2=0x55fe54c6d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.191+0000 7fd11e800640 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 0x55fe54c6d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.191+0000 7fd11e800640 1 -- 172.21.15.7:0/3065113050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe548db3f0 con 0x55fe54c68420 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.191+0000 7fd11e800640 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c74110 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd118037040 tx=0x7fd118009910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2869748f25218b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.191+0000 7fd0ff400640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.192+0000 7fd0ff400640 1 -- 172.21.15.7:0/3065113050 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd118039040 con 0x55fe54c68420 2024-03-18T01:47:37.194 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.192+0000 7fd0ff400640 1 -- 172.21.15.7:0/3065113050 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd118003970 con 0x55fe54c68420 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.192+0000 7fd1277c1740 1 -- 172.21.15.7:0/3065113050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd10c0047a0 con 0x55fe54c68420 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.192+0000 7fd0ff400640 1 -- 172.21.15.7:0/3065113050 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7fd118050050 con 0x55fe54c68420 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.192+0000 7fd1277c1740 10 client.26476 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.192+0000 7fd1277c1740 1 -- 172.21.15.7:0/3065113050 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x55fe549a57d0 con 0x55fe54c68420 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.192+0000 7fd0ff400640 1 -- 172.21.15.7:0/3065113050 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd10c0047a0 con 0x55fe54c68420 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.192+0000 7fd0ff400640 1 -- 172.21.15.7:0/3065113050 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7fd11804c510 con 0x55fe54c68420 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd1277c1740 10 client.26476 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd1277c1740 10 client.26476 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd1277c1740 20 client.26476 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55fe54c78ad0 0x55fe54c7af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd1277c1740 4 client.26476 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:37.195 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd1277c1740 1 -- 172.21.15.7:0/3065113050 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe54bd4990 con 0x55fe54c78ad0 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd11de00640 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55fe54c78ad0 0x55fe54c7af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd11de00640 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55fe54c78ad0 0x55fe54c7af10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e62f47dcbb776d5 server_cookie=899234f0155c7608 in_seq=0 out_seq=0 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.193+0000 7fd0ff400640 10 client.26476 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.194+0000 7fd0ff400640 1 -- 172.21.15.7:0/3065113050 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fe54bd4990 con 0x55fe54c78ad0 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.194+0000 7fd0ff400640 10 client.26476 handle_command_reply: tid=0 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.194+0000 7fd1277c1740 1 client.26476 shutdown 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.194+0000 7fd1277c1740 1 -- 172.21.15.7:0/3065113050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 msgr2=0x55fe54c74110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.194+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c74110 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd118037040 tx=0x7fd118009910 comp rx=0 tx=0).stop 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 1 -- 172.21.15.7:0/3065113050 shutdown_connections 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55fe54c78ad0 0x55fe54c7af10 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c74110 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.197 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 0x55fe54c6d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd0ff400640 0 client.26476 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:47:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3065113050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c6cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 1 -- 172.21.15.7:0/3065113050 >> 172.21.15.7:0/3065113050 conn(0x55fe54c5de30 msgr2=0x55fe54c5d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 1 -- 172.21.15.7:0/3065113050 shutdown_connections 2024-03-18T01:47:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 1 -- 172.21.15.7:0/3065113050 wait complete. 2024-03-18T01:47:37.198 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.195+0000 7fd1277c1740 20 client.26476 trim_cache size 0 max 16384 2024-03-18T01:47:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.210+0000 7fd1277c1740 1 Processor -- start 2024-03-18T01:47:37.211 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.210+0000 7fd1277c1740 1 -- start start 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.210+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c628c0 0x55fe54c5d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.210+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c740d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.210+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c7f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.210+0000 7fd1277c1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201b0830 con 0x55fe54c628c0 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.210+0000 7fd1277c1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201b2490 con 0x55fe54c632b0 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.210+0000 7fd1277c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201b0f30 con 0x55fe54c68420 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c7f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c7f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58902/0 (socket says 172.21.15.7:58902) 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 -- 172.21.15.7:0/3049885773 learned_addr learned my addr 172.21.15.7:0/3049885773 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11d400640 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11de00640 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c628c0 0x55fe54c5d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 -- 172.21.15.7:0/3049885773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 msgr2=0x55fe54c740d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c740d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.212 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 -- 172.21.15.7:0/3049885773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c628c0 msgr2=0x55fe54c5d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c628c0 0x55fe54c5d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 -- 172.21.15.7:0/3049885773 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe54c58410 con 0x55fe54c68420 2024-03-18T01:47:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11de00640 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c628c0 0x55fe54c5d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:47:37.213 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.211+0000 7fd11e800640 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c7f7d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd11804ca00 tx=0x7fd118009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63deec0071362855 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd10a600640 1 -- 172.21.15.7:0/3049885773 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd118039030 con 0x55fe54c68420 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd10a600640 1 -- 172.21.15.7:0/3049885773 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd118003e50 con 0x55fe54c68420 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd10a600640 1 -- 172.21.15.7:0/3049885773 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd118004290 con 0x55fe54c68420 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd1277c1740 1 -- 172.21.15.7:0/3049885773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 msgr2=0x55fe54c7f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c7f7d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd11804ca00 tx=0x7fd118009540 comp rx=0 tx=0).stop 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd1277c1740 1 -- 172.21.15.7:0/3049885773 shutdown_connections 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c7f7d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c632b0 0x55fe54c740d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd1277c1740 1 --2- 172.21.15.7:0/3049885773 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c628c0 0x55fe54c5d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.216 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.212+0000 7fd1277c1740 1 -- 172.21.15.7:0/3049885773 >> 172.21.15.7:0/3049885773 conn(0x55fe54c5de30 msgr2=0x55fe54c5c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.213+0000 7fd1277c1740 1 -- 172.21.15.7:0/3049885773 shutdown_connections 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.213+0000 7fd1277c1740 1 -- 172.21.15.7:0/3049885773 wait complete. 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.213+0000 7fd1277c1740 1 -- start start 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.213+0000 7fd1277c1740 1 Processor -- start 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c719c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 0x55fe54c71f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd1277c1740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c72f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd1277c1740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201b1d60 con 0x55fe54c632b0 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd1277c1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201b1630 con 0x55fe54c628c0 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd1277c1740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd1201ae810 con 0x55fe54c68420 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd1277c1740 10 client.0 _ll_register_callbacks cb 0x7fd118037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c72f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52616/0 (socket says 172.21.15.7:52616) 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11e800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c72f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58908/0 (socket says 172.21.15.7:58908) 2024-03-18T01:47:37.217 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11de00640 1 -- 172.21.15.7:0/4077850268 learned_addr learned my addr 172.21.15.7:0/4077850268 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11d400640 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 0x55fe54c71f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11de00640 1 -- 172.21.15.7:0/4077850268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 msgr2=0x55fe54c72f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11de00640 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c72f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11de00640 1 -- 172.21.15.7:0/4077850268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 msgr2=0x55fe54c71f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11de00640 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 0x55fe54c71f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11de00640 1 -- 172.21.15.7:0/4077850268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe54c73500 con 0x55fe54c628c0 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.214+0000 7fd11e800640 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c72f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd11de00640 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c719c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd11009b680 tx=0x7fd110086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eb3c26f1c760659 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd0ff400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd0ff400640 1 -- 172.21.15.7:0/4077850268 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1100b8070 con 0x55fe54c628c0 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd0ff400640 1 -- 172.21.15.7:0/4077850268 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd1100999f0 con 0x55fe54c628c0 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd1277c1740 1 -- 172.21.15.7:0/4077850268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd118003e50 con 0x55fe54c628c0 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd0ff400640 1 -- 172.21.15.7:0/4077850268 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7fd1100037a0 con 0x55fe54c628c0 2024-03-18T01:47:37.218 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd1277c1740 10 client.26498 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd1277c1740 1 -- 172.21.15.7:0/4077850268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x55fe54cdf000 con 0x55fe54c628c0 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.215+0000 7fd0ff400640 1 -- 172.21.15.7:0/4077850268 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd118003e50 con 0x55fe54c628c0 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.216+0000 7fd0ff400640 1 -- 172.21.15.7:0/4077850268 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7fd1100afad0 con 0x55fe54c628c0 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.216+0000 7fd1277c1740 10 client.26498 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.216+0000 7fd1277c1740 10 client.26498 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.216+0000 7fd1277c1740 20 client.26498 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.216+0000 7fd1277c1740 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55fe54cdf9a0 0x55fe54ce1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.216+0000 7fd1277c1740 4 client.26498 mds_command: new command op to 24523 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:37.219 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.216+0000 7fd1277c1740 1 -- 172.21.15.7:0/4077850268 --> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe54bcece0 con 0x55fe54cdf9a0 2024-03-18T01:47:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.216+0000 7fd11d400640 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55fe54cdf9a0 0x55fe54ce1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.220 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.217+0000 7fd11d400640 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55fe54cdf9a0 0x55fe54ce1dc0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c4b8d609f2f8572b server_cookie=ea860ce4e35cabe5 in_seq=0 out_seq=0 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.217+0000 7fd0ff400640 10 client.26498 ms_handle_connect on v2:172.21.15.203:6836/2344597644 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.217+0000 7fd0ff400640 1 -- 172.21.15.7:0/4077850268 <== mds.4 v2:172.21.15.203:6836/2344597644 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe54bcece0 con 0x55fe54cdf9a0 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.217+0000 7fd0ff400640 10 client.26498 handle_command_reply: tid=0 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.217+0000 7fd1277c1740 1 client.26498 shutdown 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.217+0000 7fd1277c1740 1 -- 172.21.15.7:0/4077850268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 msgr2=0x55fe54c719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.217+0000 7fd1277c1740 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c719c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd11009b680 tx=0x7fd110086040 comp rx=0 tx=0).stop 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 1 -- 172.21.15.7:0/4077850268 shutdown_connections 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.203:6836/2344597644,v1:172.21.15.203:6837/2344597644] conn(0x55fe54cdf9a0 0x55fe54ce1dc0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fe54c68420 0x55fe54c72f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55fe54c632b0 0x55fe54c71f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 1 --2- 172.21.15.7:0/4077850268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fe54c628c0 0x55fe54c719c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 1 -- 172.21.15.7:0/4077850268 >> 172.21.15.7:0/4077850268 conn(0x55fe54c5de30 msgr2=0x55fe54c5d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd0ff400640 0 client.26498 ms_handle_reset on v2:172.21.15.203:6836/2344597644 2024-03-18T01:47:37.221 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 1 -- 172.21.15.7:0/4077850268 shutdown_connections 2024-03-18T01:47:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 1 -- 172.21.15.7:0/4077850268 wait complete. 2024-03-18T01:47:37.222 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.218+0000 7fd1277c1740 20 client.26498 trim_cache size 0 max 16384 2024-03-18T01:47:37.222 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 -- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1040a2880 msgr2=0x7fd1040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1040a2880 0x7fd1040a4d40 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd12019f560 tx=0x7fd10c005f90 comp rx=0 tx=0).stop 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 -- 172.21.15.7:0/3101475779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120074240 msgr2=0x7fd12019e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120074240 0x7fd12019e050 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fd11000e9f0 tx=0x7fd11000eeb0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 -- 172.21.15.7:0/3101475779 shutdown_connections 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fd1040a2880 0x7fd1040a4d40 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fd12010ddc0 0x7fd1201a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd120075820 0x7fd12019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 --2- 172.21.15.7:0/3101475779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd120074240 0x7fd12019e050 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 -- 172.21.15.7:0/3101475779 >> 172.21.15.7:0/3101475779 conn(0x7fd1200fcd20 msgr2=0x7fd120101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 -- 172.21.15.7:0/3101475779 shutdown_connections 2024-03-18T01:47:37.223 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.219+0000 7fd124a00640 1 -- 172.21.15.7:0/3101475779 wait complete. 2024-03-18T01:47:37.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T01:47:37.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.449+0000 7f1dd9c00640 1 -- 172.21.15.7:0/648102433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 msgr2=0x7f1dd4107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.450 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.449+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/648102433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 0x7f1dd4107000 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009900 tx=0x7f1dc002bc30 comp rx=0 tx=0).stop 2024-03-18T01:47:37.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.450+0000 7f1dd9c00640 1 -- 172.21.15.7:0/648102433 shutdown_connections 2024-03-18T01:47:37.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.450+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/648102433 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd4107540 0x7f1dd4109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.450+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/648102433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 0x7f1dd4107000 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.450+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/648102433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd41022e0 0x7f1dd41046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.450+0000 7f1dd9c00640 1 -- 172.21.15.7:0/648102433 >> 172.21.15.7:0/648102433 conn(0x7f1dd40fbc50 msgr2=0x7f1dd40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.450+0000 7f1dd9c00640 1 -- 172.21.15.7:0/648102433 shutdown_connections 2024-03-18T01:47:37.451 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.450+0000 7f1dd9c00640 1 -- 172.21.15.7:0/648102433 wait complete. 2024-03-18T01:47:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd9c00640 1 Processor -- start 2024-03-18T01:47:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd9c00640 1 -- start start 2024-03-18T01:47:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd9c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd41022e0 0x7f1dd4199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd9c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 0x7f1dd419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd9c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd4107540 0x7f1dd41a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.453 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd9c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd410d4a0 con 0x7f1dd4107540 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd9c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd410d320 con 0x7f1dd41022e0 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd9c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd410d620 con 0x7f1dd4104c10 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd41022e0 0x7f1dd4199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd2000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 0x7f1dd419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd41022e0 0x7f1dd4199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52630/0 (socket says 172.21.15.7:52630) 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd2a00640 1 -- 172.21.15.7:0/2413504101 learned_addr learned my addr 172.21.15.7:0/2413504101 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.451+0000 7f1dd3400640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd4107540 0x7f1dd41a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.452+0000 7f1dd2a00640 1 -- 172.21.15.7:0/2413504101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 msgr2=0x7f1dd419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.452+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 0x7f1dd419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.452+0000 7f1dd2a00640 1 -- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd4107540 msgr2=0x7f1dd41a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.452+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd4107540 0x7f1dd41a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.452+0000 7f1dd2a00640 1 -- 172.21.15.7:0/2413504101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd41a19c0 con 0x7f1dd41022e0 2024-03-18T01:47:37.454 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.452+0000 7f1dd2000640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 0x7f1dd419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:37.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.452+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd41022e0 0x7f1dd4199db0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400e9f0 tx=0x7f1dc400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.453+0000 7f1dcb400640 1 -- 172.21.15.7:0/2413504101 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc400cd80 con 0x7f1dd41022e0 2024-03-18T01:47:37.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.453+0000 7f1dcb400640 1 -- 172.21.15.7:0/2413504101 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc4005500 con 0x7f1dd41022e0 2024-03-18T01:47:37.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.453+0000 7f1dcb400640 1 -- 172.21.15.7:0/2413504101 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc4010690 con 0x7f1dd41022e0 2024-03-18T01:47:37.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.453+0000 7f1dd9c00640 1 -- 172.21.15.7:0/2413504101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd41a1cb0 con 0x7f1dd41022e0 2024-03-18T01:47:37.455 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.453+0000 7f1dd9c00640 1 -- 172.21.15.7:0/2413504101 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd41a21b0 con 0x7f1dd41022e0 2024-03-18T01:47:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.456+0000 7f1ddc870740 1 Processor -- start 2024-03-18T01:47:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.456+0000 7f1ddc870740 1 -- start start 2024-03-18T01:47:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.456+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c41f8c0 0x55e39c41fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c4202b0 0x55e39c424db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4278b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1ddc870740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e39c3e73b0 con 0x55e39c41f8c0 2024-03-18T01:47:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1ddc870740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e39c384450 con 0x55e39c425420 2024-03-18T01:47:37.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1ddc870740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e39c32ccb0 con 0x55e39c4202b0 2024-03-18T01:47:37.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c41f8c0 0x55e39c41fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4278b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.458 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd3400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c41f8c0 0x55e39c41fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39242/0 (socket says 172.21.15.7:39242) 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4278b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52632/0 (socket says 172.21.15.7:52632) 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd3400640 1 -- 172.21.15.7:0/1221607710 learned_addr learned my addr 172.21.15.7:0/1221607710 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd2000640 1 -- 172.21.15.7:0/1221607710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c4202b0 msgr2=0x55e39c424db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd2000640 1 --2- 172.21.15.7:0/1221607710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c4202b0 0x55e39c424db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd2000640 1 -- 172.21.15.7:0/1221607710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c41f8c0 msgr2=0x55e39c41fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd2000640 1 --2- 172.21.15.7:0/1221607710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c41f8c0 0x55e39c41fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.457+0000 7f1dd2000640 1 -- 172.21.15.7:0/1221607710 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e39c34e1f0 con 0x55e39c425420 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.458+0000 7f1dd2000640 1 --2- 172.21.15.7:0/1221607710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4278b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009230 tx=0x7f1dc002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=debae7d206ab710 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.458+0000 7f1dcb400640 1 -- 172.21.15.7:0/2413504101 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f1dc40046c0 con 0x7f1dd41022e0 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.458+0000 7f1dbfe00640 1 -- 172.21.15.7:0/1221607710 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0045070 con 0x55e39c425420 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.458+0000 7f1dbfe00640 1 -- 172.21.15.7:0/1221607710 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc002cd30 con 0x55e39c425420 2024-03-18T01:47:37.459 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.458+0000 7f1dbfe00640 1 -- 172.21.15.7:0/1221607710 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc00050d0 con 0x55e39c425420 2024-03-18T01:47:37.460 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 -- 172.21.15.7:0/1221607710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 msgr2=0x55e39c4278b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 --2- 172.21.15.7:0/1221607710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4278b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009230 tx=0x7f1dc002c6c0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 -- 172.21.15.7:0/1221607710 shutdown_connections 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 --2- 172.21.15.7:0/1221607710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4278b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 --2- 172.21.15.7:0/1221607710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c4202b0 0x55e39c424db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 --2- 172.21.15.7:0/1221607710 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c41f8c0 0x55e39c41fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 -- 172.21.15.7:0/1221607710 >> 172.21.15.7:0/1221607710 conn(0x55e39c41ae30 msgr2=0x55e39c41b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 -- 172.21.15.7:0/1221607710 shutdown_connections 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.459+0000 7f1ddc870740 1 -- 172.21.15.7:0/1221607710 wait complete. 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.460+0000 7f1ddc870740 1 -- start start 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.460+0000 7f1dcb400640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1db80a2790 0x7f1db80a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.460+0000 7f1dcb400640 1 -- 172.21.15.7:0/2413504101 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1dc40cf210 con 0x7f1dd41022e0 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.460+0000 7f1ddc870740 1 Processor -- start 2024-03-18T01:47:37.461 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.460+0000 7f1dd3400640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1db80a2790 0x7f1db80a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c429b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1ddc870740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x55e39c340110 con 0x55e39c4202b0 2024-03-18T01:47:37.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1ddc870740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x55e39c3804f0 con 0x55e39c425420 2024-03-18T01:47:37.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1ddc870740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e39c428370 con 0x55e39c41f8c0 2024-03-18T01:47:37.464 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1ddc870740 10 client.0 _ll_register_callbacks cb 0x55e39c32db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1dd2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c429b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1dd2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c429b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58932/0 (socket says 172.21.15.7:58932) 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1dd2a00640 1 -- 172.21.15.7:0/3815766591 learned_addr learned my addr 172.21.15.7:0/3815766591 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1dd2000640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.461+0000 7f1dd2a00640 1 -- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 msgr2=0x55e39c4310f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd3400640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4310f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd2a00640 1 -- 172.21.15.7:0/3815766591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 msgr2=0x55e39c42a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd2a00640 1 -- 172.21.15.7:0/3815766591 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e39c0983f0 con 0x55e39c41f8c0 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd2000640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd3400640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4310f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:47:37.465 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c429b70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c6a0 tx=0x7f1dc409b150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=107c0dd53ef092c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dbcc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dd3400640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1db80a2790 0x7f1db80a4c50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x55e39c420b10 tx=0x7f1dcc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3815766591 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40b4060 con 0x55e39c41f8c0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3815766591 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1dc40afbd0 con 0x55e39c41f8c0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1ddc870740 1 -- 172.21.15.7:0/3815766591 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dc002cd30 con 0x55e39c41f8c0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3815766591 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f1dc40b9050 con 0x55e39c41f8c0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1ddc870740 10 client.26491 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.462+0000 7f1ddc870740 1 -- 172.21.15.7:0/3815766591 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x55e39c1627d0 con 0x55e39c41f8c0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3815766591 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc002cd30 con 0x55e39c41f8c0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3815766591 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f1dc40bb070 con 0x55e39c41f8c0 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1ddc870740 10 client.26491 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1ddc870740 10 client.26491 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1ddc870740 20 client.26491 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:37.466 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55e39c435a50 0x55e39c437eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1ddc870740 4 client.26491 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:37.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1ddc870740 1 -- 172.21.15.7:0/3815766591 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e39c391990 con 0x55e39c435a50 2024-03-18T01:47:37.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.463+0000 7f1dd2000640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55e39c435a50 0x55e39c437eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.464+0000 7f1dd2000640 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55e39c435a50 0x55e39c437eb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ceda6f1511ff3fbd server_cookie=eded7c374d3a14bc in_seq=0 out_seq=0 2024-03-18T01:47:37.467 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.464+0000 7f1dbcc00640 10 client.26491 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3815766591 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e39c391990 con 0x55e39c435a50 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1dbcc00640 10 client.26491 handle_command_reply: tid=0 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 client.26491 shutdown 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 -- 172.21.15.7:0/3815766591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 msgr2=0x55e39c429b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c429b70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400c6a0 tx=0x7f1dc409b150 comp rx=0 tx=0).stop 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 -- 172.21.15.7:0/3815766591 shutdown_connections 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55e39c435a50 0x55e39c437eb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c4310f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.468 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3815766591 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c429b70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 -- 172.21.15.7:0/3815766591 >> 172.21.15.7:0/3815766591 conn(0x55e39c41ae30 msgr2=0x55e39c41a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1dbcc00640 0 client.26491 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:47:37.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 -- 172.21.15.7:0/3815766591 shutdown_connections 2024-03-18T01:47:37.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 1 -- 172.21.15.7:0/3815766591 wait complete. 2024-03-18T01:47:37.469 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.465+0000 7f1ddc870740 20 client.26491 trim_cache size 0 max 16384 2024-03-18T01:47:37.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.491+0000 7f1ddc870740 1 Processor -- start 2024-03-18T01:47:37.491 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.491+0000 7f1ddc870740 1 -- start start 2024-03-18T01:47:37.492 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.491+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c41f8c0 0x55e39c41b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.491+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c419960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.491+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c425420 0x55e39c419ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.491+0000 7f1ddc870740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd41ac480 con 0x55e39c4202b0 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.491+0000 7f1ddc870740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd41ae0e0 con 0x55e39c41f8c0 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.491+0000 7f1ddc870740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd41acb80 con 0x55e39c425420 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c41f8c0 0x55e39c41b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c41f8c0 0x55e39c41b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52646/0 (socket says 172.21.15.7:52646) 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2a00640 1 -- 172.21.15.7:0/1889713978 learned_addr learned my addr 172.21.15.7:0/1889713978 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2000640 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c419960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd3400640 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c425420 0x55e39c419ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2a00640 1 -- 172.21.15.7:0/1889713978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c425420 msgr2=0x55e39c419ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c425420 0x55e39c419ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.493 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2a00640 1 -- 172.21.15.7:0/1889713978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 msgr2=0x55e39c419960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c419960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.492+0000 7f1dd2a00640 1 -- 172.21.15.7:0/1889713978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e39c416c10 con 0x55e39c41f8c0 2024-03-18T01:47:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.493+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c41f8c0 0x55e39c41b210 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40097d0 tx=0x7f1dc409b280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43531031f74d901c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.493+0000 7f1dbf400640 1 -- 172.21.15.7:0/1889713978 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40b4040 con 0x55e39c41f8c0 2024-03-18T01:47:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.493+0000 7f1dbf400640 1 -- 172.21.15.7:0/1889713978 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc409b7f0 con 0x55e39c41f8c0 2024-03-18T01:47:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.493+0000 7f1dbf400640 1 -- 172.21.15.7:0/1889713978 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40a3f90 con 0x55e39c41f8c0 2024-03-18T01:47:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 -- 172.21.15.7:0/1889713978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c41f8c0 msgr2=0x55e39c41b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c41f8c0 0x55e39c41b210 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40097d0 tx=0x7f1dc409b280 comp rx=0 tx=0).stop 2024-03-18T01:47:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 -- 172.21.15.7:0/1889713978 shutdown_connections 2024-03-18T01:47:37.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c425420 0x55e39c419ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c419960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 --2- 172.21.15.7:0/1889713978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c41f8c0 0x55e39c41b210 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 -- 172.21.15.7:0/1889713978 >> 172.21.15.7:0/1889713978 conn(0x55e39c41ae30 msgr2=0x55e39c416080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 -- 172.21.15.7:0/1889713978 shutdown_connections 2024-03-18T01:47:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 -- 172.21.15.7:0/1889713978 wait complete. 2024-03-18T01:47:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.494+0000 7f1ddc870740 1 -- start start 2024-03-18T01:47:37.496 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1ddc870740 1 Processor -- start 2024-03-18T01:47:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c42ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1ddc870740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c42ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1ddc870740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd41ad9b0 con 0x55e39c4202b0 2024-03-18T01:47:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1ddc870740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd41ad280 con 0x55e39c425420 2024-03-18T01:47:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1ddc870740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd41aa460 con 0x55e39c41f8c0 2024-03-18T01:47:37.499 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1ddc870740 10 client.0 _ll_register_callbacks cb 0x7f1dc4009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.495+0000 7f1dd2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c42ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c42ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c42ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58964/0 (socket says 172.21.15.7:58964) 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2a00640 1 -- 172.21.15.7:0/3585258320 learned_addr learned my addr 172.21.15.7:0/3585258320 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2a00640 1 -- 172.21.15.7:0/3585258320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 msgr2=0x55e39c42ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c42ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2a00640 1 -- 172.21.15.7:0/3585258320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 msgr2=0x55e39c42ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2a00640 1 -- 172.21.15.7:0/3585258320 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e39c430540 con 0x55e39c41f8c0 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dd2a00640 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c42ea00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40a3a60 tx=0x7f1dc4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4a6b71d87e7cd85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dbcc00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3585258320 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dc40a3f90 con 0x55e39c41f8c0 2024-03-18T01:47:37.500 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.496+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3585258320 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1dc409add0 con 0x55e39c41f8c0 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 1 -- 172.21.15.7:0/3585258320 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dc409b7f0 con 0x55e39c41f8c0 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3585258320 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f1dc40afbd0 con 0x55e39c41f8c0 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 10 client.26497 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 1 -- 172.21.15.7:0/3585258320 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x55e39c49c300 con 0x55e39c41f8c0 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3585258320 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dc409b7f0 con 0x55e39c41f8c0 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3585258320 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f1dc40bb040 con 0x55e39c41f8c0 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 10 client.26497 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 10 client.26497 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 20 client.26497 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55e39c49cab0 0x55e39c49eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 4 client.26497 mds_command: new command op to 24542 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.497+0000 7f1ddc870740 1 -- 172.21.15.7:0/3585258320 --> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e39c38bce0 con 0x55e39c49cab0 2024-03-18T01:47:37.501 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1dd2000640 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55e39c49cab0 0x55e39c49eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1dd2000640 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55e39c49cab0 0x55e39c49eed0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7bc29b805bd57bd6 server_cookie=6129d3b2fd6d76a7 in_seq=0 out_seq=0 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1dbcc00640 10 client.26497 ms_handle_connect on v2:172.21.15.46:6836/1509246906 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1dbcc00640 1 -- 172.21.15.7:0/3585258320 <== mds.3 v2:172.21.15.46:6836/1509246906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e39c38bce0 con 0x55e39c49cab0 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1dbcc00640 10 client.26497 handle_command_reply: tid=0 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1ddc870740 1 client.26497 shutdown 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1ddc870740 1 -- 172.21.15.7:0/3585258320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 msgr2=0x55e39c42ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c42ea00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40a3a60 tx=0x7f1dc4004a40 comp rx=0 tx=0).stop 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1ddc870740 1 -- 172.21.15.7:0/3585258320 shutdown_connections 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.498+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.46:6836/1509246906,v1:172.21.15.46:6837/1509246906] conn(0x55e39c49cab0 0x55e39c49eed0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e39c425420 0x55e39c42ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55e39c4202b0 0x55e39c42ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1dbcc00640 0 client.26497 ms_handle_reset on v2:172.21.15.46:6836/1509246906 2024-03-18T01:47:37.502 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1ddc870740 1 --2- 172.21.15.7:0/3585258320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e39c41f8c0 0x55e39c42ea00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1ddc870740 1 -- 172.21.15.7:0/3585258320 >> 172.21.15.7:0/3585258320 conn(0x55e39c41ae30 msgr2=0x55e39c416080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1ddc870740 1 -- 172.21.15.7:0/3585258320 shutdown_connections 2024-03-18T01:47:37.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1ddc870740 1 -- 172.21.15.7:0/3585258320 wait complete. 2024-03-18T01:47:37.503 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1ddc870740 20 client.26497 trim_cache size 0 max 16384 2024-03-18T01:47:37.503 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1dd9c00640 1 -- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1db80a2790 msgr2=0x7f1db80a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1db80a2790 0x7f1db80a4c50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x55e39c420b10 tx=0x7f1dcc009500 comp rx=0 tx=0).stop 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1dd9c00640 1 -- 172.21.15.7:0/2413504101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd41022e0 msgr2=0x7f1dd4199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.499+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd41022e0 0x7f1dd4199db0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1dc400e9f0 tx=0x7f1dc400edf0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.500+0000 7f1dd9c00640 1 -- 172.21.15.7:0/2413504101 shutdown_connections 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.500+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f1db80a2790 0x7f1db80a4c50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.500+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f1dd4107540 0x7f1dd41a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.500+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1dd4104c10 0x7f1dd419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.504 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.500+0000 7f1dd9c00640 1 --2- 172.21.15.7:0/2413504101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1dd41022e0 0x7f1dd4199db0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.500+0000 7f1dd9c00640 1 -- 172.21.15.7:0/2413504101 >> 172.21.15.7:0/2413504101 conn(0x7f1dd40fbc50 msgr2=0x7f1dd4101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.500+0000 7f1dd9c00640 1 -- 172.21.15.7:0/2413504101 shutdown_connections 2024-03-18T01:47:37.505 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.500+0000 7f1dd9c00640 1 -- 172.21.15.7:0/2413504101 wait complete. 2024-03-18T01:47:37.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:37 smithi007 ceph-mon[31996]: pgmap v700: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-18T01:47:37.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:37 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2357921234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:47:37.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T01:47:37.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:37 smithi046 ceph-mon[36108]: pgmap v700: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-18T01:47:37.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:37 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2357921234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:47:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:37 smithi203 ceph-mon[36345]: pgmap v700: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-18T01:47:37.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:37 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2357921234' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 -- 172.21.15.7:0/4177296194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be4109670 msgr2=0x7f3be410bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 --2- 172.21.15.7:0/4177296194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be4109670 0x7f3be410bb10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc009ca0 tx=0x7f3bdc02c190 comp rx=0 tx=0).stop 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 -- 172.21.15.7:0/4177296194 shutdown_connections 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 --2- 172.21.15.7:0/4177296194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be4109670 0x7f3be410bb10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 --2- 172.21.15.7:0/4177296194 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be40741e0 0x7f3be4074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 --2- 172.21.15.7:0/4177296194 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be40757c0 0x7f3be4073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 -- 172.21.15.7:0/4177296194 >> 172.21.15.7:0/4177296194 conn(0x7f3be40fcd10 msgr2=0x7f3be40ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 -- 172.21.15.7:0/4177296194 shutdown_connections 2024-03-18T01:47:37.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.716+0000 7f3be9000640 1 -- 172.21.15.7:0/4177296194 wait complete. 2024-03-18T01:47:37.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.718+0000 7f3be9000640 1 Processor -- start 2024-03-18T01:47:37.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.718+0000 7f3be9000640 1 -- start start 2024-03-18T01:47:37.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.718+0000 7f3be9000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be40741e0 0x7f3be4199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.718+0000 7f3be9000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be40757c0 0x7f3be419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.718+0000 7f3be9000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4109670 0x7f3be41a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.718+0000 7f3be9000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3be410f640 con 0x7f3be40757c0 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.718+0000 7f3be9000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3be410f4c0 con 0x7f3be40741e0 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.718+0000 7f3be9000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3be410f7c0 con 0x7f3be4109670 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be40741e0 0x7f3be4199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be40741e0 0x7f3be4199cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52680/0 (socket says 172.21.15.7:52680) 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 -- 172.21.15.7:0/3326526782 learned_addr learned my addr 172.21.15.7:0/3326526782 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be2800640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4109670 0x7f3be41a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1400640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be40757c0 0x7f3be419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 -- 172.21.15.7:0/3326526782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4109670 msgr2=0x7f3be41a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4109670 0x7f3be41a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 -- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be40757c0 msgr2=0x7f3be419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be40757c0 0x7f3be419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.720 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 -- 172.21.15.7:0/3326526782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be41a1890 con 0x7f3be40741e0 2024-03-18T01:47:37.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be2800640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4109670 0x7f3be41a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:37.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.719+0000 7f3be1e00640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be40741e0 0x7f3be4199cd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400ea60 tx=0x7f3bd400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.720+0000 7f3bdaa00640 1 -- 172.21.15.7:0/3326526782 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400cde0 con 0x7f3be40741e0 2024-03-18T01:47:37.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.720+0000 7f3bdaa00640 1 -- 172.21.15.7:0/3326526782 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd40035f0 con 0x7f3be40741e0 2024-03-18T01:47:37.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.720+0000 7f3be9000640 1 -- 172.21.15.7:0/3326526782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be41a1b80 con 0x7f3be40741e0 2024-03-18T01:47:37.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.720+0000 7f3bdaa00640 1 -- 172.21.15.7:0/3326526782 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4010670 con 0x7f3be40741e0 2024-03-18T01:47:37.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.720+0000 7f3be9000640 1 -- 172.21.15.7:0/3326526782 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be41a2120 con 0x7f3be40741e0 2024-03-18T01:47:37.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.724+0000 7f3bebda0740 1 Processor -- start 2024-03-18T01:47:37.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3bdaa00640 1 -- 172.21.15.7:0/3326526782 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f3bd4004020 con 0x7f3be40741e0 2024-03-18T01:47:37.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3bebda0740 1 -- start start 2024-03-18T01:47:37.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73d8c0 0x561efb73dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73e2b0 0x561efb742db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.726 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 0x561efb7458b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3bebda0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561efb7053b0 con 0x561efb743420 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3bebda0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561efb6a2450 con 0x561efb73e2b0 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3bebda0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561efb64acb0 con 0x561efb73d8c0 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3be1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73e2b0 0x561efb742db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3be2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73d8c0 0x561efb73dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3be2800640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73d8c0 0x561efb73dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58978/0 (socket says 172.21.15.7:58978) 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.725+0000 7f3be2800640 1 -- 172.21.15.7:0/806453406 learned_addr learned my addr 172.21.15.7:0/806453406 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.726+0000 7f3be2800640 1 -- 172.21.15.7:0/806453406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73e2b0 msgr2=0x561efb742db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.726+0000 7f3be2800640 1 --2- 172.21.15.7:0/806453406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73e2b0 0x561efb742db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.726+0000 7f3be2800640 1 -- 172.21.15.7:0/806453406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 msgr2=0x561efb7458b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.726+0000 7f3be2800640 1 --2- 172.21.15.7:0/806453406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 0x561efb7458b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.726+0000 7f3be2800640 1 -- 172.21.15.7:0/806453406 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561efb66c1f0 con 0x561efb73d8c0 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.726+0000 7f3be1e00640 1 --2- 172.21.15.7:0/806453406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73e2b0 0x561efb742db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:37.727 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.726+0000 7f3be2800640 1 --2- 172.21.15.7:0/806453406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73d8c0 0x561efb73dce0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc02c7a0 tx=0x7f3bdc005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=749e2c1a264bcd61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.727+0000 7f3bcf400640 1 -- 172.21.15.7:0/806453406 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc043070 con 0x561efb73d8c0 2024-03-18T01:47:37.728 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.727+0000 7f3bcf400640 1 -- 172.21.15.7:0/806453406 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bdc003e50 con 0x561efb73d8c0 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.727+0000 7f3bcf400640 1 -- 172.21.15.7:0/806453406 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc039040 con 0x561efb73d8c0 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.727+0000 7f3bdaa00640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3bc80a2880 0x7f3bc80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.727+0000 7f3bdaa00640 1 -- 172.21.15.7:0/3326526782 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3bd40cf0f0 con 0x7f3be40741e0 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.727+0000 7f3be1e00640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3bc80a2880 0x7f3bc80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.727+0000 7f3bebda0740 1 -- 172.21.15.7:0/806453406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73d8c0 msgr2=0x561efb73dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.727+0000 7f3bebda0740 1 --2- 172.21.15.7:0/806453406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73d8c0 0x561efb73dce0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc02c7a0 tx=0x7f3bdc005f80 comp rx=0 tx=0).stop 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.728+0000 7f3bebda0740 1 -- 172.21.15.7:0/806453406 shutdown_connections 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.728+0000 7f3bebda0740 1 --2- 172.21.15.7:0/806453406 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 0x561efb7458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.728+0000 7f3bebda0740 1 --2- 172.21.15.7:0/806453406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73e2b0 0x561efb742db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.728+0000 7f3bebda0740 1 --2- 172.21.15.7:0/806453406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73d8c0 0x561efb73dce0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.728+0000 7f3bebda0740 1 -- 172.21.15.7:0/806453406 >> 172.21.15.7:0/806453406 conn(0x561efb738e30 msgr2=0x561efb739230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.728+0000 7f3bebda0740 1 -- 172.21.15.7:0/806453406 shutdown_connections 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.728+0000 7f3bebda0740 1 -- 172.21.15.7:0/806453406 wait complete. 2024-03-18T01:47:37.729 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.728+0000 7f3be1e00640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3bc80a2880 0x7f3bc80a4d40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40ccc00 tx=0x7f3bd40d2bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.729+0000 7f3bebda0740 1 -- start start 2024-03-18T01:47:37.732 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.729+0000 7f3bebda0740 1 Processor -- start 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.729+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb747b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.729+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb748070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.729+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 0x561efb74f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.729+0000 7f3bebda0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x561efb65e110 con 0x561efb73d8c0 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.729+0000 7f3bebda0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561efb69e4f0 con 0x561efb743420 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.729+0000 7f3bebda0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561efb746370 con 0x561efb73e2b0 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3bebda0740 10 client.0 _ll_register_callbacks cb 0x561efb64bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb748070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb748070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58982/0 (socket says 172.21.15.7:58982) 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be1400640 1 -- 172.21.15.7:0/323668782 learned_addr learned my addr 172.21.15.7:0/323668782 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be2800640 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 0x561efb74f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.733 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be2800640 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb747b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be1400640 1 -- 172.21.15.7:0/323668782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 msgr2=0x561efb74f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be1400640 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 0x561efb74f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be1400640 1 -- 172.21.15.7:0/323668782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 msgr2=0x561efb747b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be1400640 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb747b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be1400640 1 -- 172.21.15.7:0/323668782 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561efb3b63f0 con 0x561efb73e2b0 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.730+0000 7f3be2800640 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb747b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.731+0000 7f3be1400640 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb748070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000be00 tx=0x7f3bd00107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=774b2d06f3c75ba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.731+0000 7f3bc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:37.734 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.731+0000 7f3bc3e00640 1 -- 172.21.15.7:0/323668782 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0011850 con 0x561efb73e2b0 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.731+0000 7f3bc3e00640 1 -- 172.21.15.7:0/323668782 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3bd00119f0 con 0x561efb73e2b0 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.731+0000 7f3bebda0740 1 -- 172.21.15.7:0/323668782 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bdc003e50 con 0x561efb73e2b0 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.731+0000 7f3bc3e00640 1 -- 172.21.15.7:0/323668782 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f3bd0025050 con 0x561efb73e2b0 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.731+0000 7f3bebda0740 10 client.26509 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.731+0000 7f3bebda0740 1 -- 172.21.15.7:0/323668782 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x561efb4807d0 con 0x561efb73e2b0 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3bc3e00640 1 -- 172.21.15.7:0/323668782 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bdc003e50 con 0x561efb73e2b0 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3bc3e00640 1 -- 172.21.15.7:0/323668782 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f3bd001a380 con 0x561efb73e2b0 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3bebda0740 10 client.26509 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3bebda0740 10 client.26509 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3bebda0740 20 client.26509 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3bebda0740 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x561efb753a40 0x561efb755ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.735 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3bebda0740 4 client.26509 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:37.736 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3bebda0740 1 -- 172.21.15.7:0/323668782 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561efb6af990 con 0x561efb753a40 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.732+0000 7f3be2800640 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x561efb753a40 0x561efb755ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.733+0000 7f3bc3e00640 10 client.26509 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.733+0000 7f3be2800640 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x561efb753a40 0x561efb755ec0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5693a7a895b440 server_cookie=d839a66000d0a608 in_seq=0 out_seq=0 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bc3e00640 1 -- 172.21.15.7:0/323668782 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561efb6af990 con 0x561efb753a40 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bc3e00640 10 client.26509 handle_command_reply: tid=0 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 client.26509 shutdown 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 -- 172.21.15.7:0/323668782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 msgr2=0x561efb748070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb748070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000be00 tx=0x7f3bd00107a0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.737 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 -- 172.21.15.7:0/323668782 shutdown_connections 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 0x561efb74f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x561efb753a40 0x561efb755ec0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb748070 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 --2- 172.21.15.7:0/323668782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb747b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 -- 172.21.15.7:0/323668782 >> 172.21.15.7:0/323668782 conn(0x561efb738e30 msgr2=0x561efb7381f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bc3e00640 0 client.26509 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 -- 172.21.15.7:0/323668782 shutdown_connections 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.734+0000 7f3bebda0740 1 -- 172.21.15.7:0/323668782 wait complete. 2024-03-18T01:47:37.738 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.735+0000 7f3bebda0740 20 client.26509 trim_cache size 0 max 16384 2024-03-18T01:47:37.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.759+0000 7f3bebda0740 1 Processor -- start 2024-03-18T01:47:37.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3bebda0740 1 -- start start 2024-03-18T01:47:37.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb746770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 0x561efb746cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3bebda0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41ac430 con 0x561efb73d8c0 2024-03-18T01:47:37.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3bebda0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41ae090 con 0x561efb743420 2024-03-18T01:47:37.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3bebda0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41acb30 con 0x561efb73e2b0 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3be1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3be1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58984/0 (socket says 172.21.15.7:58984) 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3be1400640 1 -- 172.21.15.7:0/3492019341 learned_addr learned my addr 172.21.15.7:0/3492019341 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.760+0000 7f3be2800640 1 --2- 172.21.15.7:0/3492019341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb746770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3be1400640 1 -- 172.21.15.7:0/3492019341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 msgr2=0x561efb746cb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3be1400640 1 --2- 172.21.15.7:0/3492019341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 0x561efb746cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3be1400640 1 -- 172.21.15.7:0/3492019341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 msgr2=0x561efb746770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3be1400640 1 --2- 172.21.15.7:0/3492019341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb746770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.762 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3be1400640 1 -- 172.21.15.7:0/3492019341 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561efb748b20 con 0x561efb73e2b0 2024-03-18T01:47:37.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3be1400640 1 --2- 172.21.15.7:0/3492019341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74f070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0000c00 tx=0x7f3bd000ba70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=267720ea144a7cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3bcea00640 1 -- 172.21.15.7:0/3492019341 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0022040 con 0x561efb73e2b0 2024-03-18T01:47:37.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3bcea00640 1 -- 172.21.15.7:0/3492019341 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd0005520 con 0x561efb73e2b0 2024-03-18T01:47:37.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.761+0000 7f3bcea00640 1 -- 172.21.15.7:0/3492019341 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0011650 con 0x561efb73e2b0 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.762+0000 7f3bebda0740 1 -- 172.21.15.7:0/3492019341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 msgr2=0x561efb74f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.762+0000 7f3bebda0740 1 --2- 172.21.15.7:0/3492019341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74f070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0000c00 tx=0x7f3bd000ba70 comp rx=0 tx=0).stop 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.762+0000 7f3bebda0740 1 -- 172.21.15.7:0/3492019341 shutdown_connections 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.762+0000 7f3bebda0740 1 --2- 172.21.15.7:0/3492019341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb743420 0x561efb746cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.762+0000 7f3bebda0740 1 --2- 172.21.15.7:0/3492019341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74f070 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.762+0000 7f3bebda0740 1 --2- 172.21.15.7:0/3492019341 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb73d8c0 0x561efb746770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.762+0000 7f3bebda0740 1 -- 172.21.15.7:0/3492019341 >> 172.21.15.7:0/3492019341 conn(0x561efb738e30 msgr2=0x561efb739210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.763+0000 7f3bebda0740 1 -- 172.21.15.7:0/3492019341 shutdown_connections 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.763+0000 7f3bebda0740 1 -- 172.21.15.7:0/3492019341 wait complete. 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.763+0000 7f3bebda0740 1 -- start start 2024-03-18T01:47:37.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.763+0000 7f3bebda0740 1 Processor -- start 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73d8c0 0x561efb74c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3bebda0740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 0x561efb74d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3bebda0740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41ad960 con 0x561efb743420 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3bebda0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41ad230 con 0x561efb73d8c0 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3bebda0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3be41aa410 con 0x561efb73e2b0 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3bebda0740 10 client.0 _ll_register_callbacks cb 0x7f3bd000b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3be1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3be2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73d8c0 0x561efb74c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3be1400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:58990/0 (socket says 172.21.15.7:58990) 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3be1400640 1 -- 172.21.15.7:0/2101002114 learned_addr learned my addr 172.21.15.7:0/2101002114 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:37.766 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.764+0000 7f3be2800640 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 0x561efb74d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3be1400640 1 -- 172.21.15.7:0/2101002114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73d8c0 msgr2=0x561efb74c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3be1400640 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73d8c0 0x561efb74c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3be1400640 1 -- 172.21.15.7:0/2101002114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 msgr2=0x561efb74d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3be1400640 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 0x561efb74d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.767 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3be1400640 1 -- 172.21.15.7:0/2101002114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561efb74def0 con 0x561efb73e2b0 2024-03-18T01:47:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3be1400640 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74c8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3bd001a7d0 tx=0x7f3bd001ac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3c0cb247db221ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3bc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:37.769 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3bc3e00640 1 -- 172.21.15.7:0/2101002114 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0005270 con 0x561efb73e2b0 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3bc3e00640 1 -- 172.21.15.7:0/2101002114 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3bd0011650 con 0x561efb73e2b0 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.765+0000 7f3bebda0740 1 -- 172.21.15.7:0/2101002114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bd0005520 con 0x561efb73e2b0 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bc3e00640 1 -- 172.21.15.7:0/2101002114 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f3bd0041050 con 0x561efb73e2b0 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bebda0740 10 client.26515 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bebda0740 1 -- 172.21.15.7:0/2101002114 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x561efb74f1f0 con 0x561efb73e2b0 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bc3e00640 1 -- 172.21.15.7:0/2101002114 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd0005520 con 0x561efb73e2b0 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bc3e00640 1 -- 172.21.15.7:0/2101002114 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f3bd0003680 con 0x561efb73e2b0 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bebda0740 10 client.26515 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bebda0740 10 client.26515 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bebda0740 20 client.26515 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bebda0740 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x561efb7ba360 0x561efb7bc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bebda0740 4 client.26515 mds_command: new command op to 24544 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:37.770 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.766+0000 7f3bebda0740 1 -- 172.21.15.7:0/2101002114 --> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561efb6a9ce0 con 0x561efb7ba360 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.767+0000 7f3be2800640 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x561efb7ba360 0x561efb7bc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.767+0000 7f3be2800640 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x561efb7ba360 0x561efb7bc780 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=444bff73a1b13e8a server_cookie=ba3e6acc46b450d8 in_seq=0 out_seq=0 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.767+0000 7f3bc3e00640 10 client.26515 ms_handle_connect on v2:172.21.15.203:6838/3801010143 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.767+0000 7f3bc3e00640 1 -- 172.21.15.7:0/2101002114 <== mds.2 v2:172.21.15.203:6838/3801010143 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561efb6a9ce0 con 0x561efb7ba360 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.767+0000 7f3bc3e00640 10 client.26515 handle_command_reply: tid=0 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.767+0000 7f3bebda0740 1 client.26515 shutdown 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 -- 172.21.15.7:0/2101002114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 msgr2=0x561efb74c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74c8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3bd001a7d0 tx=0x7f3bd001ac60 comp rx=0 tx=0).stop 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 -- 172.21.15.7:0/2101002114 shutdown_connections 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x561efb743420 0x561efb74d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.203:6838/3801010143,v1:172.21.15.203:6839/3801010143] conn(0x561efb7ba360 0x561efb7bc780 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561efb73e2b0 0x561efb74c8f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 --2- 172.21.15.7:0/2101002114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561efb73d8c0 0x561efb74c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 -- 172.21.15.7:0/2101002114 >> 172.21.15.7:0/2101002114 conn(0x561efb738e30 msgr2=0x561efb746770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.771 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bc3e00640 0 client.26515 ms_handle_reset on v2:172.21.15.203:6838/3801010143 2024-03-18T01:47:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 -- 172.21.15.7:0/2101002114 shutdown_connections 2024-03-18T01:47:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 1 -- 172.21.15.7:0/2101002114 wait complete. 2024-03-18T01:47:37.772 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.768+0000 7f3bebda0740 20 client.26515 trim_cache size 0 max 16384 2024-03-18T01:47:37.772 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 -- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3bc80a2880 msgr2=0x7f3bc80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3bc80a2880 0x7f3bc80a4d40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40ccc00 tx=0x7f3bd40d2bd0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 -- 172.21.15.7:0/3326526782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be40741e0 msgr2=0x7f3be4199cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be40741e0 0x7f3be4199cd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400ea60 tx=0x7f3bd400ef20 comp rx=0 tx=0).stop 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 -- 172.21.15.7:0/3326526782 shutdown_connections 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3be4109670 0x7f3be41a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3bc80a2880 0x7f3bc80a4d40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3be40757c0 0x7f3be419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 --2- 172.21.15.7:0/3326526782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3be40741e0 0x7f3be4199cd0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 -- 172.21.15.7:0/3326526782 >> 172.21.15.7:0/3326526782 conn(0x7f3be40fcd10 msgr2=0x7f3be4105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 -- 172.21.15.7:0/3326526782 shutdown_connections 2024-03-18T01:47:37.773 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:37.769+0000 7f3be9000640 1 -- 172.21.15.7:0/3326526782 wait complete. 2024-03-18T01:47:37.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T01:47:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 -- 172.21.15.7:0/806103659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258108f70 msgr2=0x7f82581093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 --2- 172.21.15.7:0/806103659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258108f70 0x7f82581093f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8244009900 tx=0x7f824402bc30 comp rx=0 tx=0).stop 2024-03-18T01:47:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 -- 172.21.15.7:0/806103659 shutdown_connections 2024-03-18T01:47:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 --2- 172.21.15.7:0/806103659 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258109930 0x7f8258110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 --2- 172.21.15.7:0/806103659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258108f70 0x7f82581093f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 --2- 172.21.15.7:0/806103659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8258107e20 0x7f8258108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.006 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 -- 172.21.15.7:0/806103659 >> 172.21.15.7:0/806103659 conn(0x7f8258076de0 msgr2=0x7f8258079200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 -- 172.21.15.7:0/806103659 shutdown_connections 2024-03-18T01:47:38.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.005+0000 7f825fa00640 1 -- 172.21.15.7:0/806103659 wait complete. 2024-03-18T01:47:38.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825fa00640 1 Processor -- start 2024-03-18T01:47:38.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825fa00640 1 -- start start 2024-03-18T01:47:38.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825fa00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258107e20 0x7f825819e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825fa00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258108f70 0x7f825819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825fa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8258109930 0x7f82581a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825fa00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f8258113af0 con 0x7f8258108f70 2024-03-18T01:47:38.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825fa00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8258113970 con 0x7f8258109930 2024-03-18T01:47:38.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825fa00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8258113c70 con 0x7f8258107e20 2024-03-18T01:47:38.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258107e20 0x7f825819e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f8257e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258108f70 0x7f825819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258107e20 0x7f825819e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59004/0 (socket says 172.21.15.7:59004) 2024-03-18T01:47:38.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825d000640 1 -- 172.21.15.7:0/913189454 learned_addr learned my addr 172.21.15.7:0/913189454 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.007+0000 7f825da00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8258109930 0x7f82581a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.009 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f825d000640 1 -- 172.21.15.7:0/913189454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8258109930 msgr2=0x7f82581a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f825d000640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8258109930 0x7f82581a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f825d000640 1 -- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258108f70 msgr2=0x7f825819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f825d000640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258108f70 0x7f825819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f825d000640 1 -- 172.21.15.7:0/913189454 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82581a5cf0 con 0x7f8258107e20 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f825d000640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258107e20 0x7f825819e0d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f824c00e9f0 tx=0x7f824c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f8255600640 1 -- 172.21.15.7:0/913189454 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c00cd80 con 0x7f8258107e20 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f8255600640 1 -- 172.21.15.7:0/913189454 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f824c010910 con 0x7f8258107e20 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.008+0000 7f825fa00640 1 -- 172.21.15.7:0/913189454 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82581a5fe0 con 0x7f8258107e20 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.009+0000 7f8255600640 1 -- 172.21.15.7:0/913189454 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c018990 con 0x7f8258107e20 2024-03-18T01:47:38.010 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.009+0000 7f825fa00640 1 -- 172.21.15.7:0/913189454 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82581a6450 con 0x7f8258107e20 2024-03-18T01:47:38.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.013+0000 7f82627c9740 1 Processor -- start 2024-03-18T01:47:38.013 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.013+0000 7f82627c9740 1 -- start start 2024-03-18T01:47:38.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.013+0000 7f82627c9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500a8c0 0x55710500ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.013+0000 7f8255600640 1 -- 172.21.15.7:0/913189454 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f824c010ac0 con 0x7f8258107e20 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.013+0000 7f82627c9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x55710500fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.013+0000 7f82627c9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x5571050128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.013+0000 7f82627c9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557104fd23b0 con 0x55710500a8c0 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.013+0000 7f82627c9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557104f6f450 con 0x557105010420 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f82627c9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x557104f17cb0 con 0x55710500b2b0 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f825d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x55710500fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f8257e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x5571050128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f8257e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x5571050128b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52738/0 (socket says 172.21.15.7:52738) 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f8257e00640 1 -- 172.21.15.7:0/4294629011 learned_addr learned my addr 172.21.15.7:0/4294629011 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f825da00640 1 --2- 172.21.15.7:0/4294629011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500a8c0 0x55710500ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f825d000640 1 -- 172.21.15.7:0/4294629011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 msgr2=0x5571050128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f825d000640 1 --2- 172.21.15.7:0/4294629011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x5571050128b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f825d000640 1 -- 172.21.15.7:0/4294629011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500a8c0 msgr2=0x55710500ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f825d000640 1 --2- 172.21.15.7:0/4294629011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500a8c0 0x55710500ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.014+0000 7f825d000640 1 -- 172.21.15.7:0/4294629011 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557104f391f0 con 0x55710500b2b0 2024-03-18T01:47:38.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.015+0000 7f825d000640 1 --2- 172.21.15.7:0/4294629011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x55710500fdb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f824c0cc790 tx=0x7f824c0d76f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4286c54bb1ebf529 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.017 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.015+0000 7f824a000640 1 -- 172.21.15.7:0/4294629011 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c0e2070 con 0x55710500b2b0 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.015+0000 7f8255600640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82400a2880 0x7f82400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.015+0000 7f824a000640 1 -- 172.21.15.7:0/4294629011 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f824c0d12e0 con 0x55710500b2b0 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.015+0000 7f824a000640 1 -- 172.21.15.7:0/4294629011 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c0dd910 con 0x55710500b2b0 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.015+0000 7f8255600640 1 -- 172.21.15.7:0/913189454 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f824c0cf0b0 con 0x7f8258107e20 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f825da00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82400a2880 0x7f82400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 -- 172.21.15.7:0/4294629011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 msgr2=0x55710500fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 --2- 172.21.15.7:0/4294629011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x55710500fdb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f824c0cc790 tx=0x7f824c0d76f0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 -- 172.21.15.7:0/4294629011 shutdown_connections 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 --2- 172.21.15.7:0/4294629011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x5571050128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 --2- 172.21.15.7:0/4294629011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x55710500fdb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 --2- 172.21.15.7:0/4294629011 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500a8c0 0x55710500ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 -- 172.21.15.7:0/4294629011 >> 172.21.15.7:0/4294629011 conn(0x557105005e30 msgr2=0x557105006230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 -- 172.21.15.7:0/4294629011 shutdown_connections 2024-03-18T01:47:38.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.016+0000 7f82627c9740 1 -- 172.21.15.7:0/4294629011 wait complete. 2024-03-18T01:47:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.017+0000 7f825da00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82400a2880 0x7f82400a4d40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x55710500bb10 tx=0x7f8250009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.017+0000 7f82627c9740 1 -- start start 2024-03-18T01:47:38.019 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.017+0000 7f82627c9740 1 Processor -- start 2024-03-18T01:47:38.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f82627c9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105014be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f82627c9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500b2b0 0x557105015120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f82627c9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x55710501c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f82627c9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x557104f2b110 con 0x55710500b2b0 2024-03-18T01:47:38.021 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f82627c9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x557104f6b4f0 con 0x557105010420 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f82627c9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5571050133d0 con 0x55710500a8c0 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f82627c9740 10 client.0 _ll_register_callbacks cb 0x557104f18b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105014be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105014be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59016/0 (socket says 172.21.15.7:59016) 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f8257e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500b2b0 0x557105015120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825d000640 1 -- 172.21.15.7:0/2642934040 learned_addr learned my addr 172.21.15.7:0/2642934040 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825da00640 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x55710501c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825d000640 1 -- 172.21.15.7:0/2642934040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 msgr2=0x55710501c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825d000640 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x55710501c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825d000640 1 -- 172.21.15.7:0/2642934040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500b2b0 msgr2=0x557105015120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825d000640 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500b2b0 0x557105015120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.018+0000 7f825d000640 1 -- 172.21.15.7:0/2642934040 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557104c833f0 con 0x55710500a8c0 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.019+0000 7f825d000640 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105014be0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f824c00e9c0 tx=0x7f824c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=732493407fee507f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.019+0000 7f823ea00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.019+0000 7f823ea00640 1 -- 172.21.15.7:0/2642934040 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c0979e0 con 0x55710500a8c0 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.019+0000 7f823ea00640 1 -- 172.21.15.7:0/2642934040 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f824c0cc900 con 0x55710500a8c0 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.019+0000 7f82627c9740 1 -- 172.21.15.7:0/2642934040 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f824c0d12e0 con 0x55710500a8c0 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.019+0000 7f823ea00640 1 -- 172.21.15.7:0/2642934040 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f824c09b0a0 con 0x55710500a8c0 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.019+0000 7f82627c9740 10 client.26527 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.019+0000 7f82627c9740 1 -- 172.21.15.7:0/2642934040 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x557104d4d7d0 con 0x55710500a8c0 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f823ea00640 1 -- 172.21.15.7:0/2642934040 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f824c0d12e0 con 0x55710500a8c0 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f823ea00640 1 -- 172.21.15.7:0/2642934040 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f824c09b9a0 con 0x55710500a8c0 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f82627c9740 10 client.26527 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f82627c9740 10 client.26527 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f82627c9740 20 client.26527 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f82627c9740 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557105020650 0x557105022a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f82627c9740 4 client.26527 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f82627c9740 1 -- 172.21.15.7:0/2642934040 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557104f7c990 con 0x557105020650 2024-03-18T01:47:38.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.020+0000 7f8257e00640 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557105020650 0x557105022a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.021+0000 7f8257e00640 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557105020650 0x557105022a40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6a9d72bae33a90a9 server_cookie=bca45b5b224509a4 in_seq=0 out_seq=0 2024-03-18T01:47:38.024 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.021+0000 7f823ea00640 10 client.26527 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f823ea00640 1 -- 172.21.15.7:0/2642934040 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557104f7c990 con 0x557105020650 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f823ea00640 10 client.26527 handle_command_reply: tid=0 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 client.26527 shutdown 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 -- 172.21.15.7:0/2642934040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 msgr2=0x557105014be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105014be0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f824c00e9c0 tx=0x7f824c009310 comp rx=0 tx=0).stop 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 -- 172.21.15.7:0/2642934040 shutdown_connections 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557105020650 0x557105022a40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557105010420 0x55710501c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.025 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x55710500b2b0 0x557105015120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 --2- 172.21.15.7:0/2642934040 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105014be0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f823ea00640 0 client.26527 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:47:38.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.022+0000 7f82627c9740 1 -- 172.21.15.7:0/2642934040 >> 172.21.15.7:0/2642934040 conn(0x557105005e30 msgr2=0x5571050051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.023+0000 7f82627c9740 1 -- 172.21.15.7:0/2642934040 shutdown_connections 2024-03-18T01:47:38.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.023+0000 7f82627c9740 1 -- 172.21.15.7:0/2642934040 wait complete. 2024-03-18T01:47:38.026 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.023+0000 7f82627c9740 20 client.26527 trim_cache size 0 max 16384 2024-03-18T01:47:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.038+0000 7f82627c9740 1 Processor -- start 2024-03-18T01:47:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.038+0000 7f82627c9740 1 -- start start 2024-03-18T01:47:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.038+0000 7f82627c9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105015080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.038+0000 7f82627c9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500b2b0 0x55710501c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.038+0000 7f82627c9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x557105005070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.038+0000 7f82627c9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f82581b0920 con 0x557105010420 2024-03-18T01:47:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.038+0000 7f82627c9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82581b2580 con 0x55710500b2b0 2024-03-18T01:47:38.039 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.038+0000 7f82627c9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82581b1020 con 0x55710500a8c0 2024-03-18T01:47:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500b2b0 0x55710501c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f825d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105015080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500b2b0 0x55710501c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52768/0 (socket says 172.21.15.7:52768) 2024-03-18T01:47:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f825d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105015080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59018/0 (socket says 172.21.15.7:59018) 2024-03-18T01:47:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 -- 172.21.15.7:0/1170772825 learned_addr learned my addr 172.21.15.7:0/1170772825 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f825da00640 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x557105005070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.040 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 -- 172.21.15.7:0/1170772825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 msgr2=0x557105015080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105015080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 -- 172.21.15.7:0/1170772825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 msgr2=0x557105005070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x557105005070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 -- 172.21.15.7:0/1170772825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557105001c10 con 0x55710500b2b0 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f825da00640 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x557105005070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.039+0000 7f8257e00640 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500b2b0 0x55710501c120 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8244009230 tx=0x7f824402df90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2b376f61dc39d50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.040+0000 7f8249600640 1 -- 172.21.15.7:0/1170772825 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824404e070 con 0x55710500b2b0 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.040+0000 7f8249600640 1 -- 172.21.15.7:0/1170772825 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f824402cb40 con 0x55710500b2b0 2024-03-18T01:47:38.041 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.040+0000 7f8249600640 1 -- 172.21.15.7:0/1170772825 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824404a590 con 0x55710500b2b0 2024-03-18T01:47:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 -- 172.21.15.7:0/1170772825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500b2b0 msgr2=0x55710501c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500b2b0 0x55710501c120 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8244009230 tx=0x7f824402df90 comp rx=0 tx=0).stop 2024-03-18T01:47:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 -- 172.21.15.7:0/1170772825 shutdown_connections 2024-03-18T01:47:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x557105005070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500b2b0 0x55710501c120 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 --2- 172.21.15.7:0/1170772825 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500a8c0 0x557105015080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 -- 172.21.15.7:0/1170772825 >> 172.21.15.7:0/1170772825 conn(0x557105005e30 msgr2=0x557105004960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.042 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 -- 172.21.15.7:0/1170772825 shutdown_connections 2024-03-18T01:47:38.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 -- 172.21.15.7:0/1170772825 wait complete. 2024-03-18T01:47:38.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.041+0000 7f82627c9740 1 -- start start 2024-03-18T01:47:38.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f82627c9740 1 Processor -- start 2024-03-18T01:47:38.046 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f82627c9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500a8c0 0x557105019780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f82627c9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x557105019cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f82627c9740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x55710501ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f82627c9740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f82581b1e50 con 0x557105010420 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f82627c9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f82581b1720 con 0x55710500a8c0 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f82627c9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82581ae900 con 0x55710500b2b0 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f82627c9740 10 client.0 _ll_register_callbacks cb 0x7f824c0e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f825d000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500a8c0 0x557105019780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f825d000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500a8c0 0x557105019780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52780/0 (socket says 172.21.15.7:52780) 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f825d000640 1 -- 172.21.15.7:0/2973187444 learned_addr learned my addr 172.21.15.7:0/2973187444 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.042+0000 7f8257e00640 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x557105019cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f825da00640 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x55710501ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f825d000640 1 -- 172.21.15.7:0/2973187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 msgr2=0x557105019cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f825d000640 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x557105019cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f825d000640 1 -- 172.21.15.7:0/2973187444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 msgr2=0x55710501ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f825d000640 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x55710501ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f825d000640 1 -- 172.21.15.7:0/2973187444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55710501b270 con 0x55710500a8c0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f8257e00640 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x557105019cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f825d000640 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500a8c0 0x557105019780 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f824c009380 tx=0x7f824c009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726c2c6a6e0113e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f823ea00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f823ea00640 1 -- 172.21.15.7:0/2973187444 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c09b280 con 0x55710500a8c0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f823ea00640 1 -- 172.21.15.7:0/2973187444 <== mon.2 v2:172.21.15.46:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f824c09b9a0 con 0x55710500a8c0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f82627c9740 1 -- 172.21.15.7:0/2973187444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f824402cb40 con 0x55710500a8c0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f823ea00640 1 -- 172.21.15.7:0/2973187444 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f824c09b510 con 0x55710500a8c0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f82627c9740 10 client.26546 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.043+0000 7f82627c9740 1 -- 172.21.15.7:0/2973187444 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x5571050867e0 con 0x55710500a8c0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f823ea00640 1 -- 172.21.15.7:0/2973187444 <== mon.2 v2:172.21.15.46:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f824c0b8020 con 0x55710500a8c0 2024-03-18T01:47:38.048 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f823ea00640 1 -- 172.21.15.7:0/2973187444 <== mon.2 v2:172.21.15.46:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f824c0d2b60 con 0x55710500a8c0 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f82627c9740 10 client.26546 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f82627c9740 10 client.26546 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f82627c9740 20 client.26546 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f82627c9740 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557105086e00 0x557105089220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f82627c9740 4 client.26546 mds_command: new command op to 24562 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f82627c9740 1 -- 172.21.15.7:0/2973187444 --> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557104f76ce0 con 0x557105086e00 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.044+0000 7f8257e00640 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557105086e00 0x557105089220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f8257e00640 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557105086e00 0x557105089220 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41f2d83f54dc0974 server_cookie=484143364843606 in_seq=0 out_seq=0 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f823ea00640 10 client.26546 ms_handle_connect on v2:172.21.15.7:6838/3166631604 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f823ea00640 1 -- 172.21.15.7:0/2973187444 <== mds.1 v2:172.21.15.7:6838/3166631604 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557104f76ce0 con 0x557105086e00 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f823ea00640 10 client.26546 handle_command_reply: tid=0 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f82627c9740 1 client.26546 shutdown 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f82627c9740 1 -- 172.21.15.7:0/2973187444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500a8c0 msgr2=0x557105019780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f82627c9740 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500a8c0 0x557105019780 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f824c009380 tx=0x7f824c009700 comp rx=0 tx=0).stop 2024-03-18T01:47:38.049 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f82627c9740 1 -- 172.21.15.7:0/2973187444 shutdown_connections 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f82627c9740 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.7:6838/3166631604,v1:172.21.15.7:6839/3166631604] conn(0x557105086e00 0x557105089220 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f82627c9740 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x557105010420 0x55710501ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f82627c9740 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55710500b2b0 0x557105019cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.045+0000 7f82627c9740 1 --2- 172.21.15.7:0/2973187444 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55710500a8c0 0x557105019780 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.046+0000 7f82627c9740 1 -- 172.21.15.7:0/2973187444 >> 172.21.15.7:0/2973187444 conn(0x557105005e30 msgr2=0x557105014660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.046+0000 7f823ea00640 0 client.26546 ms_handle_reset on v2:172.21.15.7:6838/3166631604 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.046+0000 7f82627c9740 1 -- 172.21.15.7:0/2973187444 shutdown_connections 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.046+0000 7f82627c9740 1 -- 172.21.15.7:0/2973187444 wait complete. 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.046+0000 7f82627c9740 20 client.26546 trim_cache size 0 max 16384 2024-03-18T01:47:38.050 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:38.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 -- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82400a2880 msgr2=0x7f82400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82400a2880 0x7f82400a4d40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x55710500bb10 tx=0x7f8250009500 comp rx=0 tx=0).stop 2024-03-18T01:47:38.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 -- 172.21.15.7:0/913189454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258107e20 msgr2=0x7f825819e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258107e20 0x7f825819e0d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f824c00e9f0 tx=0x7f824c00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.051 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 -- 172.21.15.7:0/913189454 shutdown_connections 2024-03-18T01:47:38.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8258109930 0x7f82581a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f82400a2880 0x7f82400a4d40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f8258108f70 0x7f825819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 --2- 172.21.15.7:0/913189454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8258107e20 0x7f825819e0d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 -- 172.21.15.7:0/913189454 >> 172.21.15.7:0/913189454 conn(0x7f8258076de0 msgr2=0x7f825810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 -- 172.21.15.7:0/913189454 shutdown_connections 2024-03-18T01:47:38.052 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.047+0000 7f825fa00640 1 -- 172.21.15.7:0/913189454 wait complete. 2024-03-18T01:47:38.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:47:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 -- 172.21.15.7:0/1905158588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e64107590 msgr2=0x7f3e64109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/1905158588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e64107590 0x7f3e64109980 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c009be0 tx=0x7f3e5c02bef0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 -- 172.21.15.7:0/1905158588 shutdown_connections 2024-03-18T01:47:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/1905158588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e64107590 0x7f3e64109980 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/1905158588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e64104c60 0x7f3e64107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/1905158588 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e64102330 0x7f3e64104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 -- 172.21.15.7:0/1905158588 >> 172.21.15.7:0/1905158588 conn(0x7f3e640fbc40 msgr2=0x7f3e640fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 -- 172.21.15.7:0/1905158588 shutdown_connections 2024-03-18T01:47:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.315+0000 7f3e6ba00640 1 -- 172.21.15.7:0/1905158588 wait complete. 2024-03-18T01:47:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.317+0000 7f3e6ba00640 1 Processor -- start 2024-03-18T01:47:38.317 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.317+0000 7f3e6ba00640 1 -- start start 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.317+0000 7f3e6ba00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e64102330 0x7f3e64199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.317+0000 7f3e6ba00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e64104c60 0x7f3e6419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.317+0000 7f3e6ba00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e64107590 0x7f3e641a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.317+0000 7f3e6ba00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6410d4d0 con 0x7f3e64102330 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.317+0000 7f3e6ba00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6410d350 con 0x7f3e64107590 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.317+0000 7f3e6ba00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6410d650 con 0x7f3e64104c60 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e64102330 0x7f3e64199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e64102330 0x7f3e64199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39372/0 (socket says 172.21.15.7:39372) 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69000640 1 -- 172.21.15.7:0/921565353 learned_addr learned my addr 172.21.15.7:0/921565353 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e63e00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e64104c60 0x7f3e6419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69a00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e64107590 0x7f3e641a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69000640 1 -- 172.21.15.7:0/921565353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e64104c60 msgr2=0x7f3e6419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69000640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e64104c60 0x7f3e6419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69000640 1 -- 172.21.15.7:0/921565353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e64107590 msgr2=0x7f3e641a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69000640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e64107590 0x7f3e641a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.318+0000 7f3e69000640 1 -- 172.21.15.7:0/921565353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e641a18c0 con 0x7f3e64102330 2024-03-18T01:47:38.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.319+0000 7f3e69000640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e64102330 0x7f3e64199c80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800e9f0 tx=0x7f3e5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.319+0000 7f3e61600640 1 -- 172.21.15.7:0/921565353 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5800cd80 con 0x7f3e64102330 2024-03-18T01:47:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.319+0000 7f3e6ba00640 1 -- 172.21.15.7:0/921565353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e641a1bb0 con 0x7f3e64102330 2024-03-18T01:47:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.319+0000 7f3e61600640 1 -- 172.21.15.7:0/921565353 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e58010910 con 0x7f3e64102330 2024-03-18T01:47:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.319+0000 7f3e61600640 1 -- 172.21.15.7:0/921565353 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58018990 con 0x7f3e64102330 2024-03-18T01:47:38.321 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.319+0000 7f3e6ba00640 1 -- 172.21.15.7:0/921565353 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e641a2100 con 0x7f3e64102330 2024-03-18T01:47:38.322 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.322+0000 7f3e6e667740 1 Processor -- start 2024-03-18T01:47:38.323 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.322+0000 7f3e6e667740 1 -- start start 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a5f8c0 0x564b14a5fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a602b0 0x564b14a64db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a65420 0x564b14a678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e6e667740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564b14a273b0 con 0x564b14a602b0 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e6e667740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564b149c4450 con 0x564b14a5f8c0 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e6e667740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564b1496ccb0 con 0x564b14a65420 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e69a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a5f8c0 0x564b14a5fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e63e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a65420 0x564b14a678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e63e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a65420 0x564b14a678b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59052/0 (socket says 172.21.15.7:59052) 2024-03-18T01:47:38.324 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e69a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a5f8c0 0x564b14a5fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52808/0 (socket says 172.21.15.7:52808) 2024-03-18T01:47:38.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e63e00640 1 -- 172.21.15.7:0/4273283626 learned_addr learned my addr 172.21.15.7:0/4273283626 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e63e00640 1 -- 172.21.15.7:0/4273283626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a5f8c0 msgr2=0x564b14a5fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e63e00640 1 --2- 172.21.15.7:0/4273283626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a5f8c0 0x564b14a5fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e63e00640 1 -- 172.21.15.7:0/4273283626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a602b0 msgr2=0x564b14a64db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:47:38.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e63e00640 1 --2- 172.21.15.7:0/4273283626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a602b0 0x564b14a64db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.323+0000 7f3e63e00640 1 -- 172.21.15.7:0/4273283626 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b1498e1f0 con 0x564b14a65420 2024-03-18T01:47:38.325 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.324+0000 7f3e63e00640 1 --2- 172.21.15.7:0/4273283626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a65420 0x564b14a678b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009c60 tx=0x7f3e50011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b010f59d92be88e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.326 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.324+0000 7f3e56000640 1 -- 172.21.15.7:0/4273283626 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e50004220 con 0x564b14a65420 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.324+0000 7f3e56000640 1 -- 172.21.15.7:0/4273283626 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e50009e10 con 0x564b14a65420 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.324+0000 7f3e56000640 1 -- 172.21.15.7:0/4273283626 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e500198d0 con 0x564b14a65420 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.324+0000 7f3e61600640 1 -- 172.21.15.7:0/921565353 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f3e58010ac0 con 0x7f3e64102330 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 -- 172.21.15.7:0/4273283626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a65420 msgr2=0x564b14a678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 --2- 172.21.15.7:0/4273283626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a65420 0x564b14a678b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009c60 tx=0x7f3e50011810 comp rx=0 tx=0).stop 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 -- 172.21.15.7:0/4273283626 shutdown_connections 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 --2- 172.21.15.7:0/4273283626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a65420 0x564b14a678b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 --2- 172.21.15.7:0/4273283626 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a602b0 0x564b14a64db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 --2- 172.21.15.7:0/4273283626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a5f8c0 0x564b14a5fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 -- 172.21.15.7:0/4273283626 >> 172.21.15.7:0/4273283626 conn(0x564b14a5ae30 msgr2=0x564b14a5b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 -- 172.21.15.7:0/4273283626 shutdown_connections 2024-03-18T01:47:38.327 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.325+0000 7f3e6e667740 1 -- 172.21.15.7:0/4273283626 wait complete. 2024-03-18T01:47:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.326+0000 7f3e6e667740 1 -- start start 2024-03-18T01:47:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.326+0000 7f3e61600640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e4c0a2710 0x7f3e4c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.326+0000 7f3e6e667740 1 Processor -- start 2024-03-18T01:47:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.326+0000 7f3e61600640 1 -- 172.21.15.7:0/921565353 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e580cf170 con 0x7f3e64102330 2024-03-18T01:47:38.328 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.326+0000 7f3e69a00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e4c0a2710 0x7f3e4c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a69bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a71140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e6e667740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x564b14980110 con 0x564b14a5f8c0 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e6e667740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564b149c04f0 con 0x564b14a65420 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e6e667740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564b14a683d0 con 0x564b14a602b0 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e6e667740 10 client.0 _ll_register_callbacks cb 0x564b1496db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e69000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a69bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e69000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a69bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:39380/0 (socket says 172.21.15.7:39380) 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e69a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a71140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.327+0000 7f3e69000640 1 -- 172.21.15.7:0/1691516824 learned_addr learned my addr 172.21.15.7:0/1691516824 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.331 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e63e00640 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e69a00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e4c0a2710 0x7f3e4c0a4bd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c006010 tx=0x7f3e5c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e69000640 1 -- 172.21.15.7:0/1691516824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 msgr2=0x564b14a6a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e69000640 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e69000640 1 -- 172.21.15.7:0/1691516824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 msgr2=0x564b14a71140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e69000640 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a71140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e69000640 1 -- 172.21.15.7:0/1691516824 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b146d83f0 con 0x564b14a5f8c0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e69a00640 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a71140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e69000640 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a69bc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800e9c0 tx=0x7f3e5809b430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24354402b93398a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.328+0000 7f3e4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.7:3300/0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.329+0000 7f3e4aa00640 1 -- 172.21.15.7:0/1691516824 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e580a3ef0 con 0x564b14a5f8c0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.329+0000 7f3e4aa00640 1 -- 172.21.15.7:0/1691516824 <== mon.0 v2:172.21.15.7:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e58018b30 con 0x564b14a5f8c0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.329+0000 7f3e6e667740 1 -- 172.21.15.7:0/1691516824 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e50009e10 con 0x564b14a5f8c0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.329+0000 7f3e4aa00640 1 -- 172.21.15.7:0/1691516824 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f3e58018d30 con 0x564b14a5f8c0 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.329+0000 7f3e6e667740 10 client.16230 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:38.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.329+0000 7f3e6e667740 1 -- 172.21.15.7:0/1691516824 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x564b147a27d0 con 0x564b14a5f8c0 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.329+0000 7f3e4aa00640 1 -- 172.21.15.7:0/1691516824 <== mon.0 v2:172.21.15.7:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e50009e10 con 0x564b14a5f8c0 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.330+0000 7f3e4aa00640 1 -- 172.21.15.7:0/1691516824 <== mon.0 v2:172.21.15.7:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f3e58097670 con 0x564b14a5f8c0 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.330+0000 7f3e6e667740 10 client.16230 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.330+0000 7f3e6e667740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.330+0000 7f3e6e667740 20 client.16230 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.330+0000 7f3e6e667740 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564b14a75a40 0x564b14a77e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.330+0000 7f3e6e667740 4 client.16230 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.330+0000 7f3e6e667740 1 -- 172.21.15.7:0/1691516824 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b149d1990 con 0x564b14a75a40 2024-03-18T01:47:38.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.330+0000 7f3e63e00640 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564b14a75a40 0x564b14a77e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.331+0000 7f3e63e00640 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564b14a75a40 0x564b14a77e30 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=780b2861e8e595b5 server_cookie=2462abd902766059 in_seq=0 out_seq=0 2024-03-18T01:47:38.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.331+0000 7f3e4aa00640 10 client.16230 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:38.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.332+0000 7f3e4aa00640 1 -- 172.21.15.7:0/1691516824 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564b149d1990 con 0x564b14a75a40 2024-03-18T01:47:38.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.332+0000 7f3e4aa00640 10 client.16230 handle_command_reply: tid=0 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.332+0000 7f3e6e667740 1 client.16230 shutdown 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.332+0000 7f3e6e667740 1 -- 172.21.15.7:0/1691516824 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 msgr2=0x564b14a69bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.332+0000 7f3e6e667740 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a69bc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800e9c0 tx=0x7f3e5809b430 comp rx=0 tx=0).stop 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.332+0000 7f3e6e667740 1 -- 172.21.15.7:0/1691516824 shutdown_connections 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.332+0000 7f3e6e667740 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564b14a75a40 0x564b14a77e30 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.333+0000 7f3e6e667740 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a71140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.333+0000 7f3e6e667740 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.333+0000 7f3e6e667740 1 --2- 172.21.15.7:0/1691516824 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a69bc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.333+0000 7f3e4aa00640 0 client.16230 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.333+0000 7f3e6e667740 1 -- 172.21.15.7:0/1691516824 >> 172.21.15.7:0/1691516824 conn(0x564b14a5ae30 msgr2=0x564b14a5a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.333+0000 7f3e6e667740 1 -- 172.21.15.7:0/1691516824 shutdown_connections 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.333+0000 7f3e6e667740 1 -- 172.21.15.7:0/1691516824 wait complete. 2024-03-18T01:47:38.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.333+0000 7f3e6e667740 20 client.16230 trim_cache size 0 max 16384 2024-03-18T01:47:38.362 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.361+0000 7f3e6e667740 1 Processor -- start 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.361+0000 7f3e6e667740 1 -- start start 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a5f8c0 0x564b14a6a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a602b0 0x564b14a59960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a65420 0x564b14a59ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e6e667740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3e641ac530 con 0x564b14a65420 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e6e667740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e641ae190 con 0x564b14a602b0 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e6e667740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e641acc30 con 0x564b14a5f8c0 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e69000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a5f8c0 0x564b14a6a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e63e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a602b0 0x564b14a59960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e69000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a5f8c0 0x564b14a6a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59070/0 (socket says 172.21.15.7:59070) 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e69000640 1 -- 172.21.15.7:0/88549326 learned_addr learned my addr 172.21.15.7:0/88549326 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.363 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.362+0000 7f3e69a00640 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a65420 0x564b14a59ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e69000640 1 -- 172.21.15.7:0/88549326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a602b0 msgr2=0x564b14a59960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e69000640 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a602b0 0x564b14a59960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e69000640 1 -- 172.21.15.7:0/88549326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a65420 msgr2=0x564b14a59ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e69000640 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a65420 0x564b14a59ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e69000640 1 -- 172.21.15.7:0/88549326 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b14a56c10 con 0x564b14a5f8c0 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e63e00640 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a602b0 0x564b14a59960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e69000640 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a5f8c0 0x564b14a6a060 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3e5809b790 tx=0x7f3e58097b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28f3aec988e2500d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e55600640 1 -- 172.21.15.7:0/88549326 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5809b800 con 0x564b14a5f8c0 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e55600640 1 -- 172.21.15.7:0/88549326 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e58086110 con 0x564b14a5f8c0 2024-03-18T01:47:38.365 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.363+0000 7f3e55600640 1 -- 172.21.15.7:0/88549326 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e580977b0 con 0x564b14a5f8c0 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 -- 172.21.15.7:0/88549326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a5f8c0 msgr2=0x564b14a6a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a5f8c0 0x564b14a6a060 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3e5809b790 tx=0x7f3e58097b50 comp rx=0 tx=0).stop 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 -- 172.21.15.7:0/88549326 shutdown_connections 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a65420 0x564b14a59ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a602b0 0x564b14a59960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 --2- 172.21.15.7:0/88549326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a5f8c0 0x564b14a6a060 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 -- 172.21.15.7:0/88549326 >> 172.21.15.7:0/88549326 conn(0x564b14a5ae30 msgr2=0x564b14a5b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 -- 172.21.15.7:0/88549326 shutdown_connections 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 -- 172.21.15.7:0/88549326 wait complete. 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.364+0000 7f3e6e667740 1 -- start start 2024-03-18T01:47:38.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e6e667740 1 Processor -- start 2024-03-18T01:47:38.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a6e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e6e667740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a6f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e6e667740 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3e641ada60 con 0x564b14a5f8c0 2024-03-18T01:47:38.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e6e667740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e641ad330 con 0x564b14a65420 2024-03-18T01:47:38.369 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e6e667740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e641aa510 con 0x564b14a602b0 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e6e667740 10 client.0 _ll_register_callbacks cb 0x7f3e58009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e63e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e69a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a6f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e69a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a6f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52834/0 (socket says 172.21.15.7:52834) 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e69a00640 1 -- 172.21.15.7:0/266059021 learned_addr learned my addr 172.21.15.7:0/266059021 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.365+0000 7f3e69000640 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a6e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e63e00640 1 -- 172.21.15.7:0/266059021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 msgr2=0x564b14a6f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e63e00640 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a6f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e63e00640 1 -- 172.21.15.7:0/266059021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 msgr2=0x564b14a6e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e63e00640 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a6e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e63e00640 1 -- 172.21.15.7:0/266059021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b14a6fb40 con 0x564b14a602b0 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e63e00640 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6e540 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009c30 tx=0x7f3e50009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49c81e273b47fe8d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-03-18T01:47:38.370 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e4aa00640 1 -- 172.21.15.7:0/266059021 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e500041a0 con 0x564b14a602b0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e6e667740 1 -- 172.21.15.7:0/266059021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e58086110 con 0x564b14a602b0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.366+0000 7f3e4aa00640 1 -- 172.21.15.7:0/266059021 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e50004340 con 0x564b14a602b0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e4aa00640 1 -- 172.21.15.7:0/266059021 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=150) ==== 24+0+0 (secure 0 0 0) 0x7f3e5003c050 con 0x564b14a602b0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e6e667740 10 client.26551 fetch_fsmap learned FSMap version 150 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e6e667740 1 -- 172.21.15.7:0/266059021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=150,monmap=4+}) -- 0x564b14a70eb0 con 0x564b14a602b0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e4aa00640 1 -- 172.21.15.7:0/266059021 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e58086110 con 0x564b14a602b0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e4aa00640 1 -- 172.21.15.7:0/266059021 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 150) ==== 8390+0+0 (secure 0 0 0) 0x7f3e5002c250 con 0x564b14a602b0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e6e667740 10 client.26551 fetch_fsmap finished waiting for FSMap version 150 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e6e667740 10 client.26551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e6e667740 20 client.26551 populate_metadata read hostname 'smithi007' 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e6e667740 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564b14adc180 0x564b14ade5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e6e667740 4 client.26551 mds_command: new command op to 24596 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.367+0000 7f3e6e667740 1 -- 172.21.15.7:0/266059021 --> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564b149cbce0 con 0x564b14adc180 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.368+0000 7f3e69000640 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564b14adc180 0x564b14ade5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.368+0000 7f3e69000640 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564b14adc180 0x564b14ade5a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48ef7ed923250723 server_cookie=d76e5adabe67c1c6 in_seq=0 out_seq=0 2024-03-18T01:47:38.371 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.368+0000 7f3e4aa00640 10 client.26551 ms_handle_connect on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.368+0000 7f3e4aa00640 1 -- 172.21.15.7:0/266059021 <== mds.0 v2:172.21.15.46:6838/1986652812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564b149cbce0 con 0x564b14adc180 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.368+0000 7f3e4aa00640 10 client.26551 handle_command_reply: tid=0 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.368+0000 7f3e6e667740 1 client.26551 shutdown 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 -- 172.21.15.7:0/266059021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 msgr2=0x564b14a6e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6e540 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009c30 tx=0x7f3e50009f50 comp rx=0 tx=0).stop 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 -- 172.21.15.7:0/266059021 shutdown_connections 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.46:6838/1986652812,v1:172.21.15.46:6839/1986652812] conn(0x564b14adc180 0x564b14ade5a0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564b14a65420 0x564b14a6f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564b14a602b0 0x564b14a6e540 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 --2- 172.21.15.7:0/266059021 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x564b14a5f8c0 0x564b14a6e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e4aa00640 0 client.26551 ms_handle_reset on v2:172.21.15.46:6838/1986652812 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 -- 172.21.15.7:0/266059021 >> 172.21.15.7:0/266059021 conn(0x564b14a5ae30 msgr2=0x564b14a5e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 -- 172.21.15.7:0/266059021 shutdown_connections 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 1 -- 172.21.15.7:0/266059021 wait complete. 2024-03-18T01:47:38.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.369+0000 7f3e6e667740 20 client.26551 trim_cache size 0 max 16384 2024-03-18T01:47:38.373 INFO:teuthology.orchestra.run.smithi007.stdout:[] 2024-03-18T01:47:38.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 -- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e4c0a2710 msgr2=0x7f3e4c0a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e4c0a2710 0x7f3e4c0a4bd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c006010 tx=0x7f3e5c004a40 comp rx=0 tx=0).stop 2024-03-18T01:47:38.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 -- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e64102330 msgr2=0x7f3e64199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e64102330 0x7f3e64199c80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800e9f0 tx=0x7f3e5800eeb0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 -- 172.21.15.7:0/921565353 shutdown_connections 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e64107590 0x7f3e641a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e64104c60 0x7f3e6419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e4c0a2710 0x7f3e4c0a4bd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 --2- 172.21.15.7:0/921565353 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e64102330 0x7f3e64199c80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 -- 172.21.15.7:0/921565353 >> 172.21.15.7:0/921565353 conn(0x7f3e640fbc40 msgr2=0x7f3e64101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 -- 172.21.15.7:0/921565353 shutdown_connections 2024-03-18T01:47:38.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:38.370+0000 7f3e6ba00640 1 -- 172.21.15.7:0/921565353 wait complete. 2024-03-18T01:47:39.390 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-18T01:47:39.391 INFO:tasks.fwd_scrub:done joining 2024-03-18T01:47:39.391 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T01:47:39.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:47:39.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:39 smithi007 ceph-mon[31996]: pgmap v701: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:39.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:39 smithi046 ceph-mon[36108]: pgmap v701: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:39.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:39 smithi203 ceph-mon[36345]: pgmap v701: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:39.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.645+0000 7f86da200640 1 -- 172.21.15.7:0/1547858476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4104bf0 msgr2=0x7f86d4105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:39.646 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.645+0000 7f86da200640 1 --2- 172.21.15.7:0/1547858476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4104bf0 0x7f86d4105070 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f86c0009900 tx=0x7f86c002bc40 comp rx=0 tx=0).stop 2024-03-18T01:47:39.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.646+0000 7f86da200640 1 -- 172.21.15.7:0/1547858476 shutdown_connections 2024-03-18T01:47:39.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.646+0000 7f86da200640 1 --2- 172.21.15.7:0/1547858476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86d41055b0 0x7f86d410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.646+0000 7f86da200640 1 --2- 172.21.15.7:0/1547858476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4104bf0 0x7f86d4105070 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.646+0000 7f86da200640 1 --2- 172.21.15.7:0/1547858476 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d4103a00 0x7f86d4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.646+0000 7f86da200640 1 -- 172.21.15.7:0/1547858476 >> 172.21.15.7:0/1547858476 conn(0x7f86d40fef90 msgr2=0x7f86d41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:39.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.646+0000 7f86da200640 1 -- 172.21.15.7:0/1547858476 shutdown_connections 2024-03-18T01:47:39.647 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.646+0000 7f86da200640 1 -- 172.21.15.7:0/1547858476 wait complete. 2024-03-18T01:47:39.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.647+0000 7f86da200640 1 Processor -- start 2024-03-18T01:47:39.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.647+0000 7f86da200640 1 -- start start 2024-03-18T01:47:39.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86da200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4103a00 0x7f86d4199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:39.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86da200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86d4104bf0 0x7f86d419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:39.648 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86da200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d41055b0 0x7f86d41a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:39.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86da200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f86d410f6d0 con 0x7f86d41055b0 2024-03-18T01:47:39.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86da200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f86d410f550 con 0x7f86d4104bf0 2024-03-18T01:47:39.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86da200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86d410f850 con 0x7f86d4103a00 2024-03-18T01:47:39.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4103a00 0x7f86d4199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:39.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4103a00 0x7f86d4199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59108/0 (socket says 172.21.15.7:59108) 2024-03-18T01:47:39.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3400640 1 -- 172.21.15.7:0/1915165854 learned_addr learned my addr 172.21.15.7:0/1915165854 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:39.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d2a00640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86d4104bf0 0x7f86d419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:39.649 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3e00640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d41055b0 0x7f86d41a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:39.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3400640 1 -- 172.21.15.7:0/1915165854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86d4104bf0 msgr2=0x7f86d419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:39.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3400640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86d4104bf0 0x7f86d419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3400640 1 -- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d41055b0 msgr2=0x7f86d41a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:39.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3400640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d41055b0 0x7f86d41a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.650 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d3400640 1 -- 172.21.15.7:0/1915165854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d41a1920 con 0x7f86d4103a00 2024-03-18T01:47:39.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.648+0000 7f86d2a00640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86d4104bf0 0x7f86d419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:39.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.649+0000 7f86d3400640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4103a00 0x7f86d4199d10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f86c400d770 tx=0x7f86c400dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:39.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.649+0000 7f86cbe00640 1 -- 172.21.15.7:0/1915165854 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4017070 con 0x7f86d4103a00 2024-03-18T01:47:39.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.649+0000 7f86cbe00640 1 -- 172.21.15.7:0/1915165854 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86c4009c30 con 0x7f86d4103a00 2024-03-18T01:47:39.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.649+0000 7f86cbe00640 1 -- 172.21.15.7:0/1915165854 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4003dd0 con 0x7f86d4103a00 2024-03-18T01:47:39.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.649+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d41a1c10 con 0x7f86d4103a00 2024-03-18T01:47:39.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.649+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d41a2110 con 0x7f86d4103a00 2024-03-18T01:47:39.651 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.650+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d410acc0 con 0x7f86d4103a00 2024-03-18T01:47:39.653 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.652+0000 7f86cbe00640 1 -- 172.21.15.7:0/1915165854 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f86c4003f70 con 0x7f86d4103a00 2024-03-18T01:47:39.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.654+0000 7f86cbe00640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f86b80a2880 0x7f86b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:39.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.654+0000 7f86cbe00640 1 -- 172.21.15.7:0/1915165854 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f86c40ceac0 con 0x7f86d4103a00 2024-03-18T01:47:39.654 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.654+0000 7f86d2a00640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f86b80a2880 0x7f86b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:39.655 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.655+0000 7f86d2a00640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f86b80a2880 0x7f86b80a4d40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f86c002efd0 tx=0x7f86c002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:39.657 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.656+0000 7f86cbe00640 1 -- 172.21.15.7:0/1915165854 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f86c4009de0 con 0x7f86d4103a00 2024-03-18T01:47:39.865 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.864+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86d402d460 con 0x7f86d4103a00 2024-03-18T01:47:39.867 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.866+0000 7f86cbe00640 1 -- 172.21.15.7:0/1915165854 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7f86c409b1d0 con 0x7f86d4103a00 2024-03-18T01:47:39.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:47:39.868 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:47:39.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f86b80a2880 msgr2=0x7f86b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:39.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f86b80a2880 0x7f86b80a4d40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f86c002efd0 tx=0x7f86c002cf90 comp rx=0 tx=0).stop 2024-03-18T01:47:39.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4103a00 msgr2=0x7f86d4199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:39.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4103a00 0x7f86d4199d10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f86c400d770 tx=0x7f86c400dc30 comp rx=0 tx=0).stop 2024-03-18T01:47:39.871 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 shutdown_connections 2024-03-18T01:47:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f86b80a2880 0x7f86b80a4d40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f86d41055b0 0x7f86d41a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86d4104bf0 0x7f86d419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 --2- 172.21.15.7:0/1915165854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86d4103a00 0x7f86d4199d10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.870+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 >> 172.21.15.7:0/1915165854 conn(0x7f86d40fef90 msgr2=0x7f86d4107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.871+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 shutdown_connections 2024-03-18T01:47:39.872 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:39.871+0000 7f86da200640 1 -- 172.21.15.7:0/1915165854 wait complete. 2024-03-18T01:47:39.894 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:47:39.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:47:40.128 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 -- 172.21.15.7:0/3859521185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec104c10 msgr2=0x7f6cec107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 --2- 172.21.15.7:0/3859521185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec104c10 0x7f6cec107000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009900 tx=0x7f6cd802bc30 comp rx=0 tx=0).stop 2024-03-18T01:47:40.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 -- 172.21.15.7:0/3859521185 shutdown_connections 2024-03-18T01:47:40.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 --2- 172.21.15.7:0/3859521185 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cec107540 0x7f6cec109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 --2- 172.21.15.7:0/3859521185 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec104c10 0x7f6cec107000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 --2- 172.21.15.7:0/3859521185 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cec1022e0 0x7f6cec1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 -- 172.21.15.7:0/3859521185 >> 172.21.15.7:0/3859521185 conn(0x7f6cec0fbc70 msgr2=0x7f6cec0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:40.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 -- 172.21.15.7:0/3859521185 shutdown_connections 2024-03-18T01:47:40.129 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.128+0000 7f6cf1600640 1 -- 172.21.15.7:0/3859521185 wait complete. 2024-03-18T01:47:40.131 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.129+0000 7f6cf1600640 1 Processor -- start 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.129+0000 7f6cf1600640 1 -- start start 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.129+0000 7f6cf1600640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec1022e0 0x7f6cec199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.129+0000 7f6cf1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cec104c10 0x7f6cec19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.129+0000 7f6cf1600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cec107540 0x7f6cec1a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.129+0000 7f6cf1600640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec10d4a0 con 0x7f6cec1022e0 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.129+0000 7f6cf1600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec10d320 con 0x7f6cec104c10 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.129+0000 7f6cf1600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6cec10d620 con 0x7f6cec107540 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6ceaa00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec1022e0 0x7f6cec199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6cea000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cec104c10 0x7f6cec19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6cea000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cec104c10 0x7f6cec19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52862/0 (socket says 172.21.15.7:52862) 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6cea000640 1 -- 172.21.15.7:0/2574452584 learned_addr learned my addr 172.21.15.7:0/2574452584 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6ceaa00640 1 -- 172.21.15.7:0/2574452584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cec107540 msgr2=0x7f6cec1a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.132 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6ceaa00640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cec107540 0x7f6cec1a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6ceaa00640 1 -- 172.21.15.7:0/2574452584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cec104c10 msgr2=0x7f6cec19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6ceaa00640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cec104c10 0x7f6cec19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6ceaa00640 1 -- 172.21.15.7:0/2574452584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cec1a1970 con 0x7f6cec1022e0 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6cea000640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cec104c10 0x7f6cec19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.130+0000 7f6ceaa00640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec1022e0 0x7f6cec199d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00e9f0 tx=0x7f6cdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.131+0000 7f6ce3400640 1 -- 172.21.15.7:0/2574452584 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc00ccc0 con 0x7f6cec1022e0 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.131+0000 7f6ce3400640 1 -- 172.21.15.7:0/2574452584 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cdc010910 con 0x7f6cec1022e0 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.131+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cec1a1c60 con 0x7f6cec1022e0 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.131+0000 7f6ce3400640 1 -- 172.21.15.7:0/2574452584 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cdc018960 con 0x7f6cec1022e0 2024-03-18T01:47:40.133 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.131+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cec1a21d0 con 0x7f6cec1022e0 2024-03-18T01:47:40.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.132+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cec109030 con 0x7f6cec1022e0 2024-03-18T01:47:40.134 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.134+0000 7f6ce3400640 1 -- 172.21.15.7:0/2574452584 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f6cdc010ac0 con 0x7f6cec1022e0 2024-03-18T01:47:40.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.137+0000 7f6ce3400640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6cd00a2880 0x7f6cd00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:40.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.137+0000 7f6ce3400640 1 -- 172.21.15.7:0/2574452584 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f6cdc0cf6d0 con 0x7f6cec1022e0 2024-03-18T01:47:40.137 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.137+0000 7f6cea000640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6cd00a2880 0x7f6cd00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:40.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.138+0000 7f6cea000640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6cd00a2880 0x7f6cd00a4d40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8006010 tx=0x7f6cd802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:40.138 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.138+0000 7f6ce3400640 1 -- 172.21.15.7:0/2574452584 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6cdc09c3b0 con 0x7f6cec1022e0 2024-03-18T01:47:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.375+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6cec062480 con 0x7f6cec1022e0 2024-03-18T01:47:40.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.376+0000 7f6ce3400640 1 -- 172.21.15.7:0/2574452584 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f6cdc09bcb0 con 0x7f6cec1022e0 2024-03-18T01:47:40.377 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:47:40.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6cd00a2880 msgr2=0x7f6cd00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6cd00a2880 0x7f6cd00a4d40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8006010 tx=0x7f6cd802df90 comp rx=0 tx=0).stop 2024-03-18T01:47:40.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec1022e0 msgr2=0x7f6cec199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec1022e0 0x7f6cec199d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00e9f0 tx=0x7f6cdc00eeb0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 shutdown_connections 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cec107540 0x7f6cec1a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cec104c10 0x7f6cec19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f6cd00a2880 0x7f6cd00a4d40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 --2- 172.21.15.7:0/2574452584 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6cec1022e0 0x7f6cec199d50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.379+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 >> 172.21.15.7:0/2574452584 conn(0x7f6cec0fbc70 msgr2=0x7f6cec101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.380+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 shutdown_connections 2024-03-18T01:47:40.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.380+0000 7f6cf1600640 1 -- 172.21.15.7:0/2574452584 wait complete. 2024-03-18T01:47:40.400 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:47:40.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:40 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1915165854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:47:40.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:40 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1915165854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:47:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.611+0000 7f9f92400640 1 -- 172.21.15.7:0/2337396565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f8c075820 msgr2=0x7f9f8c073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.611+0000 7f9f92400640 1 --2- 172.21.15.7:0/2337396565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f8c075820 0x7f9f8c073c70 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009930 tx=0x7f9f8002bc30 comp rx=0 tx=0).stop 2024-03-18T01:47:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.611+0000 7f9f92400640 1 -- 172.21.15.7:0/2337396565 shutdown_connections 2024-03-18T01:47:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.611+0000 7f9f92400640 1 --2- 172.21.15.7:0/2337396565 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f8c10dc00 0x7f9f8c10fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.611+0000 7f9f92400640 1 --2- 172.21.15.7:0/2337396565 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f8c074240 0x7f9f8c0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.613 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.611+0000 7f9f92400640 1 --2- 172.21.15.7:0/2337396565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f8c075820 0x7f9f8c073c70 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.611+0000 7f9f92400640 1 -- 172.21.15.7:0/2337396565 >> 172.21.15.7:0/2337396565 conn(0x7f9f8c0fcba0 msgr2=0x7f9f8c0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.611+0000 7f9f92400640 1 -- 172.21.15.7:0/2337396565 shutdown_connections 2024-03-18T01:47:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.612+0000 7f9f92400640 1 -- 172.21.15.7:0/2337396565 wait complete. 2024-03-18T01:47:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.612+0000 7f9f92400640 1 Processor -- start 2024-03-18T01:47:40.614 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f92400640 1 -- start start 2024-03-18T01:47:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f92400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f8c074240 0x7f9f8c19dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f92400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f8c075820 0x7f9f8c19e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f92400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f8c10dc00 0x7f9f8c1a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:40.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f92400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c1138f0 con 0x7f9f8c075820 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f92400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c113770 con 0x7f9f8c10dc00 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f92400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8c113a70 con 0x7f9f8c074240 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f91000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f8c074240 0x7f9f8c19dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f8be00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f8c075820 0x7f9f8c19e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f91000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f8c074240 0x7f9f8c19dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59144/0 (socket says 172.21.15.7:59144) 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f91000640 1 -- 172.21.15.7:0/34175959 learned_addr learned my addr 172.21.15.7:0/34175959 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.613+0000 7f9f91a00640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f8c10dc00 0x7f9f8c1a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f91000640 1 -- 172.21.15.7:0/34175959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f8c10dc00 msgr2=0x7f9f8c1a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f91000640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f8c10dc00 0x7f9f8c1a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f91000640 1 -- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f8c075820 msgr2=0x7f9f8c19e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f91000640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f8c075820 0x7f9f8c19e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f91000640 1 -- 172.21.15.7:0/34175959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c1a5b80 con 0x7f9f8c074240 2024-03-18T01:47:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f8be00640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f8c075820 0x7f9f8c19e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f91000640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f8c074240 0x7f9f8c19dfb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9f80000c00 tx=0x7f9f8002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f89600640 1 -- 172.21.15.7:0/34175959 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80043020 con 0x7f9f8c074240 2024-03-18T01:47:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f89600640 1 -- 172.21.15.7:0/34175959 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f800054e0 con 0x7f9f8c074240 2024-03-18T01:47:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.614+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8c1a5e10 con 0x7f9f8c074240 2024-03-18T01:47:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.615+0000 7f9f89600640 1 -- 172.21.15.7:0/34175959 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f80003c40 con 0x7f9f8c074240 2024-03-18T01:47:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.615+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8c1a62d0 con 0x7f9f8c074240 2024-03-18T01:47:40.618 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.616+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8c103dd0 con 0x7f9f8c074240 2024-03-18T01:47:40.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.618+0000 7f9f89600640 1 -- 172.21.15.7:0/34175959 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f9f80046020 con 0x7f9f8c074240 2024-03-18T01:47:40.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.619+0000 7f9f89600640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9f740a2880 0x7f9f740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:40.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.619+0000 7f9f89600640 1 -- 172.21.15.7:0/34175959 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9f800f1280 con 0x7f9f8c074240 2024-03-18T01:47:40.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.619+0000 7f9f8be00640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9f740a2880 0x7f9f740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:40.620 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.620+0000 7f9f8be00640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9f740a2880 0x7f9f740a4d40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c19f470 tx=0x7f9f78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:40.626 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.626+0000 7f9f89600640 1 -- 172.21.15.7:0/34175959 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9f8004d020 con 0x7f9f8c074240 2024-03-18T01:47:40.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:40 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1915165854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:47:40.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.829+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9f8c02d340 con 0x7f9f8c074240 2024-03-18T01:47:40.830 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.830+0000 7f9f89600640 1 -- 172.21.15.7:0/34175959 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f9f800bdf60 con 0x7f9f8c074240 2024-03-18T01:47:40.831 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:47:40.833 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9f740a2880 msgr2=0x7f9f740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9f740a2880 0x7f9f740a4d40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c19f470 tx=0x7f9f78005f90 comp rx=0 tx=0).stop 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f8c074240 msgr2=0x7f9f8c19dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f8c074240 0x7f9f8c19dfb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9f80000c00 tx=0x7f9f8002c650 comp rx=0 tx=0).stop 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 shutdown_connections 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f8c10dc00 0x7f9f8c1a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f9f740a2880 0x7f9f740a4d40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f9f8c075820 0x7f9f8c19e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 --2- 172.21.15.7:0/34175959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9f8c074240 0x7f9f8c19dfb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 >> 172.21.15.7:0/34175959 conn(0x7f9f8c0fcba0 msgr2=0x7f9f8c100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 shutdown_connections 2024-03-18T01:47:40.834 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:40.833+0000 7f9f92400640 1 -- 172.21.15.7:0/34175959 wait complete. 2024-03-18T01:47:40.854 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 -- 172.21.15.7:0/2479588267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164102110 msgr2=0x7f7164104500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 --2- 172.21.15.7:0/2479588267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164102110 0x7f7164104500 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7154009900 tx=0x7f715402bc30 comp rx=0 tx=0).stop 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 -- 172.21.15.7:0/2479588267 shutdown_connections 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 --2- 172.21.15.7:0/2479588267 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7164107370 0x7f7164109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 --2- 172.21.15.7:0/2479588267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7164104a40 0x7f7164106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 --2- 172.21.15.7:0/2479588267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164102110 0x7f7164104500 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 -- 172.21.15.7:0/2479588267 >> 172.21.15.7:0/2479588267 conn(0x7f71640fbac0 msgr2=0x7f71640fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 -- 172.21.15.7:0/2479588267 shutdown_connections 2024-03-18T01:47:41.080 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.079+0000 7f7163e00640 1 -- 172.21.15.7:0/2479588267 wait complete. 2024-03-18T01:47:41.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.081+0000 7f7163e00640 1 Processor -- start 2024-03-18T01:47:41.081 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.081+0000 7f7163e00640 1 -- start start 2024-03-18T01:47:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.081+0000 7f7163e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7164102110 0x7f7164101a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.081+0000 7f7163e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164104a40 0x7f71641000c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.081+0000 7f7163e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7164107370 0x7f7164100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.081+0000 7f7163e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f716410d300 con 0x7f7164102110 2024-03-18T01:47:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.081+0000 7f7163e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f716410d180 con 0x7f7164107370 2024-03-18T01:47:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.081+0000 7f7163e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f716410d480 con 0x7f7164104a40 2024-03-18T01:47:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7163400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7164107370 0x7f7164100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:41.082 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164104a40 0x7f71641000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:41.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7163400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7164107370 0x7f7164100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52884/0 (socket says 172.21.15.7:52884) 2024-03-18T01:47:41.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7163400640 1 -- 172.21.15.7:0/2380102995 learned_addr learned my addr 172.21.15.7:0/2380102995 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:41.083 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162a00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7164102110 0x7f7164101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162000640 1 -- 172.21.15.7:0/2380102995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7164107370 msgr2=0x7f7164100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162000640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7164107370 0x7f7164100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162000640 1 -- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7164102110 msgr2=0x7f7164101a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162000640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7164102110 0x7f7164101a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162000640 1 -- 172.21.15.7:0/2380102995 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7164100e90 con 0x7f7164104a40 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7163400640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7164107370 0x7f7164100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162a00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7164102110 0x7f7164101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.082+0000 7f7162000640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164104a40 0x7f71641000c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f715000e9c0 tx=0x7f715000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.083+0000 7f715b400640 1 -- 172.21.15.7:0/2380102995 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715000cd80 con 0x7f7164104a40 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.083+0000 7f715b400640 1 -- 172.21.15.7:0/2380102995 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71500035f0 con 0x7f7164104a40 2024-03-18T01:47:41.084 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.083+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71641a1a70 con 0x7f7164104a40 2024-03-18T01:47:41.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.083+0000 7f715b400640 1 -- 172.21.15.7:0/2380102995 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150010670 con 0x7f7164104a40 2024-03-18T01:47:41.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.083+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71641a1fc0 con 0x7f7164104a40 2024-03-18T01:47:41.085 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.084+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71640fd5b0 con 0x7f7164104a40 2024-03-18T01:47:41.087 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.087+0000 7f715b400640 1 -- 172.21.15.7:0/2380102995 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f71500037a0 con 0x7f7164104a40 2024-03-18T01:47:41.088 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.088+0000 7f715b400640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71480a2800 0x7f71480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:41.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.088+0000 7f715b400640 1 -- 172.21.15.7:0/2380102995 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f71500cf1f0 con 0x7f7164104a40 2024-03-18T01:47:41.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.088+0000 7f7162a00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71480a2800 0x7f71480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:41.089 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.089+0000 7f7162a00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71480a2800 0x7f71480a4cc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f715402c240 tx=0x7f715402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:41.090 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.089+0000 7f715b400640 1 -- 172.21.15.7:0/2380102995 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f715009bed0 con 0x7f7164104a40 2024-03-18T01:47:41.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:41 smithi007 ceph-mon[31996]: pgmap v702: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:41.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:41 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2574452584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:47:41.233 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:41 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/34175959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:47:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.314+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f716402d3c0 con 0x7f7164104a40 2024-03-18T01:47:41.316 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.315+0000 7f715b400640 1 -- 172.21.15.7:0/2380102995 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f715009b7d0 con 0x7f7164104a40 2024-03-18T01:47:41.316 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:47:41.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.317+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71480a2800 msgr2=0x7f71480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.317+0000 7f7163e00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71480a2800 0x7f71480a4cc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f715402c240 tx=0x7f715402df90 comp rx=0 tx=0).stop 2024-03-18T01:47:41.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.317+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164104a40 msgr2=0x7f71641000c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.317+0000 7f7163e00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164104a40 0x7f71641000c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f715000e9c0 tx=0x7f715000ee80 comp rx=0 tx=0).stop 2024-03-18T01:47:41.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.317+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 shutdown_connections 2024-03-18T01:47:41.319 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.317+0000 7f7163e00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7164107370 0x7f7164100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.317+0000 7f7163e00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7164104a40 0x7f71641000c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.318+0000 7f7163e00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f71480a2800 0x7f71480a4cc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.318+0000 7f7163e00640 1 --2- 172.21.15.7:0/2380102995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7164102110 0x7f7164101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.318+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 >> 172.21.15.7:0/2380102995 conn(0x7f71640fbac0 msgr2=0x7f71641029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:41.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.318+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 shutdown_connections 2024-03-18T01:47:41.320 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.318+0000 7f7163e00640 1 -- 172.21.15.7:0/2380102995 wait complete. 2024-03-18T01:47:41.340 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:47:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:41 smithi046 ceph-mon[36108]: pgmap v702: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:41 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2574452584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:47:41.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:41 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/34175959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:47:41.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 -- 172.21.15.7:0/969346634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ca41031d0 msgr2=0x7f3ca4109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 --2- 172.21.15.7:0/969346634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ca41031d0 0x7f3ca4109a60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c009900 tx=0x7f3c9c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:47:41.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 -- 172.21.15.7:0/969346634 shutdown_connections 2024-03-18T01:47:41.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 --2- 172.21.15.7:0/969346634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ca41031d0 0x7f3ca4109a60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 --2- 172.21.15.7:0/969346634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ca4102810 0x7f3ca4102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 --2- 172.21.15.7:0/969346634 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca4101620 0x7f3ca4101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 -- 172.21.15.7:0/969346634 >> 172.21.15.7:0/969346634 conn(0x7f3ca40fcb90 msgr2=0x7f3ca40fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:41.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 -- 172.21.15.7:0/969346634 shutdown_connections 2024-03-18T01:47:41.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.573+0000 7f3caa200640 1 -- 172.21.15.7:0/969346634 wait complete. 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.574+0000 7f3caa200640 1 Processor -- start 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.574+0000 7f3caa200640 1 -- start start 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.574+0000 7f3caa200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ca4101620 0x7f3ca4199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.574+0000 7f3caa200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca4102810 0x7f3ca419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3caa200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ca41031d0 0x7f3ca41a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3caa200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca410d2d0 con 0x7f3ca4102810 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3caa200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca410d150 con 0x7f3ca41031d0 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3caa200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca410d450 con 0x7f3ca4101620 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ca4101620 0x7f3ca4199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca8e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ca4101620 0x7f3ca4199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59194/0 (socket says 172.21.15.7:59194) 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca8e00640 1 -- 172.21.15.7:0/4132218834 learned_addr learned my addr 172.21.15.7:0/4132218834 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:41.578 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca3e00640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca4102810 0x7f3ca419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca9800640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ca41031d0 0x7f3ca41a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca8e00640 1 -- 172.21.15.7:0/4132218834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ca41031d0 msgr2=0x7f3ca41a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca8e00640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ca41031d0 0x7f3ca41a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca8e00640 1 -- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca4102810 msgr2=0x7f3ca419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca8e00640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca4102810 0x7f3ca419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.575+0000 7f3ca8e00640 1 -- 172.21.15.7:0/4132218834 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca41a1720 con 0x7f3ca4101620 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.576+0000 7f3ca8e00640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ca4101620 0x7f3ca4199b00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009d20 tx=0x7f3c9400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.576+0000 7f3ca1600640 1 -- 172.21.15.7:0/4132218834 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94010630 con 0x7f3ca4101620 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.576+0000 7f3ca1600640 1 -- 172.21.15.7:0/4132218834 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c94010cb0 con 0x7f3ca4101620 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.576+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca41a1a10 con 0x7f3ca4101620 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.576+0000 7f3ca1600640 1 -- 172.21.15.7:0/4132218834 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94018de0 con 0x7f3ca4101620 2024-03-18T01:47:41.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.576+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca41a1f60 con 0x7f3ca4101620 2024-03-18T01:47:41.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.577+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca41a2190 con 0x7f3ca4101620 2024-03-18T01:47:41.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.579+0000 7f3ca1600640 1 -- 172.21.15.7:0/4132218834 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f3c9401f020 con 0x7f3ca4101620 2024-03-18T01:47:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.580+0000 7f3ca1600640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3c880a2800 0x7f3c880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.580+0000 7f3ca1600640 1 -- 172.21.15.7:0/4132218834 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3c940cf290 con 0x7f3ca4101620 2024-03-18T01:47:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.581+0000 7f3ca3e00640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3c880a2800 0x7f3c880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:41.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.581+0000 7f3ca3e00640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3c880a2800 0x7f3c880a4cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3ca419b010 tx=0x7f3c8c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:41.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.589+0000 7f3ca1600640 1 -- 172.21.15.7:0/4132218834 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3c9409bf70 con 0x7f3ca4101620 2024-03-18T01:47:41.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:41 smithi203 ceph-mon[36345]: pgmap v702: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T01:47:41.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:41 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2574452584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:47:41.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:41 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/34175959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:47:41.839 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.838+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3ca41a2360 con 0x7f3ca4101620 2024-03-18T01:47:41.840 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.838+0000 7f3ca1600640 1 -- 172.21.15.7:0/4132218834 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f3c9409b870 con 0x7f3ca4101620 2024-03-18T01:47:41.840 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:47:41.842 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3c880a2800 msgr2=0x7f3c880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3c880a2800 0x7f3c880a4cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3ca419b010 tx=0x7f3c8c00b390 comp rx=0 tx=0).stop 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ca4101620 msgr2=0x7f3ca4199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ca4101620 0x7f3ca4199b00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009d20 tx=0x7f3c9400c6a0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 shutdown_connections 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ca41031d0 0x7f3ca41a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3c880a2800 0x7f3c880a4cc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3ca4102810 0x7f3ca419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 --2- 172.21.15.7:0/4132218834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ca4101620 0x7f3ca4199b00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 >> 172.21.15.7:0/4132218834 conn(0x7f3ca40fcb90 msgr2=0x7f3ca4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 shutdown_connections 2024-03-18T01:47:41.843 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:41.841+0000 7f3caa200640 1 -- 172.21.15.7:0/4132218834 wait complete. 2024-03-18T01:47:41.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T01:47:42.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 -- 172.21.15.7:0/1805015873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 msgr2=0x7f52ac077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:42.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 --2- 172.21.15.7:0/1805015873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 0x7f52ac077840 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5298009860 tx=0x7f529802bb70 comp rx=0 tx=0).stop 2024-03-18T01:47:42.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 -- 172.21.15.7:0/1805015873 shutdown_connections 2024-03-18T01:47:42.066 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 --2- 172.21.15.7:0/1805015873 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac077d80 0x7f52ac078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:42.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 --2- 172.21.15.7:0/1805015873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 0x7f52ac077840 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:42.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 --2- 172.21.15.7:0/1805015873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ac0761a0 0x7f52ac0765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:42.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 -- 172.21.15.7:0/1805015873 >> 172.21.15.7:0/1805015873 conn(0x7f52ac0ff040 msgr2=0x7f52ac101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:42.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 -- 172.21.15.7:0/1805015873 shutdown_connections 2024-03-18T01:47:42.067 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.064+0000 7f52b0c00640 1 -- 172.21.15.7:0/1805015873 wait complete. 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.065+0000 7f52b0c00640 1 Processor -- start 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52b0c00640 1 -- start start 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52b0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ac0761a0 0x7f52ac19e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52b0c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 0x7f52ac19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52b0c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac077d80 0x7f52ac1a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52b0c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac113a70 con 0x7f52ac077d80 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52b0c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac1138f0 con 0x7f52ac0761a0 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52b0c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac113bf0 con 0x7f52ac0773e0 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52a9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 0x7f52ac19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52a9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ac0761a0 0x7f52ac19e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52a9400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 0x7f52ac19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:59202/0 (socket says 172.21.15.7:59202) 2024-03-18T01:47:42.069 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52aa800640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac077d80 0x7f52ac1a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52a9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ac0761a0 0x7f52ac19e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:52902/0 (socket says 172.21.15.7:52902) 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.066+0000 7f52a9400640 1 -- 172.21.15.7:0/3713559625 learned_addr learned my addr 172.21.15.7:0/3713559625 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a9400640 1 -- 172.21.15.7:0/3713559625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ac0761a0 msgr2=0x7f52ac19e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a9400640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ac0761a0 0x7f52ac19e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a9400640 1 -- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac077d80 msgr2=0x7f52ac1a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a9400640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac077d80 0x7f52ac1a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a9400640 1 -- 172.21.15.7:0/3713559625 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ac1a5d40 con 0x7f52ac0773e0 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a9400640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 0x7f52ac19e660 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f52980349e0 tx=0x7f52980042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a2a00640 1 -- 172.21.15.7:0/3713559625 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298041030 con 0x7f52ac0773e0 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a2a00640 1 -- 172.21.15.7:0/3713559625 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52980347b0 con 0x7f52ac0773e0 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ac1a5fd0 con 0x7f52ac0773e0 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52a2a00640 1 -- 172.21.15.7:0/3713559625 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298039040 con 0x7f52ac0773e0 2024-03-18T01:47:42.070 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.067+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ac1a6440 con 0x7f52ac0773e0 2024-03-18T01:47:42.071 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.069+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ac110580 con 0x7f52ac0773e0 2024-03-18T01:47:42.072 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.070+0000 7f52a2a00640 1 -- 172.21.15.7:0/3713559625 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f5298044070 con 0x7f52ac0773e0 2024-03-18T01:47:42.073 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.072+0000 7f52a2a00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52900a2880 0x7f52900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:47:42.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.072+0000 7f52a2a00640 1 -- 172.21.15.7:0/3713559625 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f52980f0380 con 0x7f52ac0773e0 2024-03-18T01:47:42.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.072+0000 7f52a9e00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52900a2880 0x7f52900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:47:42.074 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.073+0000 7f52a9e00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52900a2880 0x7f52900a4d40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f529c009530 tx=0x7f529c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:47:42.077 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.075+0000 7f52a2a00640 1 -- 172.21.15.7:0/3713559625 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f52980bd060 con 0x7f52ac0773e0 2024-03-18T01:47:42.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.335+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f52ac062480 con 0x7f52ac0773e0 2024-03-18T01:47:42.366 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.365+0000 7f52a2a00640 1 -- 172.21.15.7:0/3713559625 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+739821 (secure 0 0 0) 0x7f52980bc960 con 0x7f52ac0773e0 2024-03-18T01:47:42.378 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:47:42.378 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:42.378 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 18, 2024-03-18T01:47:42.378 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.378 INFO:teuthology.orchestra.run.smithi007.stdout: "active_gid": 14152, 2024-03-18T01:47:42.378 INFO:teuthology.orchestra.run.smithi007.stdout: "active_name": "x", 2024-03-18T01:47:42.378 INFO:teuthology.orchestra.run.smithi007.stdout: "active_addrs": { 2024-03-18T01:47:42.378 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6800", 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2743163723 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v1", 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:6801", 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2743163723 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: "active_addr": "172.21.15.7:6801/2743163723", 2024-03-18T01:47:42.379 INFO:teuthology.orchestra.run.smithi007.stdout: "active_change": "2024-03-18T01:26:57.023376+0000", 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "active_mgr_features": 4540701547738038271, 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "available": true, 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "standbys": [ 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "gid": 24109, 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "z", 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_features": 4540701547738038271, 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "available_modules": [ 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "alerts", 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.380 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.381 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.382 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.383 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.384 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.385 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.386 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_destination": { 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_destination", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email address to send alerts to", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.387 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_from_name": { 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_from_name", 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "Ceph", 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email From: name", 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.388 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_host": { 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_host", 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP server", 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.389 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_password": { 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_password", 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Password to authenticate with", 2024-03-18T01:47:42.390 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_port": { 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_port", 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "465", 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP port", 2024-03-18T01:47:42.391 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_sender": { 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_sender", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP envelope sender", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.392 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_ssl": { 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_ssl", 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.393 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_user": { 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_user", 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "User to authenticate as", 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "balancer", 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "active": { 2024-03-18T01:47:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "active", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_time": { 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_time", 2024-03-18T01:47:42.396 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0000", 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_weekday": { 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_weekday", 2024-03-18T01:47:42.397 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_max_iterations": { 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_max_iterations", 2024-03-18T01:47:42.398 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "25", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "250", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_metrics": { 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_metrics", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.399 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_step": { 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_step", 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.400 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.5", 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0.001", 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "0.999", 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "aggressiveness of optimization", 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "end_time": { 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_time", 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.401 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2359", 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "end_weekday": { 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_weekday", 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.402 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.403 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.404 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.405 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.406 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.407 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "min_score": { 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_score", 2024-03-18T01:47:42.408 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-03-18T01:47:42.409 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "upmap", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-compat", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "none", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "read", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap-read" 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Balancer mode", 2024-03-18T01:47:42.410 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_ids": { 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_ids", 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T01:47:42.411 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.412 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_deviation": { 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_deviation", 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.413 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_optimizations": { 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_optimizations", 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.414 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm", 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_down_multiplier": { 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_down_multiplier", 2024-03-18T01:47:42.415 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_refresh_rate": { 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_refresh_rate", 2024-03-18T01:47:42.416 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "20", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_starting_port": { 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_starting_port", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.417 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4721", 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_ptrace": { 2024-03-18T01:47:42.418 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_ptrace", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_interval": { 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_interval", 2024-03-18T01:47:42.419 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_memory_target_ratio": { 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T01:47:42.420 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.7", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm_log_destination": { 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm_log_destination", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.421 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "file", 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "file,syslog", 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "syslog" 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.422 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "cgroups_split": { 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cgroups_split", 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.423 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "config_checks_enabled": { 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_checks_enabled", 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.424 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "config_dashboard": { 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_dashboard", 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_alertmanager": { 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_alertmanager", 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.426 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_base": { 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_base", 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Container image name, without the tag", 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.427 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_elasticsearch": { 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_elasticsearch", 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "elasticsearch container image", 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_grafana": { 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_grafana", 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.429 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_haproxy": { 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_haproxy", 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "HAproxy container image", 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_agent": { 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_agent", 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger agent container image", 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.431 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_collector": { 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_collector", 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger collector container image", 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_query": { 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_query", 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger query container image", 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_keepalived": { 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_keepalived", 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Keepalived container image", 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_loki": { 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_loki", 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Loki container image", 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.436 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_node_exporter": { 2024-03-18T01:47:42.436 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_node_exporter", 2024-03-18T01:47:42.436 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.437 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.437 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.437 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T01:47:42.437 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.437 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.437 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_nvmeof": { 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_nvmeof", 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.438 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Nvme-of container image", 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_prometheus": { 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_prometheus", 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.439 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_promtail": { 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_promtail", 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.440 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Promtail container image", 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_snmp_gateway": { 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_snmp_gateway", 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.441 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SNMP Gateway container image", 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "container_init": { 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_init", 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.442 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "daemon_cache_timeout": { 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "daemon_cache_timeout", 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.443 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "default_cephadm_command_timeout": { 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "900", 2024-03-18T01:47:42.444 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "default_registry": { 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_registry", 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io", 2024-03-18T01:47:42.445 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "device_cache_timeout": { 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_cache_timeout", 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.446 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1800", 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache device inventory", 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "device_enhanced_scan": { 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_enhanced_scan", 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.447 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "facts_cache_timeout": { 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "facts_cache_timeout", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:42.448 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache host facts data", 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "host_check_interval": { 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "host_check_interval", 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.449 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to perform a host check", 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "hw_monitoring": { 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hw_monitoring", 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.450 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "inventory_list_all": { 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "inventory_list_all", 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.451 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.452 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.453 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "log_refresh_metadata": { 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_refresh_metadata", 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.454 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.455 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.456 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.457 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.458 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.459 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "*", 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "max_count_per_host": { 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_count_per_host", 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.460 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of daemons per service per host", 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd_draining_count": { 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_osd_draining_count", 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.461 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "migration_current": { 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "migration_current", 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.462 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "internal - do not modify", 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.463 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "root", 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm-package", 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "root" 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "oob_default_addr": { 2024-03-18T01:47:42.464 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "oob_default_addr", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "169.254.1.1", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "prometheus_alerts_path": { 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus_alerts_path", 2024-03-18T01:47:42.465 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_insecure": { 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_insecure", 2024-03-18T01:47:42.466 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_password": { 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_password", 2024-03-18T01:47:42.467 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_url": { 2024-03-18T01:47:42.468 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_url", 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.469 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_username": { 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_username", 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "secure_monitoring_stack": { 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "secure_monitoring_stack", 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.471 INFO:teuthology.orchestra.run.smithi007.stdout: "service_discovery_port": { 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "service_discovery_port", 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8765", 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "cephadm service discovery port", 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "ssh_config_file": { 2024-03-18T01:47:42.472 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssh_config_file", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "use_agent": { 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_agent", 2024-03-18T01:47:42.473 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.474 INFO:teuthology.orchestra.run.smithi007.stdout: "use_repo_digest": { 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_repo_digest", 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.475 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_failed_host_check": { 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_failed_host_check", 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.476 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_daemons": { 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_daemons", 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.477 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_hosts": { 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_hosts", 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.478 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crash", 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.479 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.481 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.482 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.483 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "retain_interval": { 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retain_interval", 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.484 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "31536000", 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_recent_interval": { 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_recent_interval", 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.485 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1209600", 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.486 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dashboard", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.487 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.488 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_ENABLED": { 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.494 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.495 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.496 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.497 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.498 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.499 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.500 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.501 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.502 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_URL": { 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_URL", 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.503 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.504 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.508 INFO:teuthology.orchestra.run.smithi007.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "{}", 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T01:47:42.509 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.510 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.511 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T01:47:42.512 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T01:47:42.513 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.514 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.515 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T01:47:42.516 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.517 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.518 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.519 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.520 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.521 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8", 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.522 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "45", 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.523 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:42.524 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.525 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.526 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.527 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.528 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.529 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.530 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "cross_origin_url": { 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cross_origin_url", 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.531 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "crt_file": { 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crt_file", 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.532 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "debug": { 2024-03-18T01:47:42.533 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "debug", 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable/disable debug options", 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.534 INFO:teuthology.orchestra.run.smithi007.stdout: "jwt_token_ttl": { 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "jwt_token_ttl", 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "28800", 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-03-18T01:47:42.535 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.536 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.537 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.538 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.539 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.540 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "motd": { 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "motd", 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.541 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "The message of the day", 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_resolve_ip_addr": { 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.542 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.543 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.544 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8080", 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.545 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl_server_port": { 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl_server_port", 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.546 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8443", 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.547 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "redirect", 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect" 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-03-18T01:47:42.548 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "url_prefix": { 2024-03-18T01:47:42.549 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url_prefix", 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "devicehealth", 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_monitoring": { 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_monitoring", 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "monitor device health metrics", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.553 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.554 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.555 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.556 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "mark_out_threshold": { 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mark_out_threshold", 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2419200", 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.557 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": { 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_name", 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "device_health_metrics", 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.558 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "retention_period": { 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retention_period", 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15552000", 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain device health metrics", 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.559 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_frequency": { 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_frequency", 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.560 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "self_heal": { 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "self_heal", 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T01:47:42.561 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T01:47:42.562 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_threshold": { 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_threshold", 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7257600", 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.563 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "diskprediction_local", 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.564 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.565 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.566 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.567 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.568 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "predict_interval": { 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predict_interval", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.569 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "predictor_model": { 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predictor_model", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "prophetstor", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.570 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.571 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "influx", 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": false, 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "influxdb python module not found", 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "batch_size": { 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "batch_size", 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5000", 2024-03-18T01:47:42.572 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "database": { 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "database", 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "ceph", 2024-03-18T01:47:42.573 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": { 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hostname", 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.574 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server hostname", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "30", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "5", 2024-03-18T01:47:42.575 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.576 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.577 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.578 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "password": { 2024-03-18T01:47:42.580 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "password", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "password of InfluxDB server user", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "port": { 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "port", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.581 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8086", 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server port", 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.582 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "false", 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "threads": { 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threads", 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.583 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "32", 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "username": { 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "username", 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.584 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "username of InfluxDB server user", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "verify_ssl": { 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "verify_ssl", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.585 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "true", 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "insights", 2024-03-18T01:47:42.586 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.587 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.588 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.589 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.590 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.591 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "iostat", 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.592 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.593 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.594 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.595 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.596 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "k8sevents", 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_event_retention_days": { 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ceph_event_retention_days", 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7", 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.597 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "config_check_secs": { 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_check_secs", 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.598 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "10", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.599 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.600 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.601 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.602 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.603 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "localpool", 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "failure_domain": { 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "failure_domain", 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.604 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "host", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "failure domain for any created local pool", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.605 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.606 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.607 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.608 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.609 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": { 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_size", 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.610 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default min_size for any created local pool", 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "num_rep": { 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "num_rep", 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3", 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.611 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default replica count for any created local pool", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": { 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_num", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "128", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T01:47:42.612 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "prefix": { 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prefix", 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.613 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name prefix for any created local pool", 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "subtree": { 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "subtree", 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "rack", 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.614 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mds_autoscaler", 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.615 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.616 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.617 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.618 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.619 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirroring", 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.620 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.621 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.622 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.623 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.624 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "nfs", 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.625 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.626 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.627 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.628 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.629 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.630 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "fail_fs": { 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "fail_fs", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.631 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.632 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.633 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.634 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.635 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator": { 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.636 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm", 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "rook", 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "test_orchestrator" 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Orchestrator backend", 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.637 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.638 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_perf_query", 2024-03-18T01:47:42.638 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.638 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.638 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.638 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.638 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.641 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.641 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.641 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.641 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.641 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.642 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.643 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.644 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.645 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_support", 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.646 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.647 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.648 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.649 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.650 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.651 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_autoscaler", 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.652 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.653 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.654 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.655 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.656 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.657 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "threshold": { 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threshold", 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1.0", 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.658 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "scaling threshold", 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "progress", 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_pg_recovery_event": { 2024-03-18T01:47:42.659 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_pg_recovery_event", 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-03-18T01:47:42.660 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.661 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.662 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.663 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.664 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.665 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "max_completed_events": { 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_completed_events", 2024-03-18T01:47:42.666 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "50", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "number of past completed events to remember", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.667 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long the module is going to sleep", 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus", 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.668 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "cache": { 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cache", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.669 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "exclude_perf_counters": { 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "exclude_perf_counters", 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.670 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.671 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.672 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.673 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.674 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.675 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools": { 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools", 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.676 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "300", 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.677 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_interval": { 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_interval", 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15.0", 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.678 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:42.679 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "9283", 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "stale_cache_strategy": { 2024-03-18T01:47:42.680 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stale_cache_strategy", 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "log", 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-03-18T01:47:42.681 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "default", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "default", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "error" 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.682 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.683 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_support", 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.684 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.685 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.686 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.687 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.688 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_snap_create": { 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_snap_create", 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.689 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "mirror_snapshot_schedule": { 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.690 INFO:teuthology.orchestra.run.smithi007.stdout: "trash_purge_schedule": { 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "trash_purge_schedule", 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.691 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "restful", 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_auth": { 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_auth", 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.692 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.693 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.694 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.696 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.697 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.698 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.699 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.700 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rgw", 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.701 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.702 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.703 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.704 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.705 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rook", 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.706 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.707 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.708 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.709 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.710 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.711 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_class": { 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "storage_class", 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "local", 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.712 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "selftest", 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.713 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.714 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.715 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.716 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.717 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "roption1": { 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption1", 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.718 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "roption2": { 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption2", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "xyz", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.719 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption1": { 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption1", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.720 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption2": { 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption2", 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.721 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption3": { 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption3", 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.722 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption4": { 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption4", 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.723 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption5": { 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption5", 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.724 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption6": { 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption6", 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.725 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption7": { 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption7", 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.726 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "42", 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "testkey": { 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testkey", 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.727 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "testlkey": { 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testlkey", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.728 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "testnewline": { 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testnewline", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.729 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snap_schedule", 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_m_granularity": { 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_m_granularity", 2024-03-18T01:47:42.730 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "dump_on_update": { 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dump_on_update", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.731 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "dump database to debug log on update", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.732 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.733 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.734 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.735 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.736 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stats", 2024-03-18T01:47:42.737 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.738 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.739 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.740 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.741 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.742 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "status", 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.743 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.744 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.745 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.746 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.747 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telegraf", 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.748 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "address": { 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "address", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.749 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15", 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.750 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.751 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.752 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.753 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.754 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.755 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telemetry", 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_basic": { 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_basic", 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.756 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_crash": { 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_crash", 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.757 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_device": { 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_device", 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.758 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_ident": { 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_ident", 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.759 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_perf": { 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_perf", 2024-03-18T01:47:42.760 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.761 INFO:teuthology.orchestra.run.smithi007.stdout: "contact": { 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "contact", 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.762 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "description": { 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "description", 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.763 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "device_url": { 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_url", 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.764 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.765 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "24", 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "8", 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.766 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "last_opt_revision": { 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "last_opt_revision", 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1", 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard": { 2024-03-18T01:47:42.767 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard", 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.768 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard_description": { 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard_description", 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.769 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.770 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.771 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.772 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.773 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.774 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "organization": { 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "organization", 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.775 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "proxy": { 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "proxy", 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.776 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "url": { 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url", 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.777 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test_orchestrator", 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.778 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.779 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.780 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.781 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.782 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "volumes", 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.783 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.784 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.785 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.786 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.787 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_clones": { 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_clones", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.788 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "periodic_async_work": { 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "periodic_async_work", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.789 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Periodically check for async work", 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_delay": { 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_delay", 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.790 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_no_wait": { 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.791 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix", 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "discovery_interval": { 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "discovery_interval", 2024-03-18T01:47:42.792 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "100", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "identifier": { 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "identifier", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.793 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.794 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.795 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.796 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.797 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.798 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.799 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_host": { 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_host", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.800 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_port": { 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_port", 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10051", 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.801 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_sender": { 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_sender", 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.802 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "gid": 24113, 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "y", 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr_features": 4540701547738038271, 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "available_modules": [ 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "alerts", 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.803 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.804 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.805 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.806 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.807 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.808 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_destination": { 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_destination", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.809 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email address to send alerts to", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_from_name": { 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_from_name", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "Ceph", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.810 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email From: name", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_host": { 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_host", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP server", 2024-03-18T01:47:42.811 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_password": { 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_password", 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Password to authenticate with", 2024-03-18T01:47:42.812 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_port": { 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_port", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "465", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP port", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.813 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_sender": { 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_sender", 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP envelope sender", 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.814 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_ssl": { 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_ssl", 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T01:47:42.815 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_user": { 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_user", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "User to authenticate as", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.816 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "balancer", 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "active": { 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "active", 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.817 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_time": { 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_time", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0000", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.818 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_weekday": { 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_weekday", 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-03-18T01:47:42.819 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_max_iterations": { 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_max_iterations", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "25", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "250", 2024-03-18T01:47:42.820 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_metrics": { 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_metrics", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T01:47:42.821 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_step": { 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_step", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.5", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0.001", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "0.999", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "aggressiveness of optimization", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T01:47:42.822 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "end_time": { 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_time", 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2359", 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.823 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "end_weekday": { 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_weekday", 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.824 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.825 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.826 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.827 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.828 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "min_score": { 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_score", 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.829 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.830 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "upmap", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-compat", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "none", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "read", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap-read" 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Balancer mode", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.831 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_ids": { 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_ids", 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:42.832 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_deviation": { 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_deviation", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.833 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_optimizations": { 2024-03-18T01:47:42.834 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_optimizations", 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.835 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_down_multiplier": { 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_down_multiplier", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T01:47:42.836 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_refresh_rate": { 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_refresh_rate", 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "20", 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.837 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_starting_port": { 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_starting_port", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4721", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T01:47:42.838 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_ptrace": { 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_ptrace", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T01:47:42.839 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_interval": { 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_interval", 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.840 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_memory_target_ratio": { 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.7", 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.841 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm_log_destination": { 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm_log_destination", 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "file", 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "file,syslog", 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: "syslog" 2024-03-18T01:47:42.842 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.843 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T01:47:42.843 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.843 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.843 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.843 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.843 INFO:teuthology.orchestra.run.smithi007.stdout: "cgroups_split": { 2024-03-18T01:47:42.843 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cgroups_split", 2024-03-18T01:47:42.845 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.845 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.845 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.845 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.845 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.845 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "config_checks_enabled": { 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_checks_enabled", 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.846 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "config_dashboard": { 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_dashboard", 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.847 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_alertmanager": { 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_alertmanager", 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T01:47:42.848 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_base": { 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_base", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.849 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Container image name, without the tag", 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_elasticsearch": { 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_elasticsearch", 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T01:47:42.850 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "elasticsearch container image", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_grafana": { 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_grafana", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.851 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_haproxy": { 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_haproxy", 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.852 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "HAproxy container image", 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_agent": { 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_agent", 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.853 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger agent container image", 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_collector": { 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_collector", 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T01:47:42.854 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger collector container image", 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_query": { 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_query", 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T01:47:42.855 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger query container image", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_keepalived": { 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_keepalived", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.856 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Keepalived container image", 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_loki": { 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_loki", 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.857 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Loki container image", 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_node_exporter": { 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_node_exporter", 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.858 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_nvmeof": { 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_nvmeof", 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.859 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Nvme-of container image", 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_prometheus": { 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_prometheus", 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.860 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_promtail": { 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_promtail", 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.861 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Promtail container image", 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_snmp_gateway": { 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_snmp_gateway", 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T01:47:42.862 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SNMP Gateway container image", 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "container_init": { 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_init", 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.863 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "daemon_cache_timeout": { 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "daemon_cache_timeout", 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.864 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "default_cephadm_command_timeout": { 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.865 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "900", 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "default_registry": { 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_registry", 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.866 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "device_cache_timeout": { 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_cache_timeout", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.867 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1800", 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache device inventory", 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "device_enhanced_scan": { 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_enhanced_scan", 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.868 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "facts_cache_timeout": { 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "facts_cache_timeout", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:42.869 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache host facts data", 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "host_check_interval": { 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "host_check_interval", 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.870 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to perform a host check", 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "hw_monitoring": { 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hw_monitoring", 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.871 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "inventory_list_all": { 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "inventory_list_all", 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.872 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.873 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.874 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "log_refresh_metadata": { 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_refresh_metadata", 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.875 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.876 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.877 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.878 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.879 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.880 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "*", 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "max_count_per_host": { 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_count_per_host", 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.881 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of daemons per service per host", 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd_draining_count": { 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_osd_draining_count", 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.882 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "migration_current": { 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "migration_current", 2024-03-18T01:47:42.883 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "internal - do not modify", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.884 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "root", 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm-package", 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "root" 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.885 INFO:teuthology.orchestra.run.smithi007.stdout: "oob_default_addr": { 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "oob_default_addr", 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "169.254.1.1", 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.886 INFO:teuthology.orchestra.run.smithi007.stdout: "prometheus_alerts_path": { 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus_alerts_path", 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.887 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_insecure": { 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_insecure", 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.888 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_password": { 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_password", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_url": { 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_url", 2024-03-18T01:47:42.889 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_username": { 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_username", 2024-03-18T01:47:42.890 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "secure_monitoring_stack": { 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "secure_monitoring_stack", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.891 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "service_discovery_port": { 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "service_discovery_port", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.892 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8765", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "cephadm service discovery port", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "ssh_config_file": { 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssh_config_file", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.893 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "use_agent": { 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_agent", 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.894 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "use_repo_digest": { 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_repo_digest", 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.895 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_failed_host_check": { 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_failed_host_check", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.896 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_daemons": { 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_daemons", 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.897 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_hosts": { 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_hosts", 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.898 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crash", 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.899 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.900 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.901 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.902 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.903 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "retain_interval": { 2024-03-18T01:47:42.904 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retain_interval", 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "31536000", 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_recent_interval": { 2024-03-18T01:47:42.905 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_recent_interval", 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1209600", 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.906 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dashboard", 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.907 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.908 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.909 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_ENABLED": { 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.910 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.911 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.912 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.913 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.914 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.915 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.916 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.917 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.918 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.919 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.920 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.921 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.922 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_URL": { 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_URL", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.923 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:42.924 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.925 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.926 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.927 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.928 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "{}", 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.929 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.930 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.931 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.932 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.933 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.934 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.935 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.936 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.937 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.938 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.939 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.940 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T01:47:42.941 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8", 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.942 INFO:teuthology.orchestra.run.smithi007.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "45", 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T01:47:42.943 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T01:47:42.944 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T01:47:42.945 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T01:47:42.946 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.947 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.948 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:42.949 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.950 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "cross_origin_url": { 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cross_origin_url", 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.951 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "crt_file": { 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crt_file", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.952 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "debug": { 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "debug", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable/disable debug options", 2024-03-18T01:47:42.953 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "jwt_token_ttl": { 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "jwt_token_ttl", 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "28800", 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.954 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.955 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.956 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.957 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.958 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.959 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "motd": { 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "motd", 2024-03-18T01:47:42.960 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "The message of the day", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_resolve_ip_addr": { 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.961 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.962 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.963 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8080", 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.964 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl_server_port": { 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl_server_port", 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.965 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8443", 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-03-18T01:47:42.966 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "redirect", 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect" 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.967 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.968 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "url_prefix": { 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url_prefix", 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.969 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "devicehealth", 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_monitoring": { 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_monitoring", 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.970 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "monitor device health metrics", 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.971 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.972 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.973 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.974 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.975 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "mark_out_threshold": { 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mark_out_threshold", 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.976 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2419200", 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": { 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_name", 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.977 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "device_health_metrics", 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "retention_period": { 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retention_period", 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.978 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15552000", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain device health metrics", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_frequency": { 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_frequency", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.979 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "self_heal": { 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "self_heal", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:42.980 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.981 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_threshold": { 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_threshold", 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.982 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7257600", 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.983 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "diskprediction_local", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.984 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.985 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.986 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.987 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.988 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "predict_interval": { 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predict_interval", 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.989 INFO:teuthology.orchestra.run.smithi007.stdout: "predictor_model": { 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predictor_model", 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "prophetstor", 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.990 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.991 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "influx", 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": false, 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "influxdb python module not found", 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "batch_size": { 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "batch_size", 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5000", 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.992 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "database": { 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "database", 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "ceph", 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.993 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": { 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hostname", 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.994 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server hostname", 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "30", 2024-03-18T01:47:42.995 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "5", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.996 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.997 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:42.998 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:42.999 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.000 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "password": { 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "password", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.001 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "password of InfluxDB server user", 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "port": { 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "port", 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8086", 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.002 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server port", 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "false", 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.003 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "threads": { 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threads", 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:43.004 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "32", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "username": { 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "username", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.005 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "username of InfluxDB server user", 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "verify_ssl": { 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "verify_ssl", 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "true", 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.006 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "insights", 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.007 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.008 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.009 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.010 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.011 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "iostat", 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.012 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.013 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.014 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.015 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.016 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "k8sevents", 2024-03-18T01:47:43.017 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_event_retention_days": { 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ceph_event_retention_days", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.018 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "config_check_secs": { 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_check_secs", 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "10", 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.019 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.020 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.021 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.022 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.023 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "localpool", 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.024 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "failure_domain": { 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "failure_domain", 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "host", 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "failure domain for any created local pool", 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.025 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.026 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.029 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": { 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_size", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.030 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default min_size for any created local pool", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "num_rep": { 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "num_rep", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default replica count for any created local pool", 2024-03-18T01:47:43.031 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": { 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_num", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "128", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.032 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "prefix": { 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prefix", 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.033 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name prefix for any created local pool", 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "subtree": { 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "subtree", 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "rack", 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.034 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mds_autoscaler", 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.035 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.036 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.037 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.038 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.039 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirroring", 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.040 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.041 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.042 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.043 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.044 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "nfs", 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.045 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.046 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.047 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.048 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.049 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-03-18T01:47:43.050 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "fail_fs": { 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "fail_fs", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.051 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.052 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.053 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.054 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.055 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator": { 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.056 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm", 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "rook", 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "test_orchestrator" 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Orchestrator backend", 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_perf_query", 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.058 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.059 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.060 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.061 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.062 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_support", 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.063 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.064 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.065 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.066 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.067 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.068 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_autoscaler", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.069 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.070 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.071 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.073 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.074 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "threshold": { 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threshold", 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1.0", 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "scaling threshold", 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T01:47:43.075 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "progress", 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_pg_recovery_event": { 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_pg_recovery_event", 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.076 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.077 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.078 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.079 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.080 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.081 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.082 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "max_completed_events": { 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_completed_events", 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.083 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "50", 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "number of past completed events to remember", 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.084 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long the module is going to sleep", 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus", 2024-03-18T01:47:43.085 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "cache": { 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cache", 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.086 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "exclude_perf_counters": { 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "exclude_perf_counters", 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T01:47:43.087 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.088 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.089 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.090 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.091 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.092 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools": { 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools", 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.093 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "300", 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.094 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_interval": { 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_interval", 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15.0", 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.095 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.096 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "9283", 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.097 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "stale_cache_strategy": { 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stale_cache_strategy", 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "log", 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.098 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "default", 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.099 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "default", 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "error" 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.100 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_support", 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.101 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.102 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.103 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.104 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.105 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_snap_create": { 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_snap_create", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.106 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "mirror_snapshot_schedule": { 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.107 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "trash_purge_schedule": { 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "trash_purge_schedule", 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.108 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "restful", 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.109 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_auth": { 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_auth", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.110 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.111 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.112 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.113 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.114 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.115 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.116 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.117 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.118 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rgw", 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.119 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.120 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.121 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.122 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.123 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rook", 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.124 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.125 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.126 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.127 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.128 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_class": { 2024-03-18T01:47:43.129 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "storage_class", 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "local", 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.130 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "selftest", 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.131 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.132 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.133 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.134 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.135 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "roption1": { 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption1", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "roption2": { 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption2", 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "xyz", 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.137 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption1": { 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption1", 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.138 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption2": { 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption2", 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.139 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption3": { 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption3", 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.140 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption4": { 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption4", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.141 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption5": { 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption5", 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.142 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption6": { 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption6", 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.143 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption7": { 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption7", 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "42", 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.144 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "testkey": { 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testkey", 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.145 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "testlkey": { 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testlkey", 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.146 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "testnewline": { 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testnewline", 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.147 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snap_schedule", 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_m_granularity": { 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_m_granularity", 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.148 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "dump_on_update": { 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dump_on_update", 2024-03-18T01:47:43.149 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "dump database to debug log on update", 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.150 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.151 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.153 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.154 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stats", 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.155 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.156 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.157 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.158 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.159 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "status", 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.160 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.161 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.162 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.163 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.164 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.165 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telegraf", 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "address": { 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "address", 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.166 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.167 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.168 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.169 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.170 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.171 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.172 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telemetry", 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_basic": { 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_basic", 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.173 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_crash": { 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_crash", 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.174 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_device": { 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_device", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.175 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_ident": { 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_ident", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.176 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_perf": { 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_perf", 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.177 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "contact": { 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "contact", 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.178 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "description": { 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "description", 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.179 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "device_url": { 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_url", 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.180 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.181 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "24", 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "8", 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.182 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "last_opt_revision": { 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "last_opt_revision", 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1", 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.183 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard": { 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard", 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.184 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard_description": { 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard_description", 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.185 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.186 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.187 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.188 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.189 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "organization": { 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "organization", 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.190 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "proxy": { 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "proxy", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.191 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "url": { 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url", 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.192 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test_orchestrator", 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.193 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.194 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.195 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.196 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.197 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "volumes", 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.198 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.199 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.200 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.201 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_clones": { 2024-03-18T01:47:43.202 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_clones", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "periodic_async_work": { 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "periodic_async_work", 2024-03-18T01:47:43.203 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Periodically check for async work", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_delay": { 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_delay", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.204 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_no_wait": { 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.205 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix", 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.206 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "discovery_interval": { 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "discovery_interval", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "100", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.207 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "identifier": { 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "identifier", 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.208 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.209 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.210 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.211 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.212 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.213 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_host": { 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_host", 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.214 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_port": { 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_port", 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.215 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10051", 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_sender": { 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_sender", 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.216 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "modules": [ 2024-03-18T01:47:43.217 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm", 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "dashboard", 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "iostat", 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "nfs", 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "restful" 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "available_modules": [ 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "alerts", 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.218 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.219 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.220 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.221 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.222 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.223 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_destination": { 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_destination", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.224 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email address to send alerts to", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_from_name": { 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_from_name", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "Ceph", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.225 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Email From: name", 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_host": { 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_host", 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.226 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP server", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_password": { 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_password", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Password to authenticate with", 2024-03-18T01:47:43.227 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_port": { 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_port", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "465", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP port", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.228 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_sender": { 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_sender", 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SMTP envelope sender", 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_ssl": { 2024-03-18T01:47:43.229 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_ssl", 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.230 INFO:teuthology.orchestra.run.smithi007.stdout: "smtp_user": { 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "smtp_user", 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "User to authenticate as", 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.231 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "balancer", 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "active": { 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "active", 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.232 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_time": { 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_time", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0000", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:47:43.233 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "begin_weekday": { 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "begin_weekday", 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:47:43.234 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_max_iterations": { 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_max_iterations", 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "25", 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "250", 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.235 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_metrics": { 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_metrics", 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.236 INFO:teuthology.orchestra.run.smithi007.stdout: "crush_compat_step": { 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crush_compat_step", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.5", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0.001", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "0.999", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "aggressiveness of optimization", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "end_time": { 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_time", 2024-03-18T01:47:43.237 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2359", 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "end_weekday": { 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "end_weekday", 2024-03-18T01:47:43.238 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "0", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "6", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.239 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.240 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.241 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.242 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.243 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "min_score": { 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_score", 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.244 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.245 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "upmap", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "crush-compat", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "none", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "read", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap-read" 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Balancer mode", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.246 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_ids": { 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_ids", 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.247 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.248 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_deviation": { 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_deviation", 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "upmap_max_optimizations": { 2024-03-18T01:47:43.249 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "upmap_max_optimizations", 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.250 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm", 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_down_multiplier": { 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_down_multiplier", 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.251 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_refresh_rate": { 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_refresh_rate", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "20", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T01:47:43.252 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "agent_starting_port": { 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "agent_starting_port", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4721", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.253 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_ptrace": { 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_ptrace", 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T01:47:43.254 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_interval": { 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_interval", 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.255 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "autotune_memory_target_ratio": { 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0.7", 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.256 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm_log_destination": { 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cephadm_log_destination", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "file", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "file,syslog", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "syslog" 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.257 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "cgroups_split": { 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cgroups_split", 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.258 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "config_checks_enabled": { 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_checks_enabled", 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.259 INFO:teuthology.orchestra.run.smithi007.stdout: "config_dashboard": { 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_dashboard", 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_alertmanager": { 2024-03-18T01:47:43.260 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_alertmanager", 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_base": { 2024-03-18T01:47:43.261 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_base", 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Container image name, without the tag", 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.262 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_elasticsearch": { 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_elasticsearch", 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "elasticsearch container image", 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.263 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_grafana": { 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_grafana", 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.264 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_haproxy": { 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_haproxy", 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "HAproxy container image", 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.265 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_agent": { 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_agent", 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger agent container image", 2024-03-18T01:47:43.266 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_collector": { 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_collector", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger collector container image", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.267 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_jaeger_query": { 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_jaeger_query", 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Jaeger query container image", 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.268 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_keepalived": { 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_keepalived", 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Keepalived container image", 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.269 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_loki": { 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_loki", 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Loki container image", 2024-03-18T01:47:43.270 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_node_exporter": { 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_node_exporter", 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:43.271 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_nvmeof": { 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_nvmeof", 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Nvme-of container image", 2024-03-18T01:47:43.272 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_prometheus": { 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_prometheus", 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Prometheus container image", 2024-03-18T01:47:43.273 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_promtail": { 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_promtail", 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Promtail container image", 2024-03-18T01:47:43.274 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "container_image_snmp_gateway": { 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_image_snmp_gateway", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "SNMP Gateway container image", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.275 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "container_init": { 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "container_init", 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T01:47:43.276 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "daemon_cache_timeout": { 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "daemon_cache_timeout", 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.277 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "default_cephadm_command_timeout": { 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "900", 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.278 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "default_registry": { 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default_registry", 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "docker.io", 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.279 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "device_cache_timeout": { 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_cache_timeout", 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1800", 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.280 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache device inventory", 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "device_enhanced_scan": { 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_enhanced_scan", 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.281 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "facts_cache_timeout": { 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "facts_cache_timeout", 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.282 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "seconds to cache host facts data", 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "host_check_interval": { 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "host_check_interval", 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.283 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to perform a host check", 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "hw_monitoring": { 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hw_monitoring", 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.284 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "inventory_list_all": { 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "inventory_list_all", 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.285 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.286 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.287 INFO:teuthology.orchestra.run.smithi007.stdout: "log_refresh_metadata": { 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_refresh_metadata", 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.288 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.289 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.290 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.291 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.292 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "*", 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.293 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "max_count_per_host": { 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_count_per_host", 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of daemons per service per host", 2024-03-18T01:47:43.294 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "max_osd_draining_count": { 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_osd_draining_count", 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T01:47:43.295 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "migration_current": { 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "migration_current", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "internal - do not modify", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.296 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "mode": { 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mode", 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "root", 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm-package", 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: "root" 2024-03-18T01:47:43.297 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "oob_default_addr": { 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "oob_default_addr", 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "169.254.1.1", 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.298 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "prometheus_alerts_path": { 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus_alerts_path", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.299 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_insecure": { 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_insecure", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.300 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_password": { 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_password", 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.301 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_url": { 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_url", 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.302 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "registry_username": { 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "registry_username", 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.303 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "secure_monitoring_stack": { 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "secure_monitoring_stack", 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "service_discovery_port": { 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "service_discovery_port", 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8765", 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "cephadm service discovery port", 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "ssh_config_file": { 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssh_config_file", 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.306 INFO:teuthology.orchestra.run.smithi007.stdout: "use_agent": { 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_agent", 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.307 INFO:teuthology.orchestra.run.smithi007.stdout: "use_repo_digest": { 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "use_repo_digest", 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.308 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_failed_host_check": { 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_failed_host_check", 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.309 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_daemons": { 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_daemons", 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.310 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_on_stray_hosts": { 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_on_stray_hosts", 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.311 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crash", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.312 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.313 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.314 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.315 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.316 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "retain_interval": { 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retain_interval", 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "31536000", 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.317 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_recent_interval": { 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_recent_interval", 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1209600", 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.318 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dashboard", 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_ENABLED": { 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.326 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.327 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T01:47:43.328 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.329 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.330 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.331 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.332 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.333 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_URL": { 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_URL", 2024-03-18T01:47:43.334 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T01:47:43.335 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T01:47:43.336 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T01:47:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T01:47:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "{}", 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.341 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.342 INFO:teuthology.orchestra.run.smithi007.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.343 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.344 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.345 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.346 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.347 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.348 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.349 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.350 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.351 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.352 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.353 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8", 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.354 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "45", 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.355 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.356 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "admin", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.357 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.358 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.359 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.360 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.361 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.362 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.363 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "cross_origin_url": { 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cross_origin_url", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.364 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "crt_file": { 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "crt_file", 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.365 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "debug": { 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "debug", 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Enable/disable debug options", 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.366 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "jwt_token_ttl": { 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "jwt_token_ttl", 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "28800", 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-03-18T01:47:43.367 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.368 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.370 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.371 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.372 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "motd": { 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "motd", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "The message of the day", 2024-03-18T01:47:43.373 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_resolve_ip_addr": { 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.374 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.375 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8080", 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.376 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.377 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl_server_port": { 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl_server_port", 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8443", 2024-03-18T01:47:43.378 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "redirect", 2024-03-18T01:47:43.379 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect" 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.380 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "url_prefix": { 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url_prefix", 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.381 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.382 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "devicehealth", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_monitoring": { 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_monitoring", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "monitor device health metrics", 2024-03-18T01:47:43.383 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.384 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.385 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.386 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.387 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.388 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "mark_out_threshold": { 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mark_out_threshold", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "2419200", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.389 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "pool_name": { 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pool_name", 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "device_health_metrics", 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T01:47:43.390 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "retention_period": { 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "retention_period", 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15552000", 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long to retain device health metrics", 2024-03-18T01:47:43.391 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_frequency": { 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_frequency", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.392 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "self_heal": { 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "self_heal", 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T01:47:43.393 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.394 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "warn_threshold": { 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "warn_threshold", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7257600", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T01:47:43.395 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "diskprediction_local", 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.396 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.397 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.398 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.399 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.400 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "predict_interval": { 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predict_interval", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "86400", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.401 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "predictor_model": { 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "predictor_model", 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.402 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "prophetstor", 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.403 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "600", 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "influx", 2024-03-18T01:47:43.404 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": false, 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "influxdb python module not found", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "batch_size": { 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "batch_size", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5000", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.405 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "database": { 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "database", 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "ceph", 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.406 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "hostname": { 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "hostname", 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server hostname", 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.407 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "30", 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "5", 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.408 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.409 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.410 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.411 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.412 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.413 INFO:teuthology.orchestra.run.smithi007.stdout: "password": { 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "password", 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "password of InfluxDB server user", 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.414 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "port": { 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "port", 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "8086", 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "InfluxDB server port", 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.415 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "ssl": { 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ssl", 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "false", 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.416 INFO:teuthology.orchestra.run.smithi007.stdout: "threads": { 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threads", 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "32", 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.417 INFO:teuthology.orchestra.run.smithi007.stdout: "username": { 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "username", 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "username of InfluxDB server user", 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.418 INFO:teuthology.orchestra.run.smithi007.stdout: "verify_ssl": { 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "verify_ssl", 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "true", 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "insights", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.420 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.421 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.422 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.423 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.424 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "iostat", 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.425 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.426 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.427 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.428 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.429 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.430 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "k8sevents", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_event_retention_days": { 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "ceph_event_retention_days", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "7", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T01:47:43.431 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "config_check_secs": { 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "config_check_secs", 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "10", 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.432 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.433 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.434 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.435 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.436 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.437 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "localpool", 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "failure_domain": { 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "failure_domain", 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.438 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "host", 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "failure domain for any created local pool", 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.439 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.440 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.441 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.442 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.443 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "min_size": { 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "min_size", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.444 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default min_size for any created local pool", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "num_rep": { 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "num_rep", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default replica count for any created local pool", 2024-03-18T01:47:43.445 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_num": { 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_num", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "128", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.446 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "prefix": { 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prefix", 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "name prefix for any created local pool", 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.447 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "subtree": { 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "subtree", 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "rack", 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.448 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mds_autoscaler", 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.449 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.450 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.451 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.452 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.453 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirroring", 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.454 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.455 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.456 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.457 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.458 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.459 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.461 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.461 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.461 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.461 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.461 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.462 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "nfs", 2024-03-18T01:47:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.463 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.464 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.465 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.466 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.467 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.468 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.469 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "fail_fs": { 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "fail_fs", 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.471 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.472 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.473 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.474 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.475 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator": { 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "orchestrator", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "cephadm", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "rook", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "test_orchestrator" 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Orchestrator backend", 2024-03-18T01:47:43.476 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_perf_query", 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.477 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.478 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.479 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.480 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.481 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "osd_support", 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.482 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.483 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.484 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.485 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.486 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "pg_autoscaler", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.488 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.489 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.490 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.491 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:43.492 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "threshold": { 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "threshold", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "3.0", 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1.0", 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "scaling threshold", 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.494 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "progress", 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_pg_recovery_event": { 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_pg_recovery_event", 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.495 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.496 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.497 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.498 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.499 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.500 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.501 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "max_completed_events": { 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_completed_events", 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.502 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "50", 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "number of past completed events to remember", 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "sleep_interval": { 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "sleep_interval", 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.503 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "5", 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "how long the module is going to sleep", 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "prometheus", 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.504 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "cache": { 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "cache", 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.505 INFO:teuthology.orchestra.run.smithi007.stdout: "exclude_perf_counters": { 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "exclude_perf_counters", 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.507 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.508 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.509 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.510 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools": { 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools", 2024-03-18T01:47:43.511 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.512 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "300", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "scrape_interval": { 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "scrape_interval", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15.0", 2024-03-18T01:47:43.513 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.514 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "::", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "9283", 2024-03-18T01:47:43.515 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "stale_cache_strategy": { 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stale_cache_strategy", 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "log", 2024-03-18T01:47:43.516 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_behaviour": { 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_behaviour", 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "default", 2024-03-18T01:47:43.517 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "default", 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "error" 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "standby_error_status_code": { 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "standby_error_status_code", 2024-03-18T01:47:43.518 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "500", 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "400", 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "599", 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_support", 2024-03-18T01:47:43.519 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.520 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.521 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.522 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.523 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_snap_create": { 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_snap_create", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.524 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "mirror_snapshot_schedule": { 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.525 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "trash_purge_schedule": { 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "trash_purge_schedule", 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.526 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "restful", 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.527 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "enable_auth": { 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enable_auth", 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.528 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "key_file": { 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "key_file", 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.529 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.530 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.531 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.532 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.533 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "server_addr": { 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_addr", 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.534 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "server_port": { 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "server_port", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.535 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rgw", 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.536 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.537 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.538 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.539 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.540 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rook", 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.541 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.542 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.543 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.544 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.545 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_class": { 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "storage_class", 2024-03-18T01:47:43.546 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "local", 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.547 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "selftest", 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.548 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.549 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.550 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.551 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.552 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "roption1": { 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption1", 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.553 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "roption2": { 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "roption2", 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "xyz", 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.554 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption1": { 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption1", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.555 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption2": { 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption2", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.556 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption3": { 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption3", 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "float", 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.557 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption4": { 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption4", 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.558 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption5": { 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption5", 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.559 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption6": { 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption6", 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "rwoption7": { 2024-03-18T01:47:43.560 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rwoption7", 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "1", 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "42", 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "testkey": { 2024-03-18T01:47:43.561 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testkey", 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.562 INFO:teuthology.orchestra.run.smithi007.stdout: "testlkey": { 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testlkey", 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.563 INFO:teuthology.orchestra.run.smithi007.stdout: "testnewline": { 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "testnewline", 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.564 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snap_schedule", 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "allow_m_granularity": { 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "allow_m_granularity", 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.565 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "dump_on_update": { 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "dump_on_update", 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.566 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "dump database to debug log on update", 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.567 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.568 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.569 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.570 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.571 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "stats", 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.572 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.573 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.574 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.575 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.576 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "status", 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.577 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.578 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.579 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.580 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.581 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telegraf", 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "address": { 2024-03-18T01:47:43.582 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "address", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.583 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "15", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.584 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.585 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.586 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.587 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.588 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "telemetry", 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_basic": { 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_basic", 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.589 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_crash": { 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_crash", 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.590 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_device": { 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_device", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.591 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_ident": { 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_ident", 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.592 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "channel_perf": { 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "channel_perf", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.593 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "contact": { 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "contact", 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.594 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "description": { 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "description", 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.595 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "device_url": { 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "device_url", 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T01:47:43.596 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": { 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "enabled", 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.597 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.598 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "24", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "8", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "last_opt_revision": { 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "last_opt_revision", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "1", 2024-03-18T01:47:43.599 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard": { 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard", 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.600 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "leaderboard_description": { 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "leaderboard_description", 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.601 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.602 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.603 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.604 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.605 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.606 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "organization": { 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "organization", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.607 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "proxy": { 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "proxy", 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "url": { 2024-03-18T01:47:43.608 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "url", 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.609 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "test_orchestrator", 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.610 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.611 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.612 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.613 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.614 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "volumes", 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.615 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.616 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.617 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.618 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.619 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "max_concurrent_clones": { 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "max_concurrent_clones", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "4", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.620 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "periodic_async_work": { 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "periodic_async_work", 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Periodically check for async work", 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.621 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_delay": { 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_delay", 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "0", 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "snapshot_clone_no_wait": { 2024-03-18T01:47:43.622 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "True", 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.623 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "can_run": true, 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "error_string": "", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "module_options": { 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "discovery_interval": { 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "discovery_interval", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "uint", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "100", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.624 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "identifier": { 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "identifier", 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.625 INFO:teuthology.orchestra.run.smithi007.stdout: "interval": { 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "interval", 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "secs", 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "60", 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.626 INFO:teuthology.orchestra.run.smithi007.stdout: "log_level": { 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_level", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.627 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster": { 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.628 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_cluster_level": { 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_cluster_level", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "info", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [ 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "critical", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "debug", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "error", 2024-03-18T01:47:43.629 INFO:teuthology.orchestra.run.smithi007.stdout: "info", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "warning" 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "log_to_file": { 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "log_to_file", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "bool", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 1, 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "False", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.630 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_host": { 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_host", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.631 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_port": { 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_port", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "int", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "10051", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.632 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "zabbix_sender": { 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "zabbix_sender", 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "str", 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "level": "advanced", 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "flags": 0, 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "min": "", 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "max": "", 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "enum_allowed": [], 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "desc": "", 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "long_desc": "", 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: "see_also": [] 2024-03-18T01:47:43.633 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "services": { 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "dashboard": "https://172.21.15.7:8443/" 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "always_on_modules": { 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "octopus": [ 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-03-18T01:47:43.634 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "pacific": [ 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-03-18T01:47:43.635 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "quincy": [ 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "reef": [ 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-03-18T01:47:43.636 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "squid": [ 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "balancer", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "crash", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "devicehealth", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "orchestrator", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "pg_autoscaler", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "progress", 2024-03-18T01:47:43.637 INFO:teuthology.orchestra.run.smithi007.stdout: "rbd_support", 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "status", 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "telemetry", 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "volumes" 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "last_failure_osd_epoch": 3, 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "active_clients": [ 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "devicehealth", 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 1840290075 2024-03-18T01:47:43.638 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "libcephsqlite", 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 217421055 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "rbd_support", 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-03-18T01:47:43.639 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 2840610426 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "volumes", 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: "addrvec": [ 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: "type": "v2", 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: "addr": "172.21.15.7:0", 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: "nonce": 930522449 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:43.640 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:43.641 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-03-18T01:47:43.641 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:42 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2380102995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:47:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:42 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/4132218834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:47:43.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:43 smithi203 ceph-mon[36345]: pgmap v703: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T01:47:43.643 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:43 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3713559625' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T01:47:43.643 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:42 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2380102995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:47:43.643 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:42 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/4132218834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:47:43.643 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:43 smithi046 ceph-mon[36108]: pgmap v703: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T01:47:43.643 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:43 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3713559625' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T01:47:43.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52900a2880 msgr2=0x7f52900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:43.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52900a2880 0x7f52900a4d40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f529c009530 tx=0x7f529c009260 comp rx=0 tx=0).stop 2024-03-18T01:47:43.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 msgr2=0x7f52ac19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 0x7f52ac19e660 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f52980349e0 tx=0x7f52980042f0 comp rx=0 tx=0).stop 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 shutdown_connections 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f52900a2880 0x7f52900a4d40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f52ac077d80 0x7f52ac1a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f52ac0773e0 0x7f52ac19e660 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 --2- 172.21.15.7:0/3713559625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52ac0761a0 0x7f52ac19e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.374+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 >> 172.21.15.7:0/3713559625 conn(0x7f52ac0ff040 msgr2=0x7f52ac100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.375+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 shutdown_connections 2024-03-18T01:47:43.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:47:42.375+0000 7f52b0c00640 1 -- 172.21.15.7:0/3713559625 wait complete. 2024-03-18T01:47:43.650 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T01:47:43.650 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T01:47:43.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:42 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2380102995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:47:43.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:42 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/4132218834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:47:43.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:43 smithi007 ceph-mon[31996]: pgmap v703: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T01:47:43.651 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:43 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3713559625' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T01:47:44.160 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:47:45.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:45 smithi046 ceph-mon[36108]: pgmap v704: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:45.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:45 smithi007 ceph-mon[31996]: pgmap v704: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:45.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:45 smithi203 ceph-mon[36345]: pgmap v704: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 213, 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 210, 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 474912, 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 129184, 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 4, 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 0, 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.085624825, 2024-03-18T01:47:47.482 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.042878326, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.037712756, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.007859126, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 209, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.047051030, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000225124 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 474912, 2024-03-18T01:47:47.483 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 129184 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 5, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 161, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 3152, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 106080, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 3, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 2, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.027117054, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.020911672, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.003893176, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T01:47:47.484 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 160, 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.034506349, 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000215664 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 106080 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.485 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 0, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 2, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 0, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 0, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 2, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 1, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.000825497, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.000658471, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:47.486 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "cct": { 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "total_workers": 0, 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: "unhealthy_workers": 0 2024-03-18T01:47:47.487 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool": { 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_items": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:47:47.488 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_items": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_items": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_items": 0, 2024-03-18T01:47:47.489 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_items": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_bytes": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_items": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_bytes": 148967, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_items": 29, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:47:47.490 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_items": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bytes": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_items": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_items": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_items": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_bytes": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_items": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_bytes": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_items": 0, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_bytes": 928, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_items": 2, 2024-03-18T01:47:47.491 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_bytes": 0, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_items": 0, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_bytes": 0, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_items": 0 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 214, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 444812, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 214, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 214, 2024-03-18T01:47:47.492 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 444812, 2024-03-18T01:47:47.493 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:47.493 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:47.493 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:47.493 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:47.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:47.493 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:47.493 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:47.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:47 smithi007 ceph-mon[31996]: pgmap v705: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:47:47.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:47 smithi046 ceph-mon[36108]: pgmap v705: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:47:47.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:47 smithi203 ceph-mon[36345]: pgmap v705: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:47:48.387 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-18T01:47:48.387 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-18T01:47:48.388 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-18T01:47:48.388 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T01:47:48.388 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T01:47:48.877 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:47:49.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:49 smithi007 ceph-mon[31996]: pgmap v706: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:49.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:49 smithi046 ceph-mon[36108]: pgmap v706: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:49.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:49 smithi203 ceph-mon[36345]: pgmap v706: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:51.149 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 219, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 216, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 476544, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 133024, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 4, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 0, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.069522550, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.032903231, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.026204366, 2024-03-18T01:47:51.150 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.008262166, 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 215, 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.034521583, 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000160565 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 476544, 2024-03-18T01:47:51.151 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 133024 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 5, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 167, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 3152, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 107136, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 3, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 2, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.025481769, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.020712141, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.002436688, 2024-03-18T01:47:51.152 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 166, 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.022380403, 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000134821 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.153 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 107136 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 0, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 2, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 0, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 0, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 2, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 1, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.001106082, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T01:47:51.154 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.000897542, 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.155 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "cct": { 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "total_workers": 0, 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "unhealthy_workers": 0 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool": { 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_items": 0, 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:47:51.156 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:47:51.157 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_items": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_items": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_items": 0, 2024-03-18T01:47:51.158 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_items": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_bytes": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_items": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_bytes": 148967, 2024-03-18T01:47:51.159 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_items": 29, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_items": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bytes": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_items": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_items": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_items": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_bytes": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_items": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:47:51.160 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_bytes": 0, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_items": 0, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_bytes": 928, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_items": 2, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_bytes": 0, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_items": 0, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_bytes": 0, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_items": 0 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-03-18T01:47:51.161 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 220, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 445592, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 220, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 220, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 445592, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:51.162 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:51.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:51.163 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:51.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:51 smithi046 ceph-mon[36108]: pgmap v707: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:47:51.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:51 smithi007 ceph-mon[31996]: pgmap v707: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:47:51.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:51 smithi203 ceph-mon[36345]: pgmap v707: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:47:51.969 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-18T01:47:51.969 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-18T01:47:51.969 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-18T01:47:51.969 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T01:47:51.970 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T01:47:52.522 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:47:53.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:53 smithi046 ceph-mon[36108]: pgmap v708: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:47:53.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:53 smithi007 ceph-mon[31996]: pgmap v708: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:47:53.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:53 smithi203 ceph-mon[36345]: pgmap v708: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:47:55.027 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:55.027 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:47:55.027 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 5634, 2024-03-18T01:47:55.027 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 5447, 2024-03-18T01:47:55.027 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 2596877, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 5225373, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 14, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 4, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 1.860777103, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 1.278219677, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.757493382, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.040851246, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 5448, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 3.844116721, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000705601 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:55.028 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 8724, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.004848879, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000555 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 801040, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 1044528 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 932, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 988, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 1801986, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 690016, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 12, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 6, 2024-03-18T01:47:55.029 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.155613432, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.053014195, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.108184107, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.018522934, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 989, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.386962920, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000391266 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 740, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000412144, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000556 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.030 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 1348336, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 147888 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 552, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 716, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 294204, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 836585, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 20, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 8, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.142805302, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.060750714, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.045690270, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.032035305, 2024-03-18T01:47:55.031 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 710, 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.269667095, 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000379812 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 126, 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000099137, 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000786 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 360400 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "cct": { 2024-03-18T01:47:55.032 INFO:teuthology.orchestra.run.smithi007.stdout: "total_workers": 1, 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "unhealthy_workers": 0 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "finisher-MDSRank": { 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "queue_len": 0, 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "complete_latency": { 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 744, 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 1.081865454, 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.001454120 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "finisher-PurgeQueue": { 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "queue_len": 0, 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "complete_latency": { 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 488, 2024-03-18T01:47:55.033 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.033542388, 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000068734 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": { 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "request": 133, 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "reply": 124, 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "reply_latency": { 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 124, 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.303407366, 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.002446833 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "slow_reply": 0, 2024-03-18T01:47:55.034 INFO:teuthology.orchestra.run.smithi007.stdout: "forward": 9, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_fetch_complete": 0, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_fetch_keys": 0, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_commit": 11, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_split": 0, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_merge": 0, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes": 15, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_top": 1, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_bottom": 1, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_pin_tail": 13, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_pinned": 13, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_expired": 428, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_with_caps": 0, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": 0, 2024-03-18T01:47:55.035 INFO:teuthology.orchestra.run.smithi007.stdout: "subtrees": 3, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse": 336, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_hit": 263, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_forward": 8, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_discover": 35, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_remote_ino": 0, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_lock": 0, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "load_cent": 3000, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "q": 3, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "exported": 2, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "exported_inodes": 324, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "imported": 36, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "imported_inodes": 334, 2024-03-18T01:47:55.036 INFO:teuthology.orchestra.run.smithi007.stdout: "openino_dir_fetch": 0, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "openino_backtrace_fetch": 324, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "openino_peer_discover": 0, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "root_rfiles": 82, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "root_rbytes": 792295, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "root_rsnaps": 0, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_backtrace_fetch": 324, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_set_tag": 0, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_dir_inodes": 73, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_dirfrag_rstats": 73, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_file_inodes": 251, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:47:55.037 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_revoke": 24, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_grant": 211, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_caps": 1704, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_caps_dirty": 1532, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_cap_release": 12, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "process_request_cap_release": 66 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache": { 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_update": 0, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_update_receipt": 0, 2024-03-18T01:47:55.038 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_try_discover": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_send_discover": 97, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_handle_discover": 31, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "num_strays": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "num_strays_delayed": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "strays_created": 25, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "strays_enqueued": 25, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "strays_reintegrated": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "strays_migrated": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "num_recovering_processing": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_started": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_completed": 0, 2024-03-18T01:47:55.039 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_exportdir": 2, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_flush": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_fragstats": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_inodestats": 0 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log": { 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "evlrg": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "evadd": 1927, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "evex": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "evtrm": 0, 2024-03-18T01:47:55.040 INFO:teuthology.orchestra.run.smithi007.stdout: "ev": 1927, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "evexg": 0, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "evexd": 0, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "segadd": 3, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "segex": 0, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "segtrm": 0, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "seg": 3, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "segmjr": 2, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "segexg": 0, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "segexd": 0, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "expos": 4194304, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "wrpos": 7149192, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "rdpos": 0, 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "jlat": { 2024-03-18T01:47:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 232, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 6.675521718, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.028773800 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "replayed": 0 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_mem": { 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "ino": 19, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "ino+": 264, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "ino-": 245, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "dir": 18, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "dir+": 93, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "dir-": 75, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "dn": 15, 2024-03-18T01:47:55.042 INFO:teuthology.orchestra.run.smithi007.stdout: "dn+": 485, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "dn-": 470, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "cap": 0, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "cap+": 258, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "cap-": 258, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "rss": 59652, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "heap": 223612 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_server": { 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "dispatch_client_request": 173, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "dispatch_server_request": 31, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_request": 133, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_session": 9, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_peer_request": 96, 2024-03-18T01:47:55.043 INFO:teuthology.orchestra.run.smithi007.stdout: "req_create_latency": { 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 7, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.042275839, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.006039405 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "req_getattr_latency": { 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "req_getfilelock_latency": { 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.044 INFO:teuthology.orchestra.run.smithi007.stdout: "req_link_latency": { 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookup_latency": { 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 86, 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.090215268, 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.001049014 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookuphash_latency": { 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.045 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookupino_latency": { 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookupname_latency": { 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookupparent_latency": { 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.046 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookupsnap_latency": { 2024-03-18T01:47:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.047 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.048 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.048 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.048 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lssnap_latency": { 2024-03-18T01:47:55.048 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.048 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "req_mkdir_latency": { 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "req_mknod_latency": { 2024-03-18T01:47:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "req_mksnap_latency": { 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "req_open_latency": { 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 4, 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.012381088, 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.003095272 2024-03-18T01:47:55.050 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "req_readdir_latency": { 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 2, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.003371015, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.001685507 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "req_rename_latency": { 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 7, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.026850545, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.003835792 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "req_renamesnap_latency": { 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.051 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "req_snapdiff_latency": { 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "req_rmdir_latency": { 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "req_rmsnap_latency": { 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.052 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "req_rmxattr_latency": { 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setattr_latency": { 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.053 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setdirlayout_latency": { 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setfilelock_latency": { 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setlayout_latency": { 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.054 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setxattr_latency": { 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "req_symlink_latency": { 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.055 INFO:teuthology.orchestra.run.smithi007.stdout: "req_unlink_latency": { 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 18, 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.128313611, 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.007128533 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "req_getvxattr_latency": { 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_sessions": { 2024-03-18T01:47:55.056 INFO:teuthology.orchestra.run.smithi007.stdout: "session_count": 1, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "session_add": 1, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "session_remove": 0, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "sessions_open": 1, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "sessions_stale": 0, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "total_load": 44, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "average_load": 44, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "avg_session_uptime": 2020, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "mdthresh_evicted": 0 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool": { 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_items": 0, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:47:55.057 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_items": 0, 2024-03-18T01:47:55.058 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_items": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_items": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_items": 0, 2024-03-18T01:47:55.059 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_bytes": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_items": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_bytes": 324311, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_items": 129, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_bytes": 528, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_items": 6, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bytes": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_items": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_items": 0, 2024-03-18T01:47:55.060 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_items": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_bytes": 8696, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_items": 166, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_bytes": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_items": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_bytes": 76613, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_items": 328, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_bytes": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_items": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_bytes": 0, 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_items": 0 2024-03-18T01:47:55.061 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter": { 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "op_active": 0, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "op_laggy": 0, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "op_send": 1174, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "op_send_bytes": 2999522, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "op_resend": 28, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "op_reply": 1174, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "op_latency": { 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 1174, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 1401.871103094, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 1.194098043 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.062 INFO:teuthology.orchestra.run.smithi007.stdout: "op_inflight": 0, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "oplen_avg": { 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 1174, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 1257 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "op": 1174, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "op_r": 328, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "op_w": 846, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "op_rmw": 0, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "op_pg": 0, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_stat": 0, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_create": 10, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_read": 4, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_write": 236, 2024-03-18T01:47:55.063 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_writefull": 37, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_writesame": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_append": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_zero": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_truncate": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_delete": 494, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_mapext": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_sparse_read": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_clonerange": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_getxattr": 324, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_setxattr": 20, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_rmxattr": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:47:55.064 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_call": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_watch": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_notify": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_pgls": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_other": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "linger_active": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "linger_send": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "linger_resend": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "linger_ping": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "poolop_active": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "poolop_send": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "poolop_resend": 0, 2024-03-18T01:47:55.065 INFO:teuthology.orchestra.run.smithi007.stdout: "poolstat_active": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "poolstat_send": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "poolstat_resend": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "statfs_active": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "statfs_send": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "statfs_resend": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "command_active": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "command_send": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "command_resend": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "map_epoch": 91, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "map_full": 0, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "map_inc": 2, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_sessions": 12, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_session_open": 12, 2024-03-18T01:47:55.066 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_session_close": 0, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_laggy": 0, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_wr": 94, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_rd": 0, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_del": 38 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "oft": { 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_total_objs": 1, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_total_updates": 0, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_total_removes": 0 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "osd-slow-ops": { 2024-03-18T01:47:55.067 INFO:teuthology.orchestra.run.smithi007.stdout: "slow_ops_count": 0 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "purge_queue": { 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executing_ops": 0, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executing_ops_high_water": 26, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executing": 0, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executing_high_water": 5, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executed_ops": 482, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executed": 25, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_item_in_journal": 0 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-03-18T01:47:55.068 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 7114, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 3944878, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 7114, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 7114, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 3944878, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.069 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-objecter_bytes": { 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 0, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 0, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 0, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 1174, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 3002247, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 1174, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 3002247, 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:55.070 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-objecter_ops": { 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 1024, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 0, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 0, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 0, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 1174, 2024-03-18T01:47:55.071 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 1174, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 1174, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 1174, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 3758096384, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:55.072 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 25, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 2725, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 25, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 4, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 2725, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:55.073 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-write_buf_throttle-0x5610378003c0": { 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 3758096384, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 1927, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 2954888, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 1927, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 232, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 2954888, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:55.074 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:55.075 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:55.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.075 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:55.075 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:55.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:55 smithi007 ceph-mon[31996]: pgmap v709: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:55.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:55 smithi046 ceph-mon[36108]: pgmap v709: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:55.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:55 smithi203 ceph-mon[36345]: pgmap v709: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:56.040 INFO:tasks.check_counter:Daemon mds.g mds.exported=2 2024-03-18T01:47:56.040 INFO:tasks.check_counter:Daemon mds.g mds.imported=36 2024-03-18T01:47:56.040 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-03-18T01:47:56.040 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T01:47:56.040 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T01:47:56.514 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:47:57.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:57 smithi007 ceph-mon[31996]: pgmap v710: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:47:57.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:57 smithi203 ceph-mon[36345]: pgmap v710: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:47:57.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:57 smithi046 ceph-mon[36108]: pgmap v710: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:47:58.896 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-03-18T01:47:58.896 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:47:58.896 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 822, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 824, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 2375276, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 261460, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 6, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 2, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.143630076, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.042345838, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.040475448, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.042043190, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 822, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.149674687, 2024-03-18T01:47:58.897 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000182085 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 968, 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 842, 2024-03-18T01:47:58.898 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 1476790, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 332005, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 4, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 3, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.208052273, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.072135618, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.082591491, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.054956873, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 841, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.177967033, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000211613 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.899 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 1325232, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 136112 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_messages": 706, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages": 870, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_bytes": 238688, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_bytes": 580470, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_created_connections": 4, 2024-03-18T01:47:58.900 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_active_connections": 3, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_total_time": 0.183936218, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_send_time": 0.092365214, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_recv_time": 0.034046134, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_running_fast_dispatch_time": 0.043967068, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 869, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.145388676, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000167305 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.901 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "msgr_send_encrypted_bytes": 357552 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "cct": { 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "total_workers": 1, 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "unhealthy_workers": 0 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "finisher-MDSRank": { 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "queue_len": 0, 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "complete_latency": { 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 3615, 2024-03-18T01:47:58.902 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.179659229, 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000049698 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "finisher-PurgeQueue": { 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "queue_len": 0, 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "complete_latency": { 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": { 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "request": 0, 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "reply": 0, 2024-03-18T01:47:58.903 INFO:teuthology.orchestra.run.smithi007.stdout: "reply_latency": { 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "slow_reply": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "forward": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_fetch_complete": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_fetch_keys": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_commit": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_split": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_merge": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes": 68, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_top": 0, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_bottom": 68, 2024-03-18T01:47:58.904 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_pin_tail": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_pinned": 68, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_expired": 5, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "inodes_with_caps": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "caps": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "subtrees": 1, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_hit": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_forward": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_discover": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_remote_ino": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "traverse_lock": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "load_cent": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "q": 0, 2024-03-18T01:47:58.905 INFO:teuthology.orchestra.run.smithi007.stdout: "exported": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "exported_inodes": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "imported": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "imported_inodes": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "openino_dir_fetch": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "openino_peer_discover": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "root_rfiles": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "root_rbytes": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "root_rsnaps": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_set_tag": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:47:58.906 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "scrub_file_inodes": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_caps": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_cap_release": 0, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "process_request_cap_release": 0 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_cache": { 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_update": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_update_receipt": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_try_discover": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_send_discover": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "dir_handle_discover": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "num_strays": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "num_strays_delayed": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "strays_created": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "strays_enqueued": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "strays_reintegrated": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "strays_migrated": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "num_recovering_processing": 0, 2024-03-18T01:47:58.908 INFO:teuthology.orchestra.run.smithi007.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_started": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "recovery_completed": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_exportdir": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_flush": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_fragstats": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ireq_inodestats": 0 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_log": { 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "evlrg": 0, 2024-03-18T01:47:58.909 INFO:teuthology.orchestra.run.smithi007.stdout: "evadd": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "evex": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "evtrm": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "ev": 1505, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "evexg": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "evexd": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "segadd": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "segex": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "segtrm": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "seg": 3, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "segmjr": 2, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "segexg": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "segexd": 0, 2024-03-18T01:47:58.910 INFO:teuthology.orchestra.run.smithi007.stdout: "expos": 4194304, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "wrpos": 6355465, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "rdpos": 6354327, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "jlat": { 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "replayed": 1505 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_mem": { 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "ino": 4, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "ino+": 148, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "ino-": 144, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "dir": 2, 2024-03-18T01:47:58.911 INFO:teuthology.orchestra.run.smithi007.stdout: "dir+": 88, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "dir-": 86, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "dn": 68, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "dn+": 255, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "dn-": 187, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "cap": 0, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "cap+": 0, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "cap-": 0, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "rss": 50888, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "heap": 215420 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_server": { 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "dispatch_client_request": 0, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "dispatch_server_request": 0, 2024-03-18T01:47:58.912 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_request": 0, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_client_session": 0, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "handle_peer_request": 0, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "req_create_latency": { 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "req_getattr_latency": { 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "req_getfilelock_latency": { 2024-03-18T01:47:58.913 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "req_link_latency": { 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookup_latency": { 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.914 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookuphash_latency": { 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookupino_latency": { 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookupname_latency": { 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.915 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookupparent_latency": { 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lookupsnap_latency": { 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "req_lssnap_latency": { 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.916 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.917 INFO:teuthology.orchestra.run.smithi007.stdout: "req_mkdir_latency": { 2024-03-18T01:47:58.917 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.917 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.917 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.917 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "req_mknod_latency": { 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "req_mksnap_latency": { 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "req_open_latency": { 2024-03-18T01:47:58.918 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "req_readdir_latency": { 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "req_rename_latency": { 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "req_renamesnap_latency": { 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "req_snapdiff_latency": { 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "req_rmdir_latency": { 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.920 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "req_rmsnap_latency": { 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "req_rmxattr_latency": { 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setattr_latency": { 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.921 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setdirlayout_latency": { 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setfilelock_latency": { 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.922 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setlayout_latency": { 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "req_setxattr_latency": { 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "req_symlink_latency": { 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.923 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "req_unlink_latency": { 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "req_getvxattr_latency": { 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.924 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_sessions": { 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "session_count": 1, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "session_add": 2, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "session_remove": 1, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "sessions_open": 1, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "sessions_stale": 0, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "total_load": 0, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "average_load": 0, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "avg_session_uptime": 2021, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "mdthresh_evicted": 0 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "mempool": { 2024-03-18T01:47:58.925 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bloom_filter_items": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:47:58.926 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_extent_items": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_blob_items": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_txc_items": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:47:58.927 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluestore_writing_items": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_bytes": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_items": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_bytes": 191445, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_anon_items": 52, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "buffer_meta_items": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_bytes": 0, 2024-03-18T01:47:58.928 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_items": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_mapbl_items": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_pglog_items": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_bytes": 8696, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_items": 166, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_bytes": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "pgmap_items": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_bytes": 59864, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "mds_co_items": 301, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_bytes": 0, 2024-03-18T01:47:58.929 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_1_items": 0, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_bytes": 0, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "unittest_2_items": 0 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "objecter": { 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "op_active": 0, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "op_laggy": 0, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "op_send": 2139, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "op_send_bytes": 21, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "op_resend": 3, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "op_reply": 2139, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "op_latency": { 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 2139, 2024-03-18T01:47:58.930 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 48.851273862, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.022838370 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "op_inflight": 0, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "oplen_avg": { 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 2139, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 2140 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "op": 2139, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "op_r": 2139, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "op_w": 0, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "op_rmw": 0, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "op_pg": 0, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_stat": 1315, 2024-03-18T01:47:58.931 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_create": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_read": 822, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_write": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_writefull": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_writesame": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_append": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_zero": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_truncate": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_delete": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_mapext": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_sparse_read": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_clonerange": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_getxattr": 1, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_setxattr": 0, 2024-03-18T01:47:58.932 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_rmxattr": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_call": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_watch": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_notify": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_pgls": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "osdop_other": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "linger_active": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "linger_send": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "linger_resend": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "linger_ping": 0, 2024-03-18T01:47:58.933 INFO:teuthology.orchestra.run.smithi007.stdout: "poolop_active": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "poolop_send": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "poolop_resend": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "poolstat_active": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "poolstat_send": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "poolstat_resend": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "statfs_active": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "statfs_send": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "statfs_resend": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "command_active": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "command_send": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "command_resend": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "map_epoch": 91, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "map_full": 0, 2024-03-18T01:47:58.934 INFO:teuthology.orchestra.run.smithi007.stdout: "map_inc": 2, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_sessions": 5, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_session_open": 5, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_session_close": 0, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "osd_laggy": 0, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_wr": 0, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_rd": 2, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_del": 0 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "oft": { 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_total_objs": 0, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:47:58.935 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_total_updates": 0, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "omap_total_removes": 0 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "osd-slow-ops": { 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "slow_ops_count": 0 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "purge_queue": { 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executing_ops": 0, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executing": 0, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executing_high_water": 0, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executed_ops": 0, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_executed": 0, 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: "pq_item_in_journal": 0 2024-03-18T01:47:58.936 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 2492, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 3845522, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 2492, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 2492, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 3845522, 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:58.937 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-objecter_bytes": { 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 104857600, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 0, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 0, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 0, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 2139, 2024-03-18T01:47:58.938 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 2161166, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 2139, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 2161166, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-objecter_ops": { 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 1024, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 0, 2024-03-18T01:47:58.939 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 0, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 0, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 2139, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 2139, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 2139, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 2139, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:58.940 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 3758096384, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.941 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "throttle-write_buf_throttle-0x56524b744780": { 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "max": 3758096384, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "get_started": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "get": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "get_sum": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_fail": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "get_or_fail_success": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "take": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "take_sum": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "put": 0, 2024-03-18T01:47:58.942 INFO:teuthology.orchestra.run.smithi007.stdout: "put_sum": 0, 2024-03-18T01:47:58.943 INFO:teuthology.orchestra.run.smithi007.stdout: "wait": { 2024-03-18T01:47:58.943 INFO:teuthology.orchestra.run.smithi007.stdout: "avgcount": 0, 2024-03-18T01:47:58.943 INFO:teuthology.orchestra.run.smithi007.stdout: "sum": 0.000000000, 2024-03-18T01:47:58.943 INFO:teuthology.orchestra.run.smithi007.stdout: "avgtime": 0.000000000 2024-03-18T01:47:58.943 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.943 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-03-18T01:47:58.943 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-03-18T01:47:59.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:47:59 smithi007 ceph-mon[31996]: pgmap v711: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:59.571 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:47:59 smithi046 ceph-mon[36108]: pgmap v711: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:59.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:47:59 smithi203 ceph-mon[36345]: pgmap v711: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:47:59.849 INFO:tasks.check_counter:Daemon mds.j mds.exported=0 2024-03-18T01:47:59.849 INFO:tasks.check_counter:Daemon mds.j mds.imported=0 2024-03-18T01:47:59.849 INFO:tasks.check_counter:Daemon mds.j mds.dir_split=0 2024-03-18T01:47:59.849 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T01:47:59.850 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T01:48:00.320 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:48:01.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:01 smithi007 ceph-mon[31996]: pgmap v712: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:48:01.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:01 smithi203 ceph-mon[36345]: pgmap v712: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:48:01.727 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:01 smithi046 ceph-mon[36108]: pgmap v712: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:48:03.148 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-18T01:48:03.148 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:48:03.148 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 4807, 2024-03-18T01:48:03.148 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 4734, 2024-03-18T01:48:03.148 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1663989, 2024-03-18T01:48:03.148 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 10065856, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 13, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 4, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.361573168, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.119869252, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.157311984, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.079196973, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 4729, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1.140815295, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000241238 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 4484, 2024-03-18T01:48:03.149 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000914854, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000204 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 11589, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 15485, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 4718814, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 12284875, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 16, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 5, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.961942238, 2024-03-18T01:48:03.150 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.379603627, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.630389402, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.090742056, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 15486, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1.491028825, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000096282 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 20249, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.003467487, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000171 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 3416064, 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 8216144 2024-03-18T01:48:03.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 5946, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 6220, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 3858101, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 7046365, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 80, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 10, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.495345296, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.190928607, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.250527066, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.072771004, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 6220, 2024-03-18T01:48:03.152 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.763797922, 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000122797 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 7563, 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.001598118, 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000211 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1367216, 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 159632 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 1, 2024-03-18T01:48:03.153 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-MDSRank": { 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 3785, 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 3.809669157, 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001006517 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-PurgeQueue": { 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2232, 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.156262628, 2024-03-18T01:48:03.154 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000070010 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": { 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "request": 2527, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "reply": 2491, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "reply_latency": { 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2491, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 32.634118583, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.013100810 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_reply": 0, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "forward": 36, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_complete": 0, 2024-03-18T01:48:03.155 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_keys": 0, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_commit": 21, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_split": 1, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_merge": 0, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes": 100, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_top": 10, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_bottom": 6, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pin_tail": 84, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pinned": 85, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_expired": 517, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_with_caps": 2, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": 2, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "subtrees": 2, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse": 3474, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_hit": 3242, 2024-03-18T01:48:03.156 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_forward": 5, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_discover": 0, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_remote_ino": 0, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_lock": 52, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "load_cent": 0, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "q": 0, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "exported": 73, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "exported_inodes": 500, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "imported": 39, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "imported_inodes": 329, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_dir_fetch": 0, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_backtrace_fetch": 567, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_peer_discover": 0, 2024-03-18T01:48:03.157 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rfiles": 4, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rbytes": 234, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rsnaps": 0, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_fetch": 567, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_set_tag": 0, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_inodes": 202, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_base_inodes": 6, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dirfrag_rstats": 196, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_file_inodes": 365, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_inode_file_caps": 71, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_revoke": 382, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_grant": 3141, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_trunc": 6, 2024-03-18T01:48:03.158 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps": 4876, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps_dirty": 4094, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_cap_release": 218, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "process_request_cap_release": 1307 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_cache": { 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update": 3, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update_receipt": 0, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_try_discover": 0, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_send_discover": 62, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_handle_discover": 112, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays": 1, 2024-03-18T01:48:03.159 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_delayed": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_created": 775, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_enqueued": 713, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_reintegrated": 61, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_migrated": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_processing": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_started": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_completed": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_enqueue_scrub": 6, 2024-03-18T01:48:03.160 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_exportdir": 73, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_flush": 0, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragmentdir": 1, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragstats": 0, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_inodestats": 0 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_log": { 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "evlrg": 0, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "evadd": 7328, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "evex": 0, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "evtrm": 0, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "ev": 7328, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "evexg": 0, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "evexd": 0, 2024-03-18T01:48:03.161 INFO:teuthology.orchestra.run.smithi046.stdout: "segadd": 9, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "segex": 0, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "segtrm": 0, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "seg": 9, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "segmjr": 2, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "segexg": 0, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "segexd": 0, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "expos": 4194304, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "wrpos": 19856036, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "rdpos": 0, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "jlat": { 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1807, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 27937.823088288, 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 15.460887154 2024-03-18T01:48:03.162 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "replayed": 0 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_mem": { 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 25, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "ino+": 1096, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "ino-": 1071, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "dir": 19, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "dir+": 181, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "dir-": 162, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "dn": 100, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "dn+": 1729, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "dn-": 1629, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "cap": 2, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "cap+": 1095, 2024-03-18T01:48:03.163 INFO:teuthology.orchestra.run.smithi046.stdout: "cap-": 1093, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "rss": 69284, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "heap": 231804 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_server": { 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_client_request": 3459, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_server_request": 198, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_request": 2527, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_session": 89, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_peer_request": 526, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "req_create_latency": { 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 757, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1.031666719, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001362835 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.164 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getattr_latency": { 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 90, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.024883836, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000276487 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getfilelock_latency": { 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "req_link_latency": { 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 61, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.065801822, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001078718 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.165 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookup_latency": { 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 329, 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.205750992, 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000625382 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookuphash_latency": { 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupino_latency": { 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1, 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000146966, 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000146966 2024-03-18T01:48:03.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupname_latency": { 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupparent_latency": { 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupsnap_latency": { 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.167 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.168 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lssnap_latency": { 2024-03-18T01:48:03.168 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.168 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.168 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mkdir_latency": { 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 105, 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.129110034, 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001229619 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mknod_latency": { 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mksnap_latency": { 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.169 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "req_open_latency": { 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 53, 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.016469310, 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000310741 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "req_readdir_latency": { 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 207, 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1.183507766, 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.005717428 2024-03-18T01:48:03.170 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rename_latency": { 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 106, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 15.304223344, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.144379465 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "req_renamesnap_latency": { 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "req_snapdiff_latency": { 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.171 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmdir_latency": { 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 91, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 13.256578825, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.145676690 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmsnap_latency": { 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmxattr_latency": { 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.172 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setattr_latency": { 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 8, 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.034490311, 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.004311288 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setdirlayout_latency": { 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setfilelock_latency": { 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 4, 2024-03-18T01:48:03.173 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.001530372, 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000382593 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setlayout_latency": { 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setxattr_latency": { 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 8, 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.020847651, 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.002605956 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "req_symlink_latency": { 2024-03-18T01:48:03.174 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 13, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.015938780, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001226060 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "req_unlink_latency": { 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 658, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1.343171855, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.002041294 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getvxattr_latency": { 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.175 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_sessions": { 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "session_count": 1, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "session_add": 3, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "session_remove": 2, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_open": 1, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_stale": 0, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "total_load": 400, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "average_load": 400, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "avg_session_uptime": 601, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mdthresh_evicted": 0 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.176 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:48:03.177 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:48:03.178 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 312284, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 118, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 176, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 2, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:48:03.179 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 8696, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 166, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 160465, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 1653, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-18T01:48:03.180 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "objecter": { 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "op_active": 0, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "op_laggy": 0, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send": 5365, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send_bytes": 16004736, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "op_resend": 37, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "op_reply": 5365, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "op_latency": { 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 5365, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 29610.927498609, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 5.519278191 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.181 INFO:teuthology.orchestra.run.smithi046.stdout: "op_inflight": 0, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "oplen_avg": { 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 5365, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 5609 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "op": 5365, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "op_r": 641, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "op_w": 4724, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "op_rmw": 0, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "op_pg": 0, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_stat": 0, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_create": 11, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_read": 68, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_write": 1869, 2024-03-18T01:48:03.182 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writefull": 73, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writesame": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_append": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_zero": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_truncate": 1, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_delete": 2590, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_mapext": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_sparse_read": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_clonerange": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_getxattr": 573, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_setxattr": 22, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_rmxattr": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:48:03.183 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_call": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_watch": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_notify": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_other": 6, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_active": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_send": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_resend": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_ping": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_active": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_send": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_resend": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_active": 0, 2024-03-18T01:48:03.184 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_send": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_resend": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_active": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_send": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_resend": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "command_active": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "command_send": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "command_resend": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "map_epoch": 91, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "map_full": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "map_inc": 7, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_sessions": 12, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_open": 12, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_close": 0, 2024-03-18T01:48:03.185 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_laggy": 0, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_wr": 293, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_rd": 0, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_del": 103 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "oft": { 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_objs": 1, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_updates": 0, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_removes": 0 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "osd-slow-ops": { 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_ops_count": 2 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.186 INFO:teuthology.orchestra.run.smithi046.stdout: "purge_queue": { 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops": 0, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops_high_water": 27, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing": 0, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_high_water": 8, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed_ops": 2075, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed": 713, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_item_in_journal": 0 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 22338, 2024-03-18T01:48:03.187 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 7819420, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 22338, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 22338, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 7819420, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_bytes": { 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:03.188 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 5365, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 16081637, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 5365, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 16081637, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.189 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_ops": { 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 1024, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 5365, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 5365, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 5365, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 5365, 2024-03-18T01:48:03.190 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 713, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 76901, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 713, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:03.191 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 62, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 76901, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle-0x55e02913a460": { 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 7328, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 15661732, 2024-03-18T01:48:03.192 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 7328, 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 1807, 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 15661732, 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:03.193 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-18T01:48:03.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:03 smithi046 ceph-mon[36108]: pgmap v713: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:48:03.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:03 smithi007 ceph-mon[31996]: pgmap v713: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:48:03.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:03 smithi203 ceph-mon[36345]: pgmap v713: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:48:04.090 INFO:tasks.check_counter:Daemon mds.b mds.exported=73 2024-03-18T01:48:04.090 INFO:tasks.check_counter:Daemon mds.b mds.imported=39 2024-03-18T01:48:04.090 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-03-18T01:48:04.090 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T01:48:04.091 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T01:48:04.433 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:48:05.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:05 smithi007 ceph-mon[31996]: pgmap v714: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:05.620 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:05 smithi046 ceph-mon[36108]: pgmap v714: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:05.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:05 smithi203 ceph-mon[36345]: pgmap v714: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:06.353 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-18T01:48:06.353 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:48:06.353 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 708, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 710, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 4406613, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 225154, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 6, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 2, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.055551283, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.018410555, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.014876649, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.017085528, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 708, 2024-03-18T01:48:06.354 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.018139966, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000025621 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 579, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 522, 2024-03-18T01:48:06.355 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 8658648, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 215169, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 4, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.059574678, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.016183270, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.037507696, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.010803470, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 522, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.034389640, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000065880 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:06.356 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1328384, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 139040 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 314, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 481, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 4272199, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 537905, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 5, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-18T01:48:06.357 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.033286380, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.011802613, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.010044935, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.008111982, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 479, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.020001398, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000041756 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.358 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 438688 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 1, 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-MDSRank": { 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2078, 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.044604398, 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000021465 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.359 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-PurgeQueue": { 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": { 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "request": 0, 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "reply": 0, 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "reply_latency": { 2024-03-18T01:48:06.360 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_reply": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "forward": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_complete": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_keys": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_commit": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_split": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_merge": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes": 828, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_top": 0, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_bottom": 828, 2024-03-18T01:48:06.361 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pin_tail": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pinned": 828, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_expired": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_with_caps": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "subtrees": 2, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_hit": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_forward": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_discover": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_remote_ino": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_lock": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "load_cent": 0, 2024-03-18T01:48:06.362 INFO:teuthology.orchestra.run.smithi046.stdout: "q": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "exported": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "exported_inodes": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "imported": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "imported_inodes": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_dir_fetch": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_peer_discover": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rfiles": 4, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rbytes": 234, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rsnaps": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_set_tag": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:48:06.363 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_file_inodes": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_cap_release": 0, 2024-03-18T01:48:06.364 INFO:teuthology.orchestra.run.smithi046.stdout: "process_request_cap_release": 0 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_cache": { 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update_receipt": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_try_discover": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_send_discover": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_handle_discover": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_delayed": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_created": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_enqueued": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_reintegrated": 0, 2024-03-18T01:48:06.365 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_migrated": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_processing": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_started": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_completed": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_exportdir": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_flush": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragstats": 0, 2024-03-18T01:48:06.366 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_inodestats": 0 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_log": { 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "evlrg": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "evadd": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "evex": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "evtrm": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "ev": 7328, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "evexg": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "evexd": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "segadd": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "segex": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "segtrm": 0, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "seg": 9, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "segmjr": 2, 2024-03-18T01:48:06.367 INFO:teuthology.orchestra.run.smithi046.stdout: "segexg": 0, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "segexd": 0, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "expos": 4194304, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "wrpos": 19856036, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "rdpos": 19854385, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "jlat": { 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "replayed": 7328 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_mem": { 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 18, 2024-03-18T01:48:06.368 INFO:teuthology.orchestra.run.smithi046.stdout: "ino+": 1097, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "ino-": 1079, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "dir": 13, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "dir+": 188, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "dir-": 175, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "dn": 828, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "dn+": 1699, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "dn-": 871, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "cap": 0, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "cap+": 0, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "cap-": 0, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "rss": 51172, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "heap": 215420 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_server": { 2024-03-18T01:48:06.369 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_client_request": 0, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_server_request": 0, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_request": 0, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_session": 0, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_peer_request": 0, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "req_create_latency": { 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getattr_latency": { 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.370 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getfilelock_latency": { 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "req_link_latency": { 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookup_latency": { 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.371 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookuphash_latency": { 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupino_latency": { 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupname_latency": { 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.372 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupparent_latency": { 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupsnap_latency": { 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lssnap_latency": { 2024-03-18T01:48:06.373 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.374 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.374 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.374 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.374 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mkdir_latency": { 2024-03-18T01:48:06.374 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mknod_latency": { 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mksnap_latency": { 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.375 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "req_open_latency": { 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "req_readdir_latency": { 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rename_latency": { 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.376 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "req_renamesnap_latency": { 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "req_snapdiff_latency": { 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmdir_latency": { 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.377 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmsnap_latency": { 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmxattr_latency": { 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setattr_latency": { 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.378 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setdirlayout_latency": { 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setfilelock_latency": { 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setlayout_latency": { 2024-03-18T01:48:06.379 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setxattr_latency": { 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "req_symlink_latency": { 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.380 INFO:teuthology.orchestra.run.smithi046.stdout: "req_unlink_latency": { 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getvxattr_latency": { 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_sessions": { 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "session_count": 1, 2024-03-18T01:48:06.381 INFO:teuthology.orchestra.run.smithi046.stdout: "session_add": 3, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "session_remove": 2, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_open": 1, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_stale": 0, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "total_load": 0, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "average_load": 0, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "avg_session_uptime": 2303, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "mdthresh_evicted": 0 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:48:06.382 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:48:06.383 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-18T01:48:06.384 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 189438, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 52, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 8696, 2024-03-18T01:48:06.385 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 166, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 639793, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 3628, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "objecter": { 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "op_active": 0, 2024-03-18T01:48:06.386 INFO:teuthology.orchestra.run.smithi046.stdout: "op_laggy": 0, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send": 1242, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send_bytes": 26, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "op_resend": 4, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "op_reply": 1242, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "op_latency": { 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1242, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 379.093706403, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.305228427 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "op_inflight": 0, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "oplen_avg": { 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1242, 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1243 2024-03-18T01:48:06.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "op": 1242, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "op_r": 1242, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "op_w": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "op_rmw": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "op_pg": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_stat": 752, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_create": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_read": 487, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_write": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writefull": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writesame": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_append": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_zero": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_truncate": 0, 2024-03-18T01:48:06.388 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_delete": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_mapext": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_sparse_read": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_clonerange": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_getxattr": 2, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_setxattr": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_rmxattr": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_call": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_watch": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_notify": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_other": 0, 2024-03-18T01:48:06.389 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_active": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_send": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_resend": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_ping": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_active": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_send": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_resend": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_active": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_send": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_resend": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_active": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_send": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_resend": 0, 2024-03-18T01:48:06.390 INFO:teuthology.orchestra.run.smithi046.stdout: "command_active": 0, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "command_send": 0, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "command_resend": 0, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "map_epoch": 91, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "map_full": 0, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "map_inc": 2, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_sessions": 5, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_open": 5, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_close": 0, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_laggy": 0, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_wr": 0, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_rd": 2, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_del": 0 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.391 INFO:teuthology.orchestra.run.smithi046.stdout: "oft": { 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_objs": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_updates": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_removes": 0 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "osd-slow-ops": { 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_ops_count": 0 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "purge_queue": { 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_high_water": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed_ops": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed": 0, 2024-03-18T01:48:06.392 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_item_in_journal": 0 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 1597, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 17172689, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 1597, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 1597, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 17172689, 2024-03-18T01:48:06.393 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_bytes": { 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:06.394 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 1242, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 15661742, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 1242, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 15661742, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_ops": { 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 1024, 2024-03-18T01:48:06.395 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 1242, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 1242, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 1242, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1242, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.396 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 0, 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:06.397 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle-0x55e635dd06e0": { 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:06.398 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 0, 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 0, 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:06.399 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-18T01:48:07.124 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T01:48:07.124 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T01:48:07.124 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-18T01:48:07.124 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T01:48:07.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T01:48:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:48:07.510 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:07 smithi046 ceph-mon[36108]: pgmap v715: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:07.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:07 smithi007 ceph-mon[31996]: pgmap v715: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:07.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:07 smithi203 ceph-mon[36345]: pgmap v715: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:09.204 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-18T01:48:09.204 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:48:09.204 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 2110, 2024-03-18T01:48:09.204 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 2259, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 2314104, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 3318038, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 10, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 4, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.196150968, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.079122082, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.105453792, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.025144029, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2263, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.269234750, 2024-03-18T01:48:09.205 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000118972 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2503, 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000697275, 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000278 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1343488, 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 144112 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 408, 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 576, 2024-03-18T01:48:09.206 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 105909, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 669965, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 20, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 7, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.047070796, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.014914974, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.015360838, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.013660161, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 572, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.071886273, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000125675 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:09.207 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 8, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000748, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000093 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 442384 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 3110, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 3190, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1493614, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 1720817, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 17, 2024-03-18T01:48:09.208 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 8, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.222154894, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.083509928, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.215750168, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.006932297, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 3186, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1.282542612, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000402555 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 4680, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.001420384, 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000303 2024-03-18T01:48:09.209 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1107792, 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 1219424 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 1, 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-MDSRank": { 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 433, 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.532466375, 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001229714 2024-03-18T01:48:09.210 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-PurgeQueue": { 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 734, 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.026154683, 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000035633 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": { 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "request": 185, 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "reply": 177, 2024-03-18T01:48:09.211 INFO:teuthology.orchestra.run.smithi046.stdout: "reply_latency": { 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 177, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.136505275, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000771216 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_reply": 0, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "forward": 8, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_complete": 0, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_keys": 0, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_commit": 11, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_split": 0, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_merge": 0, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes": 15, 2024-03-18T01:48:09.212 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_top": 0, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_bottom": 1, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pin_tail": 14, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pinned": 14, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_expired": 15, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_with_caps": 2, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": 2, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "subtrees": 3, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse": 353, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_hit": 313, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_forward": 7, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_discover": 4, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:48:09.213 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_remote_ino": 0, 2024-03-18T01:48:09.214 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_lock": 0, 2024-03-18T01:48:09.214 INFO:teuthology.orchestra.run.smithi046.stdout: "load_cent": 0, 2024-03-18T01:48:09.214 INFO:teuthology.orchestra.run.smithi046.stdout: "q": 0, 2024-03-18T01:48:09.214 INFO:teuthology.orchestra.run.smithi046.stdout: "exported": 2, 2024-03-18T01:48:09.214 INFO:teuthology.orchestra.run.smithi046.stdout: "exported_inodes": 5, 2024-03-18T01:48:09.214 INFO:teuthology.orchestra.run.smithi046.stdout: "imported": 2, 2024-03-18T01:48:09.214 INFO:teuthology.orchestra.run.smithi046.stdout: "imported_inodes": 33, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_dir_fetch": 0, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_backtrace_fetch": 28, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_peer_discover": 0, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rfiles": 82, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rbytes": 792295, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rsnaps": 0, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_fetch": 28, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_set_tag": 0, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:48:09.215 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_file_inodes": 28, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_revoke": 26, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_grant": 176, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps": 2379, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps_dirty": 2204, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_cap_release": 7, 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: "process_request_cap_release": 59 2024-03-18T01:48:09.216 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_cache": { 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update": 0, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update_receipt": 3, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_try_discover": 2, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_send_discover": 6, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_handle_discover": 0, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays": 0, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_delayed": 0, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_created": 28, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_enqueued": 28, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_reintegrated": 0, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_migrated": 0, 2024-03-18T01:48:09.217 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_processing": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_started": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_completed": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_exportdir": 2, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_flush": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragstats": 0, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_inodestats": 0 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_log": { 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "evlrg": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "evadd": 2355, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "evex": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "evtrm": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "ev": 2355, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "evexg": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "evexd": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "segadd": 4, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "segex": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "segtrm": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "seg": 4, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "segmjr": 2, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "segexg": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "segexd": 0, 2024-03-18T01:48:09.219 INFO:teuthology.orchestra.run.smithi046.stdout: "expos": 4194304, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "wrpos": 7069695, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "rdpos": 0, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "jlat": { 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 148, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 5.303977865, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.035837688 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "replayed": 0 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_mem": { 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 19, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "ino+": 53, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "ino-": 34, 2024-03-18T01:48:09.220 INFO:teuthology.orchestra.run.smithi046.stdout: "dir": 18, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "dir+": 25, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "dir-": 7, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "dn": 15, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "dn+": 86, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "dn-": 71, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "cap": 2, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "cap+": 55, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "cap-": 53, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "rss": 57632, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "heap": 223612 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.221 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_server": { 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_client_request": 214, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_server_request": 1, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_request": 185, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_session": 8, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_peer_request": 32, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "req_create_latency": { 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getattr_latency": { 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000819259, 2024-03-18T01:48:09.222 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000819259 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getfilelock_latency": { 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "req_link_latency": { 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookup_latency": { 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 135, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.084845273, 2024-03-18T01:48:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000628483 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookuphash_latency": { 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupino_latency": { 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1, 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000171795, 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000171795 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupname_latency": { 2024-03-18T01:48:09.224 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupparent_latency": { 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupsnap_latency": { 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lssnap_latency": { 2024-03-18T01:48:09.225 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.226 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.226 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.226 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.226 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mkdir_latency": { 2024-03-18T01:48:09.226 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mknod_latency": { 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mksnap_latency": { 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.227 INFO:teuthology.orchestra.run.smithi046.stdout: "req_open_latency": { 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 10, 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.006537351, 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000653735 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "req_readdir_latency": { 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2, 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.003069233, 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001534616 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rename_latency": { 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "req_renamesnap_latency": { 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "req_snapdiff_latency": { 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmdir_latency": { 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.229 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmsnap_latency": { 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmxattr_latency": { 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setattr_latency": { 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.230 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setdirlayout_latency": { 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setfilelock_latency": { 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setlayout_latency": { 2024-03-18T01:48:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setxattr_latency": { 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "req_symlink_latency": { 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.232 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "req_unlink_latency": { 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 28, 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.041062364, 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001466513 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getvxattr_latency": { 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.233 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_sessions": { 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "session_count": 1, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "session_add": 1, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "session_remove": 0, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_open": 1, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_stale": 0, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "total_load": 69, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "average_load": 69, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "avg_session_uptime": 607, 2024-03-18T01:48:09.234 INFO:teuthology.orchestra.run.smithi046.stdout: "mdthresh_evicted": 0 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:48:09.235 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:48:09.236 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 293511, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 120, 2024-03-18T01:48:09.237 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 8696, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 166, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-18T01:48:09.238 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 77293, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 329, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "objecter": { 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "op_active": 0, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "op_laggy": 0, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send": 1022, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send_bytes": 2897342, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "op_resend": 33, 2024-03-18T01:48:09.239 INFO:teuthology.orchestra.run.smithi046.stdout: "op_reply": 1022, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "op_latency": { 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1022, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1164.532793015, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 1.139464572 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "op_inflight": 0, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "oplen_avg": { 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1022, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1089 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "op": 1022, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "op_r": 29, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "op_w": 993, 2024-03-18T01:48:09.240 INFO:teuthology.orchestra.run.smithi046.stdout: "op_rmw": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "op_pg": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_stat": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_create": 10, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_read": 1, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_write": 149, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writefull": 34, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writesame": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_append": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_zero": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_truncate": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_delete": 740, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_mapext": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_sparse_read": 0, 2024-03-18T01:48:09.241 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_clonerange": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_getxattr": 28, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_setxattr": 20, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_rmxattr": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_call": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_watch": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_notify": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_other": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_active": 0, 2024-03-18T01:48:09.242 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_send": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_resend": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_ping": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_active": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_send": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_resend": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_active": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_send": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_resend": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_active": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_send": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_resend": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "command_active": 0, 2024-03-18T01:48:09.243 INFO:teuthology.orchestra.run.smithi046.stdout: "command_send": 0, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "command_resend": 0, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "map_epoch": 91, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "map_full": 0, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "map_inc": 2, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_sessions": 12, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_open": 12, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_close": 0, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_laggy": 0, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_wr": 72, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_rd": 0, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_del": 35 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "oft": { 2024-03-18T01:48:09.244 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_objs": 1, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_updates": 0, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_removes": 0 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "osd-slow-ops": { 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_ops_count": 0 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "purge_queue": { 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops": 0, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops_high_water": 26, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing": 0, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_high_water": 1, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed_ops": 728, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed": 28, 2024-03-18T01:48:09.245 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_item_in_journal": 0 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 5624, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 3265746, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 5624, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 5624, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 3265746, 2024-03-18T01:48:09.246 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_bytes": { 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 1022, 2024-03-18T01:48:09.247 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 2900394, 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 1022, 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 2900394, 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_ops": { 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:09.248 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 1024, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 1022, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 1022, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 1022, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1022, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 28, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 3052, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 28, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:09.250 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 1, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 3052, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle-0x55677330e460": { 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 2355, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 2875391, 2024-03-18T01:48:09.251 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 2355, 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 148, 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 2875391, 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:09.252 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-18T01:48:09.519 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:09 smithi046 ceph-mon[36108]: pgmap v716: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-18T01:48:09.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:09 smithi007 ceph-mon[31996]: pgmap v716: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-18T01:48:09.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:09 smithi203 ceph-mon[36345]: pgmap v716: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-18T01:48:10.120 INFO:tasks.check_counter:Daemon mds.h mds.exported=2 2024-03-18T01:48:10.120 INFO:tasks.check_counter:Daemon mds.h mds.imported=2 2024-03-18T01:48:10.121 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-18T01:48:10.121 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T01:48:10.121 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T01:48:10.419 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:48:11.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:11 smithi007 ceph-mon[31996]: pgmap v717: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T01:48:11.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:11 smithi203 ceph-mon[36345]: pgmap v717: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T01:48:11.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:11 smithi046 ceph-mon[36108]: pgmap v717: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T01:48:12.328 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 1590, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 1592, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 3345642, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 505621, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 6, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 2, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.136375867, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.038966308, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.035837032, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.050957165, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1590, 2024-03-18T01:48:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.057782557, 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000036341 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 607, 2024-03-18T01:48:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 776, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 151810, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 634258, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 6, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 4, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.055834737, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.017982888, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.012233781, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.020363732, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 775, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.036954267, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000047682 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 443072 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 358, 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 232, 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1326860, 2024-03-18T01:48:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 139646, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 3, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 2, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.035825644, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.010332942, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.025127794, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.005160434, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 231, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.021225192, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000091883 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1326592, 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 139328 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 1, 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-MDSRank": { 2024-03-18T01:48:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 3727, 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.092868899, 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000024917 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-PurgeQueue": { 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": { 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "request": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "reply": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "reply_latency": { 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_reply": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "forward": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_complete": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_keys": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_commit": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_split": 0, 2024-03-18T01:48:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_merge": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes": 29, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_top": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_bottom": 29, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pin_tail": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pinned": 29, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_expired": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_with_caps": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "subtrees": 1, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_hit": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_forward": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_discover": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:48:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_remote_ino": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_lock": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "load_cent": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "q": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "exported": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "exported_inodes": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "imported": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "imported_inodes": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_dir_fetch": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_peer_discover": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rfiles": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rbytes": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rsnaps": 0, 2024-03-18T01:48:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_set_tag": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_file_inodes": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps": 0, 2024-03-18T01:48:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_cap_release": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "process_request_cap_release": 0 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_cache": { 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update_receipt": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_try_discover": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_send_discover": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_handle_discover": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_delayed": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_created": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_enqueued": 0, 2024-03-18T01:48:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_reintegrated": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_migrated": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_processing": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_started": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_completed": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_exportdir": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_flush": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragstats": 0, 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_inodestats": 0 2024-03-18T01:48:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_log": { 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "evlrg": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "evadd": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "evex": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "evtrm": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "ev": 2355, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "evexg": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "evexd": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "segadd": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "segex": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "segtrm": 0, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "seg": 4, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "segmjr": 2, 2024-03-18T01:48:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "segexg": 0, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "segexd": 0, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "expos": 4194304, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "wrpos": 7069695, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "rdpos": 7067335, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "jlat": { 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "replayed": 2355 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_mem": { 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 4, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "ino+": 45, 2024-03-18T01:48:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "ino-": 41, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "dir": 2, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "dir+": 16, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "dir-": 14, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "dn": 29, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "dn+": 74, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "dn-": 45, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "cap": 0, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "cap+": 0, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "cap-": 0, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "rss": 48932, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "heap": 215420 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_server": { 2024-03-18T01:48:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_client_request": 0, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_server_request": 0, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_request": 0, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_session": 0, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_peer_request": 0, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "req_create_latency": { 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getattr_latency": { 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getfilelock_latency": { 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "req_link_latency": { 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookup_latency": { 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookuphash_latency": { 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupino_latency": { 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupname_latency": { 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupparent_latency": { 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupsnap_latency": { 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lssnap_latency": { 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mkdir_latency": { 2024-03-18T01:48:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mknod_latency": { 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mksnap_latency": { 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "req_open_latency": { 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "req_readdir_latency": { 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rename_latency": { 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "req_renamesnap_latency": { 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "req_snapdiff_latency": { 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmdir_latency": { 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmsnap_latency": { 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmxattr_latency": { 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setattr_latency": { 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setdirlayout_latency": { 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setfilelock_latency": { 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setlayout_latency": { 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setxattr_latency": { 2024-03-18T01:48:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "req_symlink_latency": { 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "req_unlink_latency": { 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:48:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getvxattr_latency": { 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_sessions": { 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "session_count": 1, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "session_add": 1, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "session_remove": 0, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_open": 1, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_stale": 0, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "total_load": 0, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "average_load": 0, 2024-03-18T01:48:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "avg_session_uptime": 2311, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "mdthresh_evicted": 0 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:48:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:48:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 189361, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 50, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:48:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 8696, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 166, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 32400, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 144, 2024-03-18T01:48:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "objecter": { 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "op_active": 0, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "op_laggy": 0, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send": 2192, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send_bytes": 21, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "op_resend": 3, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "op_reply": 2192, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "op_latency": { 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2192, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 16.383860208, 2024-03-18T01:48:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.007474388 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "op_inflight": 0, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "oplen_avg": { 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2192, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 2193 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "op": 2192, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "op_r": 2192, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "op_w": 0, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "op_rmw": 0, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "op_pg": 0, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_stat": 1344, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_create": 0, 2024-03-18T01:48:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_read": 846, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_write": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writefull": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writesame": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_append": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_zero": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_truncate": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_delete": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_mapext": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_sparse_read": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_clonerange": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_getxattr": 1, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_setxattr": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:48:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_rmxattr": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_call": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_watch": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_notify": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_other": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_active": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_send": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_resend": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_ping": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_active": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_send": 0, 2024-03-18T01:48:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_resend": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_active": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_send": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_resend": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_active": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_send": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_resend": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "command_active": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "command_send": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "command_resend": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "map_epoch": 91, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "map_full": 0, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "map_inc": 2, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_sessions": 5, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_open": 5, 2024-03-18T01:48:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_close": 0, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_laggy": 0, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_wr": 0, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_rd": 2, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_del": 0 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "oft": { 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_objs": 0, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_updates": 0, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_removes": 0 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "osd-slow-ops": { 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_ops_count": 0 2024-03-18T01:48:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "purge_queue": { 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops": 0, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing": 0, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_high_water": 0, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed_ops": 0, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed": 0, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_item_in_journal": 0 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 2551, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 4573467, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 2551, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 2551, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 4573467, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_bytes": { 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 2192, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 2875396, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 2192, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 2875396, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_ops": { 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 1024, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 2192, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 2192, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 2192, 2024-03-18T01:48:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 2192, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 0, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 0, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle-0x55c64ea126e0": { 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-18T01:48:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-18T01:48:12.373 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-18T01:48:13.135 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-18T01:48:13.135 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-18T01:48:13.135 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-18T01:48:13.135 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T01:48:13.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T01:48:13.584 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:48:13.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:13 smithi203 ceph-mon[36345]: pgmap v718: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T01:48:13.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:13 smithi046 ceph-mon[36108]: pgmap v718: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T01:48:13.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:13 smithi007 ceph-mon[31996]: pgmap v718: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T01:48:15.425 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:15 smithi203 ceph-mon[36345]: pgmap v719: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-18T01:48:15.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:15 smithi046 ceph-mon[36108]: pgmap v719: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-18T01:48:15.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:15 smithi007 ceph-mon[31996]: pgmap v719: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-03-18T01:48:15.907 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 1802, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2118, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 987338, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 4281168, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 10, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.144034936, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.054688555, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.074669278, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.021335243, 2024-03-18T01:48:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2116, 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.246232694, 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000116367 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2610, 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000522764, 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000200 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.909 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 4254, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 4521, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1779901, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 2518384, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 23, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.304715307, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.115401059, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.276404258, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.015534131, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4520, 2024-03-18T01:48:15.910 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.893449869, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000197665 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 6534, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001404787, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000214 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1503040, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 2089376 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 902, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 778, 2024-03-18T01:48:15.911 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1616543, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 542257, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 14, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 7, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.084854539, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.027204884, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.042076916, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.020200084, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 777, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.072990791, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000093939 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:15.912 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 116, 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000018829, 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000162 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1337440, 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 153600 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-18T01:48:15.913 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 477, 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.635033505, 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001331307 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1014, 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.034405119, 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000033930 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.914 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 232, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 232, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 232, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.229709508, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000990127 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-03-18T01:48:15.915 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 15, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 15, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 13, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 10, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 3, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 461, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 420, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 3, 2024-03-18T01:48:15.916 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 2, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 2, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 38, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 38, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 82, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 792295, 2024-03-18T01:48:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 38, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 38, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 33, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 250, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:48:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 3227, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 2975, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 8, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 80 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 3, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 2, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 4, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-03-18T01:48:15.919 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 38, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 38, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 2, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-03-18T01:48:15.920 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 3201, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 3201, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 5, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-03-18T01:48:15.921 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 5, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 8101840, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 155, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 11.099258505, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.071608119 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-03-18T01:48:15.922 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 19, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 62, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 43, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 18, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 25, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 7, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 15, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 101, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 86, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 74, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 74, 2024-03-18T01:48:15.923 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 56688, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223612 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 270, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 1, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 232, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 7, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 44, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.924 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.925 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 186, 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.144104825, 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000774757 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.926 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-03-18T01:48:15.927 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.928 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.928 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.928 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-03-18T01:48:15.928 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.928 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.928 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.928 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.929 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.003188140, 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000797035 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4, 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.004381168, 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001095292 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-03-18T01:48:15.930 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.931 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.932 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.933 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.934 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 38, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.078035375, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002053562 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.935 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 1, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 94, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 94, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2309, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:48:15.936 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:48:15.937 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:48:15.938 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 316711, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 127, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-03-18T01:48:15.939 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 8696, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 166, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 76477, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 315, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-03-18T01:48:15.940 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 1320, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 3933364, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 1320, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1320, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 5404.701456908, 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 4.094470800 2024-03-18T01:48:15.941 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1320, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1409 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 1320, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 39, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 1281, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-03-18T01:48:15.942 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 1, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 156, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 34, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 1000, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 38, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:48:15.943 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-03-18T01:48:15.944 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 91, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 2, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 12, 2024-03-18T01:48:15.945 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 12, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 103, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 47 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.946 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 2 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 26, 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 1, 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 988, 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 38, 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:48:15.947 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 6954, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 3573439, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 6954, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 6954, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3573439, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.948 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1320, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 3937506, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1320, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3937506, 2024-03-18T01:48:15.949 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:15.950 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1320, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 1320, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1320, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1320, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-18T01:48:15.951 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 38, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 4142, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 38, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 4142, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.952 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x5602a64f83c0": { 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 3201, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 3907536, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 3201, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 155, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3907536, 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:15.953 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:15.954 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:15.954 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:15.954 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.954 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:15.954 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-18T01:48:16.929 INFO:tasks.check_counter:Daemon mds.c mds.exported=2 2024-03-18T01:48:16.929 INFO:tasks.check_counter:Daemon mds.c mds.imported=2 2024-03-18T01:48:16.929 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-18T01:48:16.929 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T01:48:16.929 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T01:48:17.216 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:48:17.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:17 smithi203 ceph-mon[36345]: pgmap v720: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:17.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:17 smithi046 ceph-mon[36108]: pgmap v720: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:17.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:17 smithi007 ceph-mon[31996]: pgmap v720: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:19.303 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-18T01:48:19.303 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 2112, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2114, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 3553503, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 671623, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.165057438, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.043565759, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.041333113, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.070140168, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2112, 2024-03-18T01:48:19.304 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.059791454, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000028310 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 357, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 231, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1326604, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 138746, 2024-03-18T01:48:19.305 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 3, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.033168367, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.010026267, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.022465297, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.004811359, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 230, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.012915205, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000056153 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.306 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1326048, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 138048 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 6, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 173, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 3420, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 444478, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.008699449, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.005988662, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.001589222, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000022121, 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 172, 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.014146339, 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000082246 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 444160 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-03-18T01:48:19.308 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3677, 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.084971131, 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000023108 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-18T01:48:19.309 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.310 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 27, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 27, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 27, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-03-18T01:48:19.311 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 1, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:48:19.312 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:48:19.313 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-03-18T01:48:19.314 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-03-18T01:48:19.315 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 1927, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-03-18T01:48:19.316 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 3, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 7149192, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 7148054, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 1927 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 5, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 256, 2024-03-18T01:48:19.317 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 251, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 3, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 84, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 81, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 27, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 379, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 352, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 48728, 2024-03-18T01:48:19.318 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215420 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.319 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.320 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.321 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.322 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-03-18T01:48:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-03-18T01:48:19.325 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.326 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.326 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.326 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-03-18T01:48:19.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.326 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.326 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.327 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-03-18T01:48:19.328 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.329 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.330 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-03-18T01:48:19.331 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:48:19.332 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 1, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-03-18T01:48:19.333 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2311, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:48:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:48:19.335 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 181281, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 46, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-03-18T01:48:19.336 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 8696, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 166, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 34488, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 148, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-03-18T01:48:19.337 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 2115, 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 3, 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 2115, 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-03-18T01:48:19.338 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2115, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 53.387640536, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.025242383 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2115, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 2116 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 2115, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 2115, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-03-18T01:48:19.339 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 1254, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 859, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-03-18T01:48:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-03-18T01:48:19.341 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 91, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 2, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 4, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 4, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-03-18T01:48:19.342 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-03-18T01:48:19.343 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 2471, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 4640029, 2024-03-18T01:48:19.344 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 2471, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2471, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 4640029, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:19.345 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 2115, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 2954893, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2115, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2954893, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.346 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 2115, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 2115, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 2115, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2115, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.347 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.348 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55feae4aa6e0": { 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:19.349 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:19.350 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-18T01:48:19.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:19 smithi203 ceph-mon[36345]: pgmap v721: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:19.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:19 smithi046 ceph-mon[36108]: pgmap v721: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:19.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:19 smithi007 ceph-mon[31996]: pgmap v721: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:20.170 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-18T01:48:20.171 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-18T01:48:20.171 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-18T01:48:20.171 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T01:48:20.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T01:48:20.429 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:48:21.605 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:21 smithi203 ceph-mon[36345]: pgmap v722: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:48:21.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:21 smithi046 ceph-mon[36108]: pgmap v722: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:48:21.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:21 smithi007 ceph-mon[31996]: pgmap v722: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 2454, 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2616, 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 2086203, 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 3157184, 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 23, 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 8, 2024-03-18T01:48:22.228 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.207196779, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.076620011, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.149793528, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.022268819, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2610, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.357692983, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000137047 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2753, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000700708, 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000254 2024-03-18T01:48:22.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1955504, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 952080 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 384, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 551, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 249853, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 666135, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 10, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 7, 2024-03-18T01:48:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.039439874, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.014534949, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.013294778, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.010307519, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 549, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.083478914, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000152056 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 111, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000019314, 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000174 2024-03-18T01:48:22.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 369504 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 3074, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 3019, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1601869, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1627395, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 14, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.173479803, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.064631501, 2024-03-18T01:48:22.232 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.127638347, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.009923725, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 3023, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.217852184, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000072064 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5138, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000924163, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000179 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:48:22.233 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 848, 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.332360029, 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000391933 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.234 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 531, 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.022021601, 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000041471 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 194, 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 184, 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 184, 2024-03-18T01:48:22.235 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.449526011, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002443076 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 10, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 11, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 15, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 15, 2024-03-18T01:48:22.236 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 13, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 77, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 3, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 466, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 300, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 8, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 35, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-03-18T01:48:22.237 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 33, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 33, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 95, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 121, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 82, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 792295, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 121, 2024-03-18T01:48:22.238 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 16, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 16, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 105, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 87, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 200, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 1203, 2024-03-18T01:48:22.239 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 1048, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 31, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 130 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 3, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 2, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 36, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 62, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:48:22.240 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 67, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 67, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 33, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:48:22.241 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 1505, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 1505, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 3, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-03-18T01:48:22.242 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 3, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 6355465, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 0, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 311, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 4.575020400, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.014710676 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 0 2024-03-18T01:48:22.243 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 19, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 125, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 106, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 18, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 66, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 48, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 15, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 254, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 239, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-03-18T01:48:22.244 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 157, 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 157, 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 58084, 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 223612 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 337, 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 32, 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 194, 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 12, 2024-03-18T01:48:22.245 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 156, 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000161286, 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000161286 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.246 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 82, 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.102355271, 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001248235 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.247 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.248 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.249 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.250 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.250 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-03-18T01:48:22.250 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.250 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.250 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-03-18T01:48:22.251 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1, 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002608740, 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002608740 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 33, 2024-03-18T01:48:22.252 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.018796020, 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000569576 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-03-18T01:48:22.253 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 8, 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.057237614, 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.007154701 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.254 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-03-18T01:48:22.255 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.256 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 59, 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.268367080, 2024-03-18T01:48:22.257 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.004548594 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-03-18T01:48:22.258 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 125, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 125, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2314, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:48:22.259 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-03-18T01:48:22.260 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-03-18T01:48:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 293801, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 111, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-03-18T01:48:22.262 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 8696, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 166, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-03-18T01:48:22.263 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 76509, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 319, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 1043, 2024-03-18T01:48:22.264 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 2198488, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 33, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 1043, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1043, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 602.197653733, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.577370713 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1043, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1120 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.265 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 1043, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 142, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 901, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 0, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 10, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 21, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 332, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 47, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-03-18T01:48:22.266 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 450, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 121, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 20, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-03-18T01:48:22.267 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-03-18T01:48:22.268 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 91, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 2, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 12, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 12, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 87, 2024-03-18T01:48:22.269 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 0, 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 32 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 1, 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.270 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 26, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 6, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 438, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 67, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:22.271 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 5908, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 3317382, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 5908, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 5908, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3317382, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.272 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:22.273 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1043, 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 2205727, 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1043, 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2205727, 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:22.274 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1043, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 1043, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1043, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1043, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.275 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 67, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 7239, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 67, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 21, 2024-03-18T01:48:22.276 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 7239, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x5556f5ada460": { 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1505, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 2161161, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:22.277 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1505, 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 311, 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 2161161, 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:22.278 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-18T01:48:22.951 INFO:tasks.check_counter:Daemon mds.i mds.exported=33 2024-03-18T01:48:22.951 INFO:tasks.check_counter:Daemon mds.i mds.imported=33 2024-03-18T01:48:22.951 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-18T01:48:22.951 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T01:48:22.951 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T01:48:23.202 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:48:23.633 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:23 smithi203 ceph-mon[36345]: pgmap v723: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:48:23.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:23 smithi046 ceph-mon[36108]: pgmap v723: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:48:23.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:23 smithi007 ceph-mon[31996]: pgmap v723: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 402, 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 473, 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 136665, 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 132480, 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-03-18T01:48:25.049 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.032705559, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.010566677, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.008794724, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.009583887, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 472, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.019481236, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000041273 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.050 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 523, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 697, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 4042524, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 612261, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.040934971, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.015122941, 2024-03-18T01:48:25.051 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.012271207, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.009901495, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 695, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.025619295, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000036862 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 446208 2024-03-18T01:48:25.052 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 364, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 238, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1329028, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 145157, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.037066782, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.010926762, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.024806020, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.005522048, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 237, 2024-03-18T01:48:25.053 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.012918130, 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000054506 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 1327680, 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 144192 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-03-18T01:48:25.054 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1726, 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.034830847, 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000020180 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.055 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-03-18T01:48:25.056 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 39, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 39, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 39, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 1, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-03-18T01:48:25.057 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:48:25.058 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:48:25.059 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-03-18T01:48:25.061 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-03-18T01:48:25.062 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-03-18T01:48:25.063 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 3201, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 5, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 8101840, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 8099548, 2024-03-18T01:48:25.064 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 3201 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 4, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 52, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 48, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 2, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 14, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 12, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 39, 2024-03-18T01:48:25.065 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 87, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 48, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 49348, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215420 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-03-18T01:48:25.066 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.067 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.068 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.069 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-03-18T01:48:25.070 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.071 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-03-18T01:48:25.072 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-03-18T01:48:25.073 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.074 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.075 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.076 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 1, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-03-18T01:48:25.077 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2318, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:48:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:48:25.079 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 197481, 2024-03-18T01:48:25.080 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 55, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 8696, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 166, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:48:25.081 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 39440, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 184, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 923, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 3, 2024-03-18T01:48:25.082 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 923, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 923, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 378.055368026, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.409594114 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 923, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 924 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 923, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 923, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-03-18T01:48:25.083 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 509, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 412, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-03-18T01:48:25.084 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-03-18T01:48:25.085 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 91, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 2, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 6, 2024-03-18T01:48:25.086 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 6, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.087 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1285, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 5371147, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:25.088 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1285, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1285, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 5371147, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:25.089 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 923, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 3907541, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 923, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 3907541, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-03-18T01:48:25.090 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 923, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 923, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 923, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 923, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.091 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.092 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x555d1427c6e0": { 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-03-18T01:48:25.093 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-03-18T01:48:25.094 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-03-18T01:48:25.094 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-03-18T01:48:25.094 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-03-18T01:48:25.094 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.094 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-03-18T01:48:25.094 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-03-18T01:48:25.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:25 smithi203 ceph-mon[36345]: pgmap v724: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:25.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:25 smithi046 ceph-mon[36108]: pgmap v724: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:25.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:25 smithi007 ceph-mon[31996]: pgmap v724: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:25.901 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-18T01:48:25.901 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-18T01:48:25.902 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-18T01:48:25.902 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-18T01:48:25.912 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T01:48:25.912 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:48:25.912 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:48:25.973 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-18T01:48:25.973 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:48:26.026 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:48:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.331+0000 7fc873200640 1 -- 172.21.15.7:0/3671861442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 msgr2=0x7fc874109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.331+0000 7fc873200640 1 --2- 172.21.15.7:0/3671861442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 0x7fc874109720 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009ca0 tx=0x7fc86c02c100 comp rx=0 tx=0).stop 2024-03-18T01:48:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.331+0000 7fc873200640 1 -- 172.21.15.7:0/3671861442 shutdown_connections 2024-03-18T01:48:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.332+0000 7fc873200640 1 --2- 172.21.15.7:0/3671861442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 0x7fc874109720 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.332 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.332+0000 7fc873200640 1 --2- 172.21.15.7:0/3671861442 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc874104a00 0x7fc874106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.332+0000 7fc873200640 1 --2- 172.21.15.7:0/3671861442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8741020d0 0x7fc8741044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.332+0000 7fc873200640 1 -- 172.21.15.7:0/3671861442 >> 172.21.15.7:0/3671861442 conn(0x7fc8740fbaa0 msgr2=0x7fc8740fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:26.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.332+0000 7fc873200640 1 -- 172.21.15.7:0/3671861442 shutdown_connections 2024-03-18T01:48:26.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.332+0000 7fc873200640 1 -- 172.21.15.7:0/3671861442 wait complete. 2024-03-18T01:48:26.333 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.333+0000 7fc873200640 1 Processor -- start 2024-03-18T01:48:26.334 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.333+0000 7fc873200640 1 -- start start 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc873200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8741020d0 0x7fc8741977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc873200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc874104a00 0x7fc874197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc873200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 0x7fc87419ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc873200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fc87410d280 con 0x7fc8741020d0 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc873200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc87410d100 con 0x7fc874107330 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc873200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc87410d400 con 0x7fc874104a00 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc871e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8741020d0 0x7fc8741977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc871e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8741020d0 0x7fc8741977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54532/0 (socket says 172.21.15.7:54532) 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc871e00640 1 -- 172.21.15.7:0/1296756889 learned_addr learned my addr 172.21.15.7:0/1296756889 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.334+0000 7fc872800640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 0x7fc87419ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:26.335 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.335+0000 7fc871400640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc874104a00 0x7fc874197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:26.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.335+0000 7fc872800640 1 -- 172.21.15.7:0/1296756889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc874104a00 msgr2=0x7fc874197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:26.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.335+0000 7fc872800640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc874104a00 0x7fc874197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.335+0000 7fc872800640 1 -- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8741020d0 msgr2=0x7fc8741977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:26.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.335+0000 7fc872800640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8741020d0 0x7fc8741977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.336 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.335+0000 7fc872800640 1 -- 172.21.15.7:0/1296756889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8740ffec0 con 0x7fc874107330 2024-03-18T01:48:26.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.335+0000 7fc871e00640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8741020d0 0x7fc8741977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:48:26.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.335+0000 7fc872800640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 0x7fc87419ed60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc86c037040 tx=0x7fc86c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:26.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.337+0000 7fc86aa00640 1 -- 172.21.15.7:0/1296756889 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c039040 con 0x7fc874107330 2024-03-18T01:48:26.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.337+0000 7fc86aa00640 1 -- 172.21.15.7:0/1296756889 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc86c044070 con 0x7fc874107330 2024-03-18T01:48:26.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.337+0000 7fc86aa00640 1 -- 172.21.15.7:0/1296756889 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c004350 con 0x7fc874107330 2024-03-18T01:48:26.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.337+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc874100150 con 0x7fc874107330 2024-03-18T01:48:26.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.338+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8741005c0 con 0x7fc874107330 2024-03-18T01:48:26.341 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.340+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc874101f00 con 0x7fc874107330 2024-03-18T01:48:26.343 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.342+0000 7fc86aa00640 1 -- 172.21.15.7:0/1296756889 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fc86c0044f0 con 0x7fc874107330 2024-03-18T01:48:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.344+0000 7fc86aa00640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc8580a2920 0x7fc8580a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.344+0000 7fc86aa00640 1 -- 172.21.15.7:0/1296756889 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc86c0f1020 con 0x7fc874107330 2024-03-18T01:48:26.344 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.344+0000 7fc871e00640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc8580a2920 0x7fc8580a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:26.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.345+0000 7fc871e00640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc8580a2920 0x7fc8580a4de0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc864011500 tx=0x7fc864005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:26.348 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.348+0000 7fc86aa00640 1 -- 172.21.15.7:0/1296756889 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc86c04e020 con 0x7fc874107330 2024-03-18T01:48:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:48:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:26 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.574+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8740fd260 con 0x7fc874107330 2024-03-18T01:48:26.576 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.576+0000 7fc86aa00640 1 -- 172.21.15.7:0/1296756889 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7fc86c0bdd60 con 0x7fc874107330 2024-03-18T01:48:26.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:48:26.577 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:48:26.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc8580a2920 msgr2=0x7fc8580a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:26.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc8580a2920 0x7fc8580a4de0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc864011500 tx=0x7fc864005f90 comp rx=0 tx=0).stop 2024-03-18T01:48:26.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 msgr2=0x7fc87419ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:26.579 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 0x7fc87419ed60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc86c037040 tx=0x7fc86c035d10 comp rx=0 tx=0).stop 2024-03-18T01:48:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 shutdown_connections 2024-03-18T01:48:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc874107330 0x7fc87419ed60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc874104a00 0x7fc874197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fc8580a2920 0x7fc8580a4de0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 --2- 172.21.15.7:0/1296756889 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fc8741020d0 0x7fc8741977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 >> 172.21.15.7:0/1296756889 conn(0x7fc8740fbaa0 msgr2=0x7fc874102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 shutdown_connections 2024-03-18T01:48:26.580 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.579+0000 7fc873200640 1 -- 172.21.15.7:0/1296756889 wait complete. 2024-03-18T01:48:26.599 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T01:29:41.723841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T01:34:54.204638+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T01:34:55.616951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T01:34:59.640737+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T01:48:26.600 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:48:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:48:26.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:26 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:48:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:26 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:26.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 -- 172.21.15.7:0/731302812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 msgr2=0x7f3e9c109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:26.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 --2- 172.21.15.7:0/731302812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 0x7f3e9c109c70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3e94009ca0 tx=0x7f3e9402c100 comp rx=0 tx=0).stop 2024-03-18T01:48:26.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 -- 172.21.15.7:0/731302812 shutdown_connections 2024-03-18T01:48:26.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 --2- 172.21.15.7:0/731302812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 0x7f3e9c109c70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 --2- 172.21.15.7:0/731302812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e9c102a20 0x7f3e9c102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 --2- 172.21.15.7:0/731302812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e9c101830 0x7f3e9c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 -- 172.21.15.7:0/731302812 >> 172.21.15.7:0/731302812 conn(0x7f3e9c0fcda0 msgr2=0x7f3e9c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:26.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 -- 172.21.15.7:0/731302812 shutdown_connections 2024-03-18T01:48:26.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.802+0000 7f3ea2400640 1 -- 172.21.15.7:0/731302812 wait complete. 2024-03-18T01:48:26.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.803+0000 7f3ea2400640 1 Processor -- start 2024-03-18T01:48:26.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.803+0000 7f3ea2400640 1 -- start start 2024-03-18T01:48:26.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.803+0000 7f3ea2400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e9c101830 0x7f3e9c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.803+0000 7f3ea2400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e9c102a20 0x7f3e9c198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.803+0000 7f3ea2400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 0x7f3e9c19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.803+0000 7f3ea2400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c10d4b0 con 0x7f3e9c1033e0 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.803+0000 7f3ea2400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c10d330 con 0x7f3e9c102a20 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.803+0000 7f3ea2400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c10d630 con 0x7f3e9c101830 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e9c101830 0x7f3e9c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e9c102a20 0x7f3e9c198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9b400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e9c101830 0x7f3e9c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:50858/0 (socket says 172.21.15.7:50858) 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e9c102a20 0x7f3e9c198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46150/0 (socket says 172.21.15.7:46150) 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9b400640 1 -- 172.21.15.7:0/3423425146 learned_addr learned my addr 172.21.15.7:0/3423425146 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9be00640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 0x7f3e9c19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9aa00640 1 -- 172.21.15.7:0/3423425146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e9c101830 msgr2=0x7f3e9c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:26.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9aa00640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e9c101830 0x7f3e9c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9aa00640 1 -- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 msgr2=0x7f3e9c19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9aa00640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 0x7f3e9c19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9aa00640 1 -- 172.21.15.7:0/3423425146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c19f6f0 con 0x7f3e9c102a20 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.804+0000 7f3e9be00640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 0x7f3e9c19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.805+0000 7f3e9aa00640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e9c102a20 0x7f3e9c198010 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800e9f0 tx=0x7f3e8800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.805+0000 7f3e93e00640 1 -- 172.21.15.7:0/3423425146 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8800ccc0 con 0x7f3e9c102a20 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.805+0000 7f3e93e00640 1 -- 172.21.15.7:0/3423425146 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e88005020 con 0x7f3e9c102a20 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.805+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9c19f9e0 con 0x7f3e9c102a20 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.805+0000 7f3e93e00640 1 -- 172.21.15.7:0/3423425146 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e880049e0 con 0x7f3e9c102a20 2024-03-18T01:48:26.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.805+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9c19ff30 con 0x7f3e9c102a20 2024-03-18T01:48:26.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.807+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e9c108af0 con 0x7f3e9c102a20 2024-03-18T01:48:26.809 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.809+0000 7f3e93e00640 1 -- 172.21.15.7:0/3423425146 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f3e88005560 con 0x7f3e9c102a20 2024-03-18T01:48:26.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.811+0000 7f3e93e00640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e800a2880 0x7f3e800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:26.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.811+0000 7f3e93e00640 1 -- 172.21.15.7:0/3423425146 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e880ced80 con 0x7f3e9c102a20 2024-03-18T01:48:26.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.812+0000 7f3e9b400640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e800a2880 0x7f3e800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:26.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.812+0000 7f3e9b400640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e800a2880 0x7f3e800a4d40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0058c0 tx=0x7f3e8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:26.814 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:26.814+0000 7f3e93e00640 1 -- 172.21.15.7:0/3423425146 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e880051d0 con 0x7f3e9c102a20 2024-03-18T01:48:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.017+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e9c02d1b0 con 0x7f3e9c102a20 2024-03-18T01:48:27.018 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.018+0000 7f3e93e00640 1 -- 172.21.15.7:0/3423425146 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f3e8809ba60 con 0x7f3e9c102a20 2024-03-18T01:48:27.018 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-03-18T01:48:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e800a2880 msgr2=0x7f3e800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e800a2880 0x7f3e800a4d40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0058c0 tx=0x7f3e8c005f90 comp rx=0 tx=0).stop 2024-03-18T01:48:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e9c102a20 msgr2=0x7f3e9c198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e9c102a20 0x7f3e9c198010 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f3e8800e9f0 tx=0x7f3e8800eeb0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 shutdown_connections 2024-03-18T01:48:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f3e800a2880 0x7f3e800a4d40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3e9c1033e0 0x7f3e9c19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.022 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e9c102a20 0x7f3e9c198010 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 --2- 172.21.15.7:0/3423425146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e9c101830 0x7f3e9c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.021+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 >> 172.21.15.7:0/3423425146 conn(0x7f3e9c0fcda0 msgr2=0x7f3e9c105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.022+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 shutdown_connections 2024-03-18T01:48:27.023 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.022+0000 7f3ea2400640 1 -- 172.21.15.7:0/3423425146 wait complete. 2024-03-18T01:48:27.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 -- 172.21.15.7:0/4196122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a24104bf0 msgr2=0x7f5a24105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 --2- 172.21.15.7:0/4196122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a24104bf0 0x7f5a24105070 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009900 tx=0x7f5a1002bc40 comp rx=0 tx=0).stop 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 -- 172.21.15.7:0/4196122173 shutdown_connections 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 --2- 172.21.15.7:0/4196122173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a241055b0 0x7f5a2410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 --2- 172.21.15.7:0/4196122173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a24104bf0 0x7f5a24105070 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 --2- 172.21.15.7:0/4196122173 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24103a00 0x7f5a24103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 -- 172.21.15.7:0/4196122173 >> 172.21.15.7:0/4196122173 conn(0x7f5a240fef90 msgr2=0x7f5a241013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 -- 172.21.15.7:0/4196122173 shutdown_connections 2024-03-18T01:48:27.255 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.254+0000 7f5a29e00640 1 -- 172.21.15.7:0/4196122173 wait complete. 2024-03-18T01:48:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.255+0000 7f5a29e00640 1 Processor -- start 2024-03-18T01:48:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.255+0000 7f5a29e00640 1 -- start start 2024-03-18T01:48:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a29e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a24103a00 0x7f5a24199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a29e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24104bf0 0x7f5a2419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a29e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a241055b0 0x7f5a241a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a29e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2410f6b0 con 0x7f5a24104bf0 2024-03-18T01:48:27.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a29e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2410f530 con 0x7f5a241055b0 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a29e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2410f830 con 0x7f5a24103a00 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a22a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a24103a00 0x7f5a24199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a22a00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a24103a00 0x7f5a24199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:50874/0 (socket says 172.21.15.7:50874) 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a22a00640 1 -- 172.21.15.7:0/2502480142 learned_addr learned my addr 172.21.15.7:0/2502480142 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a23400640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a241055b0 0x7f5a241a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a22000640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24104bf0 0x7f5a2419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a22a00640 1 -- 172.21.15.7:0/2502480142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a241055b0 msgr2=0x7f5a241a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a22a00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a241055b0 0x7f5a241a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a22a00640 1 -- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24104bf0 msgr2=0x7f5a2419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.256+0000 7f5a22a00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24104bf0 0x7f5a2419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.257+0000 7f5a22a00640 1 -- 172.21.15.7:0/2502480142 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a241a1950 con 0x7f5a24103a00 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.257+0000 7f5a23400640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a241055b0 0x7f5a241a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:48:27.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.257+0000 7f5a22a00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a24103a00 0x7f5a24199d40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400e9c0 tx=0x7f5a1400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.257+0000 7f5a28a00640 1 -- 172.21.15.7:0/2502480142 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1400cde0 con 0x7f5a24103a00 2024-03-18T01:48:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.257+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a241a1c40 con 0x7f5a24103a00 2024-03-18T01:48:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.257+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a241a2190 con 0x7f5a24103a00 2024-03-18T01:48:27.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.258+0000 7f5a28a00640 1 -- 172.21.15.7:0/2502480142 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a14010910 con 0x7f5a24103a00 2024-03-18T01:48:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.258+0000 7f5a28a00640 1 -- 172.21.15.7:0/2502480142 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a14018a00 con 0x7f5a24103a00 2024-03-18T01:48:27.260 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.259+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a040052c0 con 0x7f5a24103a00 2024-03-18T01:48:27.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.261+0000 7f5a28a00640 1 -- 172.21.15.7:0/2502480142 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f5a1401f020 con 0x7f5a24103a00 2024-03-18T01:48:27.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.263+0000 7f5a28a00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5a080a2900 0x7f5a080a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:27.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.263+0000 7f5a28a00640 1 -- 172.21.15.7:0/2502480142 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a140cefc0 con 0x7f5a24103a00 2024-03-18T01:48:27.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.263+0000 7f5a22000640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5a080a2900 0x7f5a080a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:27.264 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.264+0000 7f5a22000640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5a080a2900 0x7f5a080a4dc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5a1002c250 tx=0x7f5a1002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:27.266 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.266+0000 7f5a28a00640 1 -- 172.21.15.7:0/2502480142 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a1409bca0 con 0x7f5a24103a00 2024-03-18T01:48:27.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.508+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5a04005dc0 con 0x7f5a24103a00 2024-03-18T01:48:27.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.509+0000 7f5a28a00640 1 -- 172.21.15.7:0/2502480142 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f5a1409b5a0 con 0x7f5a24103a00 2024-03-18T01:48:27.509 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5a080a2900 msgr2=0x7f5a080a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5a080a2900 0x7f5a080a4dc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5a1002c250 tx=0x7f5a1002cf90 comp rx=0 tx=0).stop 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a24103a00 msgr2=0x7f5a24199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a24103a00 0x7f5a24199d40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5a1400e9c0 tx=0x7f5a1400ee80 comp rx=0 tx=0).stop 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 shutdown_connections 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a241055b0 0x7f5a241a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f5a080a2900 0x7f5a080a4dc0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f5a24104bf0 0x7f5a2419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 --2- 172.21.15.7:0/2502480142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a24103a00 0x7f5a24199d40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 >> 172.21.15.7:0/2502480142 conn(0x7f5a240fef90 msgr2=0x7f5a24107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 shutdown_connections 2024-03-18T01:48:27.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.511+0000 7f5a29e00640 1 -- 172.21.15.7:0/2502480142 wait complete. 2024-03-18T01:48:27.533 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:48:27.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:27 smithi007 ceph-mon[31996]: pgmap v725: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:27.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:27.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:27 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:27.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:27 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1296756889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:48:27.602 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:27 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3423425146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:48:27.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:27 smithi203 ceph-mon[36345]: pgmap v725: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:27.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:27.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:27 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:27.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:27 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1296756889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:48:27.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:27 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3423425146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:48:27.743 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 -- 172.21.15.7:0/1100046245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 msgr2=0x7f77a8103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 --2- 172.21.15.7:0/1100046245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 0x7f77a8103e40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7798009900 tx=0x7f779802bc40 comp rx=0 tx=0).stop 2024-03-18T01:48:27.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 -- 172.21.15.7:0/1100046245 shutdown_connections 2024-03-18T01:48:27.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 --2- 172.21.15.7:0/1100046245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77a81055d0 0x7f77a810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 --2- 172.21.15.7:0/1100046245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a8104c10 0x7f77a8105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 --2- 172.21.15.7:0/1100046245 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 0x7f77a8103e40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 -- 172.21.15.7:0/1100046245 >> 172.21.15.7:0/1100046245 conn(0x7f77a80fef90 msgr2=0x7f77a81013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:27.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 -- 172.21.15.7:0/1100046245 shutdown_connections 2024-03-18T01:48:27.744 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.743+0000 7f77ae400640 1 -- 172.21.15.7:0/1100046245 wait complete. 2024-03-18T01:48:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.744+0000 7f77ae400640 1 Processor -- start 2024-03-18T01:48:27.746 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.744+0000 7f77ae400640 1 -- start start 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.744+0000 7f77ae400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 0x7f77a8199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.744+0000 7f77ae400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77a8104c10 0x7f77a819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.744+0000 7f77ae400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a81055d0 0x7f77a81a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.744+0000 7f77ae400640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f77a810f690 con 0x7f77a8103a20 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.744+0000 7f77ae400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77a810f510 con 0x7f77a8104c10 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.744+0000 7f77ae400640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77a810f810 con 0x7f77a81055d0 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 0x7f77a8199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 0x7f77a8199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54574/0 (socket says 172.21.15.7:54574) 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7400640 1 -- 172.21.15.7:0/1107111464 learned_addr learned my addr 172.21.15.7:0/1107111464 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a6a00640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77a8104c10 0x7f77a819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7e00640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a81055d0 0x7f77a81a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7400640 1 -- 172.21.15.7:0/1107111464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a81055d0 msgr2=0x7f77a81a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a81055d0 0x7f77a81a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.747 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7400640 1 -- 172.21.15.7:0/1107111464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77a8104c10 msgr2=0x7f77a819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77a8104c10 0x7f77a819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a7400640 1 -- 172.21.15.7:0/1107111464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a81a19d0 con 0x7f77a8103a20 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.745+0000 7f77a6a00640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77a8104c10 0x7f77a819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.746+0000 7f77a7400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 0x7f77a8199db0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7798000c00 tx=0x7f779802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.746+0000 7f779fe00640 1 -- 172.21.15.7:0/1107111464 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7798041070 con 0x7f77a8103a20 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.746+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a81a1c60 con 0x7f77a8103a20 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.746+0000 7f779fe00640 1 -- 172.21.15.7:0/1107111464 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7798004450 con 0x7f77a8103a20 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.746+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a81a2150 con 0x7f77a8103a20 2024-03-18T01:48:27.748 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.746+0000 7f779fe00640 1 -- 172.21.15.7:0/1107111464 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779804b060 con 0x7f77a8103a20 2024-03-18T01:48:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.747+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77a810ae60 con 0x7f77a8103a20 2024-03-18T01:48:27.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.749+0000 7f779fe00640 1 -- 172.21.15.7:0/1107111464 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f7798044070 con 0x7f77a8103a20 2024-03-18T01:48:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.751+0000 7f779fe00640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f778c0a2800 0x7f778c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.751+0000 7f779fe00640 1 -- 172.21.15.7:0/1107111464 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f77980f0c70 con 0x7f77a8103a20 2024-03-18T01:48:27.751 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.751+0000 7f77a6a00640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f778c0a2800 0x7f778c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:27.752 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.752+0000 7f77a6a00640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f778c0a2800 0x7f778c0a4cc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f77a819b2c0 tx=0x7f7794005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:27.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.760+0000 7f779fe00640 1 -- 172.21.15.7:0/1107111464 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f77980bd9e0 con 0x7f77a8103a20 2024-03-18T01:48:27.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:27 smithi046 ceph-mon[36108]: pgmap v725: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-03-18T01:48:27.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:27.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:27 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:27.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:27 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1296756889' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:48:27.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:27 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3423425146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:48:27.994 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.994+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f77a8062410 con 0x7f77a8103a20 2024-03-18T01:48:27.995 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.995+0000 7f779fe00640 1 -- 172.21.15.7:0/1107111464 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f77980bd2e0 con 0x7f77a8103a20 2024-03-18T01:48:27.996 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 64 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f778c0a2800 msgr2=0x7f778c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f778c0a2800 0x7f778c0a4cc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f77a819b2c0 tx=0x7f7794005f90 comp rx=0 tx=0).stop 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 msgr2=0x7f77a8199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 0x7f77a8199db0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7798000c00 tx=0x7f779802c650 comp rx=0 tx=0).stop 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 shutdown_connections 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a81055d0 0x7f77a81a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77a8104c10 0x7f77a819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f778c0a2800 0x7f778c0a4cc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 --2- 172.21.15.7:0/1107111464 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f77a8103a20 0x7f77a8199db0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 >> 172.21.15.7:0/1107111464 conn(0x7f77a80fef90 msgr2=0x7f77a8108030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 shutdown_connections 2024-03-18T01:48:27.999 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:27.998+0000 7f77ae400640 1 -- 172.21.15.7:0/1107111464 wait complete. 2024-03-18T01:48:28.017 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T01:48:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 -- 172.21.15.7:0/4041291171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7934104c10 msgr2=0x7f7934105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 --2- 172.21.15.7:0/4041291171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7934104c10 0x7f7934105090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f791c009900 tx=0x7f791c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:48:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 -- 172.21.15.7:0/4041291171 shutdown_connections 2024-03-18T01:48:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 --2- 172.21.15.7:0/4041291171 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79341055d0 0x7f793410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 --2- 172.21.15.7:0/4041291171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7934104c10 0x7f7934105090 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 --2- 172.21.15.7:0/4041291171 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7934103a20 0x7f7934103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 -- 172.21.15.7:0/4041291171 >> 172.21.15.7:0/4041291171 conn(0x7f79340fef90 msgr2=0x7f79341013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:28.227 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 -- 172.21.15.7:0/4041291171 shutdown_connections 2024-03-18T01:48:28.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.226+0000 7f793b200640 1 -- 172.21.15.7:0/4041291171 wait complete. 2024-03-18T01:48:28.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.228+0000 7f793b200640 1 Processor -- start 2024-03-18T01:48:28.228 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.228+0000 7f793b200640 1 -- start start 2024-03-18T01:48:28.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.228+0000 7f793b200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7934103a20 0x7f7934199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:28.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.228+0000 7f793b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7934104c10 0x7f793419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:28.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.228+0000 7f793b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79341055d0 0x7f79341a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:28.229 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.228+0000 7f793b200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f793410f6d0 con 0x7f7934103a20 2024-03-18T01:48:28.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.228+0000 7f793b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f793410f550 con 0x7f79341055d0 2024-03-18T01:48:28.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.228+0000 7f793b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f793410f850 con 0x7f7934104c10 2024-03-18T01:48:28.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7934104c10 0x7f793419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:28.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7934104c10 0x7f793419a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:50894/0 (socket says 172.21.15.7:50894) 2024-03-18T01:48:28.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933400640 1 -- 172.21.15.7:0/28600812 learned_addr learned my addr 172.21.15.7:0/28600812 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:28.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7939200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79341055d0 0x7f79341a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:28.230 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933e00640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7934103a20 0x7f7934199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933400640 1 -- 172.21.15.7:0/28600812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79341055d0 msgr2=0x7f79341a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933400640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79341055d0 0x7f79341a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933400640 1 -- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7934103a20 msgr2=0x7f7934199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933400640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7934103a20 0x7f7934199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.229+0000 7f7933400640 1 -- 172.21.15.7:0/28600812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79341a1950 con 0x7f7934104c10 2024-03-18T01:48:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.230+0000 7f7933400640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7934104c10 0x7f793419a280 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f791c000c00 tx=0x7f791c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.230+0000 7f7930c00640 1 -- 172.21.15.7:0/28600812 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c043020 con 0x7f7934104c10 2024-03-18T01:48:28.231 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.230+0000 7f7930c00640 1 -- 172.21.15.7:0/28600812 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f791c0054e0 con 0x7f7934104c10 2024-03-18T01:48:28.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.230+0000 7f7930c00640 1 -- 172.21.15.7:0/28600812 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c003cd0 con 0x7f7934104c10 2024-03-18T01:48:28.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.230+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79341a1be0 con 0x7f7934104c10 2024-03-18T01:48:28.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.230+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79341a2180 con 0x7f7934104c10 2024-03-18T01:48:28.232 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.232+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f793410ace0 con 0x7f7934104c10 2024-03-18T01:48:28.233 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.233+0000 7f7930c00640 1 -- 172.21.15.7:0/28600812 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f791c046020 con 0x7f7934104c10 2024-03-18T01:48:28.234 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.234+0000 7f7930c00640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f79180a2880 0x7f79180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:28.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.235+0000 7f7930c00640 1 -- 172.21.15.7:0/28600812 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f791c0f0e20 con 0x7f7934104c10 2024-03-18T01:48:28.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.235+0000 7f7933e00640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f79180a2880 0x7f79180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:28.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.235+0000 7f7933e00640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f79180a2880 0x7f79180a4d40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f793419b210 tx=0x7f7924009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:28.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.239+0000 7f7930c00640 1 -- 172.21.15.7:0/28600812 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f791c04d020 con 0x7f7934104c10 2024-03-18T01:48:28.514 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.514+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f793402d1b0 con 0x7f7934104c10 2024-03-18T01:48:28.515 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.514+0000 7f7930c00640 1 -- 172.21.15.7:0/28600812 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f791c0bdb00 con 0x7f7934104c10 2024-03-18T01:48:28.515 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 32 2024-03-18T01:48:28.518 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f79180a2880 msgr2=0x7f79180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f79180a2880 0x7f79180a4d40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f793419b210 tx=0x7f7924009500 comp rx=0 tx=0).stop 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7934104c10 msgr2=0x7f793419a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7934104c10 0x7f793419a280 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f791c000c00 tx=0x7f791c02c650 comp rx=0 tx=0).stop 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 shutdown_connections 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79341055d0 0x7f79341a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7934104c10 0x7f793419a280 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f79180a2880 0x7f79180a4d40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 --2- 172.21.15.7:0/28600812 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7934103a20 0x7f7934199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 >> 172.21.15.7:0/28600812 conn(0x7f79340fef90 msgr2=0x7f7934107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 shutdown_connections 2024-03-18T01:48:28.519 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.518+0000 7f793b200640 1 -- 172.21.15.7:0/28600812 wait complete. 2024-03-18T01:48:28.540 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T01:48:28.606 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:28 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2502480142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:48:28.606 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.606 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:28 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1107111464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:48:28.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.607 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:28 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:28 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2502480142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:48:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:28 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1107111464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:48:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:28 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.748+0000 7f7efba00640 1 -- 172.21.15.7:0/987709178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef410ddd0 msgr2=0x7f7ef41101c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.748+0000 7f7efba00640 1 --2- 172.21.15.7:0/987709178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef410ddd0 0x7f7ef41101c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7eec009ca0 tx=0x7f7eec02c100 comp rx=0 tx=0).stop 2024-03-18T01:48:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.748+0000 7f7efba00640 1 -- 172.21.15.7:0/987709178 shutdown_connections 2024-03-18T01:48:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.748+0000 7f7efba00640 1 --2- 172.21.15.7:0/987709178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef410ddd0 0x7f7ef41101c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.748+0000 7f7efba00640 1 --2- 172.21.15.7:0/987709178 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef4074240 0x7f7ef40746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.749 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.748+0000 7f7efba00640 1 --2- 172.21.15.7:0/987709178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4075820 0x7f7ef4073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.748+0000 7f7efba00640 1 -- 172.21.15.7:0/987709178 >> 172.21.15.7:0/987709178 conn(0x7f7ef40fcd70 msgr2=0x7f7ef40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.749+0000 7f7efba00640 1 -- 172.21.15.7:0/987709178 shutdown_connections 2024-03-18T01:48:28.750 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.749+0000 7f7efba00640 1 -- 172.21.15.7:0/987709178 wait complete. 2024-03-18T01:48:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7efba00640 1 Processor -- start 2024-03-18T01:48:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7efba00640 1 -- start start 2024-03-18T01:48:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7efba00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef4074240 0x7f7ef419e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:28.753 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7efba00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4075820 0x7f7ef419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7efba00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef410ddd0 0x7f7ef41a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7efba00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef4113a70 con 0x7f7ef410ddd0 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7efba00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef41138f0 con 0x7f7ef4075820 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7efba00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef4113bf0 con 0x7f7ef4074240 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7ef3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4075820 0x7f7ef419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7ef3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4075820 0x7f7ef419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46228/0 (socket says 172.21.15.7:46228) 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.751+0000 7f7ef3e00640 1 -- 172.21.15.7:0/213045230 learned_addr learned my addr 172.21.15.7:0/213045230 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.751+0000 7f7ef9a00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef410ddd0 0x7f7ef41a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:28.754 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.750+0000 7f7ef9000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef4074240 0x7f7ef419e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.751+0000 7f7ef3e00640 1 -- 172.21.15.7:0/213045230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef4074240 msgr2=0x7f7ef419e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.751+0000 7f7ef3e00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef4074240 0x7f7ef419e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.751+0000 7f7ef3e00640 1 -- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef410ddd0 msgr2=0x7f7ef41a5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.751+0000 7f7ef3e00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef410ddd0 0x7f7ef41a5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.751+0000 7f7ef3e00640 1 -- 172.21.15.7:0/213045230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef41a5cf0 con 0x7f7ef4075820 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.751+0000 7f7ef3e00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4075820 0x7f7ef419e610 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000e9f0 tx=0x7f7ee000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.752+0000 7f7ef1600640 1 -- 172.21.15.7:0/213045230 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee000cd80 con 0x7f7ef4075820 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.752+0000 7f7ef1600640 1 -- 172.21.15.7:0/213045230 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ee0010910 con 0x7f7ef4075820 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.752+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef41a5fe0 con 0x7f7ef4075820 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.752+0000 7f7ef1600640 1 -- 172.21.15.7:0/213045230 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee00188f0 con 0x7f7ef4075820 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.752+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef41a64a0 con 0x7f7ef4075820 2024-03-18T01:48:28.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.753+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef4103e90 con 0x7f7ef4075820 2024-03-18T01:48:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.755+0000 7f7ef1600640 1 -- 172.21.15.7:0/213045230 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f7ee0010ac0 con 0x7f7ef4075820 2024-03-18T01:48:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.756+0000 7f7ef1600640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7edc0a2800 0x7f7edc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.756+0000 7f7ef1600640 1 -- 172.21.15.7:0/213045230 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7ee00cf1b0 con 0x7f7ef4075820 2024-03-18T01:48:28.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.756+0000 7f7ef9000640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7edc0a2800 0x7f7edc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:28.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.757+0000 7f7ef9000640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7edc0a2800 0x7f7edc0a4cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009bd0 tx=0x7f7ee8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:28.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:28.760+0000 7f7ef1600640 1 -- 172.21.15.7:0/213045230 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7ee009b840 con 0x7f7ef4075820 2024-03-18T01:48:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:28 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2502480142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:48:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:28 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1107111464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:48:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:28.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:28 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:29.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.004+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7ef40623e0 con 0x7f7ef4075820 2024-03-18T01:48:29.004 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.004+0000 7f7ef1600640 1 -- 172.21.15.7:0/213045230 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+582 (secure 0 0 0) 0x7f7ee009f8e0 con 0x7f7ef4075820 2024-03-18T01:48:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/1169030810 2024-03-19T01:26:57.023116+0000 2024-03-18T01:48:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3833186711 2024-03-19T01:26:57.023116+0000 2024-03-18T01:48:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/4234190466 2024-03-19T01:26:57.023116+0000 2024-03-18T01:48:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6800/1791220024 2024-03-19T01:26:21.850377+0000 2024-03-18T01:48:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/1169030810 2024-03-19T01:26:57.023116+0000 2024-03-18T01:48:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:6801/1791220024 2024-03-19T01:26:21.850377+0000 2024-03-18T01:48:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3317891911 2024-03-19T01:26:21.850377+0000 2024-03-18T01:48:29.005 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2698516506 2024-03-19T01:26:21.850377+0000 2024-03-18T01:48:29.006 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/2862047901 2024-03-19T01:26:21.850377+0000 2024-03-18T01:48:29.006 INFO:teuthology.orchestra.run.smithi007.stdout:172.21.15.7:0/3780594555 2024-03-19T01:26:57.023116+0000 2024-03-18T01:48:29.006 INFO:teuthology.orchestra.run.smithi007.stderr:listed 10 entries 2024-03-18T01:48:29.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7edc0a2800 msgr2=0x7f7edc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:29.007 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7edc0a2800 0x7f7edc0a4cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009bd0 tx=0x7f7ee8005f90 comp rx=0 tx=0).stop 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4075820 msgr2=0x7f7ef419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4075820 0x7f7ef419e610 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000e9f0 tx=0x7f7ee000eeb0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 shutdown_connections 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f7edc0a2800 0x7f7edc0a4cc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f7ef410ddd0 0x7f7ef41a5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ef4075820 0x7f7ef419e610 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 --2- 172.21.15.7:0/213045230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ef4074240 0x7f7ef419e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 >> 172.21.15.7:0/213045230 conn(0x7f7ef40fcd70 msgr2=0x7f7ef4101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 shutdown_connections 2024-03-18T01:48:29.008 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.007+0000 7f7efba00640 1 -- 172.21.15.7:0/213045230 wait complete. 2024-03-18T01:48:29.030 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-18T01:48:29.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:48:29.030 DEBUG:teuthology.orchestra.run.smithi007:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-18T01:48:29.178 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi007.front.sepia.ceph.com 2024-03-18T01:48:29.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:48:29.179 DEBUG:teuthology.orchestra.run.smithi007:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T01:48:29.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:48:29.288 DEBUG:teuthology.orchestra.run.smithi007:> ip netns list 2024-03-18T01:48:29.369 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T01:48:29.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:48:29.370 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T01:48:29.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:48:29.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo ip link delete ceph-brx 2024-03-18T01:48:29.568 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T01:48:29.578 INFO:tasks.cephadm:Teardown begin 2024-03-18T01:48:29.579 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T01:48:29.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:48:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:29 smithi203 ceph-mon[36345]: pgmap v726: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:29 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/28600812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:48:29.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:29 smithi203 ceph-mon[36345]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:48:29.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:29 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/213045230' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:48:29.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:29 smithi007 ceph-mon[31996]: pgmap v726: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:29.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:29 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/28600812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:48:29.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:29 smithi007 ceph-mon[31996]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:48:29.653 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:29 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/213045230' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:48:29.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:29 smithi046 ceph-mon[36108]: pgmap v726: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T01:48:29.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:29 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/28600812' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T01:48:29.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:29 smithi046 ceph-mon[36108]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:48:29.670 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:29 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/213045230' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 -- 172.21.15.7:0/475572502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd80100a10 msgr2=0x7efd80100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 --2- 172.21.15.7:0/475572502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd80100a10 0x7efd80100e70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009900 tx=0x7efd6c02bc40 comp rx=0 tx=0).stop 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 -- 172.21.15.7:0/475572502 shutdown_connections 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 --2- 172.21.15.7:0/475572502 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd801013b0 0x7efd80101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 --2- 172.21.15.7:0/475572502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd80100a10 0x7efd80100e70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 --2- 172.21.15.7:0/475572502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd801000b0 0x7efd801004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 -- 172.21.15.7:0/475572502 >> 172.21.15.7:0/475572502 conn(0x7efd800fbc70 msgr2=0x7efd800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 -- 172.21.15.7:0/475572502 shutdown_connections 2024-03-18T01:48:29.785 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.784+0000 7efd84c00640 1 -- 172.21.15.7:0/475572502 wait complete. 2024-03-18T01:48:29.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.785+0000 7efd84c00640 1 Processor -- start 2024-03-18T01:48:29.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.785+0000 7efd84c00640 1 -- start start 2024-03-18T01:48:29.787 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd84c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd801000b0 0x7efd80199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd84c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd80100a10 0x7efd8019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd84c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd801013b0 0x7efd801a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd84c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7efd8010f6f0 con 0x7efd801013b0 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd84c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efd8010f570 con 0x7efd801000b0 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd84c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efd8010f870 con 0x7efd80100a10 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd80100a10 0x7efd8019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd801000b0 0x7efd80199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7d400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd80100a10 0x7efd8019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:50942/0 (socket says 172.21.15.7:50942) 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7d400640 1 -- 172.21.15.7:0/1670611058 learned_addr learned my addr 172.21.15.7:0/1670611058 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd801000b0 0x7efd80199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46242/0 (socket says 172.21.15.7:46242) 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7e800640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd801013b0 0x7efd801a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7d400640 1 -- 172.21.15.7:0/1670611058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd801000b0 msgr2=0x7efd80199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7d400640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd801000b0 0x7efd80199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.788 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7d400640 1 -- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd801013b0 msgr2=0x7efd801a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7d400640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd801013b0 0x7efd801a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.786+0000 7efd7d400640 1 -- 172.21.15.7:0/1670611058 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd801a1960 con 0x7efd80100a10 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.787+0000 7efd7d400640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd80100a10 0x7efd8019a290 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efd6c02c8f0 tx=0x7efd6c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.787+0000 7efd76a00640 1 -- 172.21.15.7:0/1670611058 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c041070 con 0x7efd80100a10 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.787+0000 7efd76a00640 1 -- 172.21.15.7:0/1670611058 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd6c0044d0 con 0x7efd80100a10 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.787+0000 7efd76a00640 1 -- 172.21.15.7:0/1670611058 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c034440 con 0x7efd80100a10 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.787+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd801a1bf0 con 0x7efd80100a10 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.787+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd801a2130 con 0x7efd80100a10 2024-03-18T01:48:29.789 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.788+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8010af90 con 0x7efd80100a10 2024-03-18T01:48:29.790 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.790+0000 7efd76a00640 1 -- 172.21.15.7:0/1670611058 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7efd6c044070 con 0x7efd80100a10 2024-03-18T01:48:29.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.791+0000 7efd76a00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd640a2880 0x7efd640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:29.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.791+0000 7efd76a00640 1 -- 172.21.15.7:0/1670611058 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efd6c0f0ac0 con 0x7efd80100a10 2024-03-18T01:48:29.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.792+0000 7efd7de00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd640a2880 0x7efd640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:29.792 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.792+0000 7efd7de00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd640a2880 0x7efd640a4d40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efd8019b220 tx=0x7efd70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:29.795 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:29.795+0000 7efd76a00640 1 -- 172.21.15.7:0/1670611058 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd6c0bd7a0 con 0x7efd80100a10 2024-03-18T01:48:30.043 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.043+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd8002d1b0 con 0x7efd80100a10 2024-03-18T01:48:30.044 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.044+0000 7efd76a00640 1 -- 172.21.15.7:0/1670611058 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+11937 (secure 0 0 0) 0x7efd6c0bd0a0 con 0x7efd80100a10 2024-03-18T01:48:30.044 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:48:30.045 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":151,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:48:28.027922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":164,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":160,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":188,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":160,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":158,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":157,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":182,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24544,24523,24542,24596,24562]},"id":1}]} 2024-03-18T01:48:30.045 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 151 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd640a2880 msgr2=0x7efd640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd640a2880 0x7efd640a4d40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efd8019b220 tx=0x7efd70005f90 comp rx=0 tx=0).stop 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd80100a10 msgr2=0x7efd8019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd80100a10 0x7efd8019a290 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efd6c02c8f0 tx=0x7efd6c005d50 comp rx=0 tx=0).stop 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 shutdown_connections 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7efd640a2880 0x7efd640a4d40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7efd801013b0 0x7efd801a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd80100a10 0x7efd8019a290 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 --2- 172.21.15.7:0/1670611058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd801000b0 0x7efd80199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 >> 172.21.15.7:0/1670611058 conn(0x7efd800fbc70 msgr2=0x7efd800fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 shutdown_connections 2024-03-18T01:48:30.047 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.046+0000 7efd84c00640 1 -- 172.21.15.7:0/1670611058 wait complete. 2024-03-18T01:48:30.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:48:30.253 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 -- 172.21.15.7:0/3939006813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb481049f0 msgr2=0x7fcb48104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 --2- 172.21.15.7:0/3939006813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb481049f0 0x7fcb48104e70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009900 tx=0x7fcb3002bc50 comp rx=0 tx=0).stop 2024-03-18T01:48:30.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 -- 172.21.15.7:0/3939006813 shutdown_connections 2024-03-18T01:48:30.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 --2- 172.21.15.7:0/3939006813 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb481053b0 0x7fcb4810bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 --2- 172.21.15.7:0/3939006813 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb481049f0 0x7fcb48104e70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 --2- 172.21.15.7:0/3939006813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb48103800 0x7fcb48103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 -- 172.21.15.7:0/3939006813 >> 172.21.15.7:0/3939006813 conn(0x7fcb480fed70 msgr2=0x7fcb481011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:30.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 -- 172.21.15.7:0/3939006813 shutdown_connections 2024-03-18T01:48:30.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.253+0000 7fcb47200640 1 -- 172.21.15.7:0/3939006813 wait complete. 2024-03-18T01:48:30.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.254+0000 7fcb47200640 1 Processor -- start 2024-03-18T01:48:30.256 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.254+0000 7fcb47200640 1 -- start start 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.254+0000 7fcb47200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb48103800 0x7fcb48199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.254+0000 7fcb47200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb481049f0 0x7fcb48199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.254+0000 7fcb47200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb481053b0 0x7fcb481a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.254+0000 7fcb47200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4810f4b0 con 0x7fcb481049f0 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.254+0000 7fcb47200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4810f330 con 0x7fcb481053b0 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.254+0000 7fcb47200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4810f630 con 0x7fcb48103800 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb48103800 0x7fcb48199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb48103800 0x7fcb48199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:50954/0 (socket says 172.21.15.7:50954) 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 -- 172.21.15.7:0/3862206470 learned_addr learned my addr 172.21.15.7:0/3862206470 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb3d400640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb481049f0 0x7fcb48199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb46800640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb481053b0 0x7fcb481a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:30.257 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 -- 172.21.15.7:0/3862206470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb481053b0 msgr2=0x7fcb481a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb481053b0 0x7fcb481a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 -- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb481049f0 msgr2=0x7fcb48199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb481049f0 0x7fcb48199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 -- 172.21.15.7:0/3862206470 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb481a1620 con 0x7fcb48103800 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.255+0000 7fcb45e00640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb48103800 0x7fcb48199a00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800ebc0 tx=0x7fcb3800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.256+0000 7fcb3f400640 1 -- 172.21.15.7:0/3862206470 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb38003b70 con 0x7fcb48103800 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.256+0000 7fcb3f400640 1 -- 172.21.15.7:0/3862206470 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb38003d10 con 0x7fcb48103800 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.256+0000 7fcb3f400640 1 -- 172.21.15.7:0/3862206470 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb38019980 con 0x7fcb48103800 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.256+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb481a1910 con 0x7fcb48103800 2024-03-18T01:48:30.258 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.256+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb481a1de0 con 0x7fcb48103800 2024-03-18T01:48:30.259 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.257+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb4810aa50 con 0x7fcb48103800 2024-03-18T01:48:30.261 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.261+0000 7fcb3f400640 1 -- 172.21.15.7:0/3862206470 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fcb38004b60 con 0x7fcb48103800 2024-03-18T01:48:30.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.262+0000 7fcb3f400640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcb2c0a2920 0x7fcb2c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:30.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.262+0000 7fcb3f400640 1 -- 172.21.15.7:0/3862206470 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcb380cf580 con 0x7fcb48103800 2024-03-18T01:48:30.262 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.262+0000 7fcb3d400640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcb2c0a2920 0x7fcb2c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:30.263 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.263+0000 7fcb3d400640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcb2c0a2920 0x7fcb2c0a4de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcb3002c260 tx=0x7fcb30000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:30.265 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.265+0000 7fcb3f400640 1 -- 172.21.15.7:0/3862206470 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb3809c2c0 con 0x7fcb48103800 2024-03-18T01:48:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.508+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb4802d460 con 0x7fcb48103800 2024-03-18T01:48:30.509 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.509+0000 7fcb3f400640 1 -- 172.21.15.7:0/3862206470 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+11937 (secure 0 0 0) 0x7fcb3809bbc0 con 0x7fcb48103800 2024-03-18T01:48:30.510 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:48:30.510 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14403,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6837/3527563344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6836","nonce":3527563344},{"type":"v1","addr":"172.21.15.7:6837","nonce":3527563344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.7:6841/1978448378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6840","nonce":1978448378},{"type":"v1","addr":"172.21.15.7:6841","nonce":1978448378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":151,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T01:34:56.125856+0000","modified":"2024-03-18T01:48:28.027922+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24596,"mds_1":24562,"mds_2":24544,"mds_3":24542,"mds_4":24523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":164,"addr":"172.21.15.203:6833/2100401567","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":2100401567},{"type":"v1","addr":"172.21.15.203:6833","nonce":2100401567}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.46:6833/1027889667","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1027889667},{"type":"v1","addr":"172.21.15.46:6833","nonce":1027889667}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24475":{"gid":24475,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.7:6835/49989672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6834","nonce":49989672},{"type":"v1","addr":"172.21.15.7:6835","nonce":49989672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/848080613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":848080613},{"type":"v1","addr":"172.21.15.203:6835","nonce":848080613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24512":{"gid":24512,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":160,"addr":"172.21.15.46:6835/2878820671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2878820671},{"type":"v1","addr":"172.21.15.46:6835","nonce":2878820671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":188,"addr":"172.21.15.203:6837/2344597644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":2344597644},{"type":"v1","addr":"172.21.15.203:6837","nonce":2344597644}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24542":{"gid":24542,"name":"h","rank":3,"incarnation":24,"state":"up:active","state_seq":160,"addr":"172.21.15.46:6837/1509246906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":1509246906},{"type":"v1","addr":"172.21.15.46:6837","nonce":1509246906}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24544":{"gid":24544,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":158,"addr":"172.21.15.203:6839/3801010143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":3801010143},{"type":"v1","addr":"172.21.15.203:6839","nonce":3801010143}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24562":{"gid":24562,"name":"g","rank":1,"incarnation":20,"state":"up:active","state_seq":157,"addr":"172.21.15.7:6839/3166631604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6838","nonce":3166631604},{"type":"v1","addr":"172.21.15.7:6839","nonce":3166631604}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24596":{"gid":24596,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":182,"addr":"172.21.15.46:6839/1986652812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1986652812},{"type":"v1","addr":"172.21.15.46:6839","nonce":1986652812}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24596,"qdb_cluster":[24544,24523,24542,24596,24562]},"id":1}]} 2024-03-18T01:48:30.510 INFO:teuthology.orchestra.run.smithi007.stderr:dumped fsmap epoch 151 2024-03-18T01:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcb2c0a2920 msgr2=0x7fcb2c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcb2c0a2920 0x7fcb2c0a4de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fcb3002c260 tx=0x7fcb30000bc0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.512 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb48103800 msgr2=0x7fcb48199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb48103800 0x7fcb48199a00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800ebc0 tx=0x7fcb3800c6a0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 shutdown_connections 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb481053b0 0x7fcb481a0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fcb2c0a2920 0x7fcb2c0a4de0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fcb481049f0 0x7fcb48199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 --2- 172.21.15.7:0/3862206470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb48103800 0x7fcb48199a00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 >> 172.21.15.7:0/3862206470 conn(0x7fcb480fed70 msgr2=0x7fcb48107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 shutdown_connections 2024-03-18T01:48:30.513 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.512+0000 7fcb47200640 1 -- 172.21.15.7:0/3862206470 wait complete. 2024-03-18T01:48:30.531 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:48:30.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.716+0000 7f623d200640 1 -- 172.21.15.7:0/257784307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6238101620 msgr2=0x7f6238101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.716+0000 7f623d200640 1 --2- 172.21.15.7:0/257784307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6238101620 0x7f6238101a40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6228009900 tx=0x7f622802bc40 comp rx=0 tx=0).stop 2024-03-18T01:48:30.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.716+0000 7f623d200640 1 -- 172.21.15.7:0/257784307 shutdown_connections 2024-03-18T01:48:30.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.716+0000 7f623d200640 1 --2- 172.21.15.7:0/257784307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62381031d0 0x7f6238109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.717 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.716+0000 7f623d200640 1 --2- 172.21.15.7:0/257784307 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f6238102810 0x7f6238102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.716+0000 7f623d200640 1 --2- 172.21.15.7:0/257784307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6238101620 0x7f6238101a40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.716+0000 7f623d200640 1 -- 172.21.15.7:0/257784307 >> 172.21.15.7:0/257784307 conn(0x7f62380fcb90 msgr2=0x7f62380fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:30.718 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.717+0000 7f623d200640 1 -- 172.21.15.7:0/257784307 shutdown_connections 2024-03-18T01:48:30.719 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.717+0000 7f623d200640 1 -- 172.21.15.7:0/257784307 wait complete. 2024-03-18T01:48:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f623d200640 1 Processor -- start 2024-03-18T01:48:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f623d200640 1 -- start start 2024-03-18T01:48:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f623d200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6238101620 0x7f6238199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f623d200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6238102810 0x7f6238199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f623d200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381031d0 0x7f62381a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f623d200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f623810d2f0 con 0x7f62381031d0 2024-03-18T01:48:30.721 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f623d200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f623810d170 con 0x7f6238101620 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f623d200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f623810d470 con 0x7f6238102810 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f6237400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6238101620 0x7f6238199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f6237400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6238101620 0x7f6238199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.7:46258/0 (socket says 172.21.15.7:46258) 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f6237400640 1 -- 172.21.15.7:0/1271209474 learned_addr learned my addr 172.21.15.7:0/1271209474 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f6237e00640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381031d0 0x7f62381a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.718+0000 7f6236a00640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6238102810 0x7f6238199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f6237400640 1 -- 172.21.15.7:0/1271209474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6238102810 msgr2=0x7f6238199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f6237400640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6238102810 0x7f6238199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f6237400640 1 -- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381031d0 msgr2=0x7f62381a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f6237400640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381031d0 0x7f62381a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f6237400640 1 -- 172.21.15.7:0/1271209474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62381a1680 con 0x7f6238101620 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f6237400640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6238101620 0x7f6238199a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f622802efd0 tx=0x7f622802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:30.722 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f622fe00640 1 -- 172.21.15.7:0/1271209474 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228041070 con 0x7f6238101620 2024-03-18T01:48:30.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f622fe00640 1 -- 172.21.15.7:0/1271209474 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6228004450 con 0x7f6238101620 2024-03-18T01:48:30.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62381a1910 con 0x7f6238101620 2024-03-18T01:48:30.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.719+0000 7f622fe00640 1 -- 172.21.15.7:0/1271209474 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228003e50 con 0x7f6238101620 2024-03-18T01:48:30.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.720+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62381a1e20 con 0x7f6238101620 2024-03-18T01:48:30.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.720+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62381a2050 con 0x7f6238101620 2024-03-18T01:48:30.723 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.722+0000 7f622fe00640 1 -- 172.21.15.7:0/1271209474 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f6228044070 con 0x7f6238101620 2024-03-18T01:48:30.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.724+0000 7f622fe00640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62180a2ac0 0x7f62180a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:30.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.724+0000 7f622fe00640 1 -- 172.21.15.7:0/1271209474 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f62280f0c20 con 0x7f6238101620 2024-03-18T01:48:30.724 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.724+0000 7f6236a00640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62180a2ac0 0x7f62180a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:30.725 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.725+0000 7f6236a00640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62180a2ac0 0x7f62180a4f80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f623819af70 tx=0x7f6220009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:30.731 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.731+0000 7f622fe00640 1 -- 172.21.15.7:0/1271209474 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62280bd2b0 con 0x7f6238101620 2024-03-18T01:48:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:30 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:30 smithi046 ceph-mon[36108]: pgmap v727: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T01:48:30.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:30 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1670611058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:48:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:30.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:30 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:30.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:30 smithi007 ceph-mon[31996]: pgmap v727: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T01:48:30.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:30 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1670611058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:48:30.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:30.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:30 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:30.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:30 smithi203 ceph-mon[36345]: pgmap v727: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T01:48:30.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:30 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1670611058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:48:30.934 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.933+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62381a2220 con 0x7f6238101620 2024-03-18T01:48:30.936 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.935+0000 7f622fe00640 1 -- 172.21.15.7:0/1271209474 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24543 (secure 0 0 0) 0x7f62280b7e20 con 0x7f6238101620 2024-03-18T01:48:30.936 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-18T01:48:30.936 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":91,"fsid":"3fdfdc7c-e4c6-11ee-95c9-87774f69a715","created":"2024-03-18T01:25:48.473512+0000","modified":"2024-03-18T01:35:58.796670+0000","last_up_change":"2024-03-18T01:33:57.825901+0000","last_in_change":"2024-03-18T01:33:38.065700+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T01:29:41.723841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T01:34:54.204638+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T01:34:55.616951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T01:34:59.640737+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"ae73eb8b-e60e-49d5-9fb1-9240be07f1d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6803","nonce":2740971601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6805","nonce":2740971601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6809","nonce":2740971601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2740971601},{"type":"v1","addr":"172.21.15.7:6807","nonce":2740971601}]},"public_addr":"172.21.15.7:6803/2740971601","cluster_addr":"172.21.15.7:6805/2740971601","heartbeat_back_addr":"172.21.15.7:6809/2740971601","heartbeat_front_addr":"172.21.15.7:6807/2740971601","state":["exists","up"]},{"osd":1,"uuid":"1dc69463-fc6a-4265-9ac8-5e1a6a1beb4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6801","nonce":868303069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6803","nonce":868303069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6807","nonce":868303069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":868303069},{"type":"v1","addr":"172.21.15.46:6805","nonce":868303069}]},"public_addr":"172.21.15.46:6801/868303069","cluster_addr":"172.21.15.46:6803/868303069","heartbeat_back_addr":"172.21.15.46:6807/868303069","heartbeat_front_addr":"172.21.15.46:6805/868303069","state":["exists","up"]},{"osd":2,"uuid":"32b52cb8-4fcc-4a3a-9e2a-69a8205f0844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6801","nonce":3828545760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6803","nonce":3828545760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6807","nonce":3828545760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3828545760},{"type":"v1","addr":"172.21.15.203:6805","nonce":3828545760}]},"public_addr":"172.21.15.203:6801/3828545760","cluster_addr":"172.21.15.203:6803/3828545760","heartbeat_back_addr":"172.21.15.203:6807/3828545760","heartbeat_front_addr":"172.21.15.203:6805/3828545760","state":["exists","up"]},{"osd":3,"uuid":"b3a7ea32-8f7d-4f40-9f5e-09e9f8fd8748","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6811","nonce":2660921725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6813","nonce":2660921725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6817","nonce":2660921725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2660921725},{"type":"v1","addr":"172.21.15.7:6815","nonce":2660921725}]},"public_addr":"172.21.15.7:6811/2660921725","cluster_addr":"172.21.15.7:6813/2660921725","heartbeat_back_addr":"172.21.15.7:6817/2660921725","heartbeat_front_addr":"172.21.15.7:6815/2660921725","state":["exists","up"]},{"osd":4,"uuid":"175f4003-26aa-4f4c-b0f9-cb83d4d8ab87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6809","nonce":35412412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6811","nonce":35412412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6815","nonce":35412412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":35412412},{"type":"v1","addr":"172.21.15.46:6813","nonce":35412412}]},"public_addr":"172.21.15.46:6809/35412412","cluster_addr":"172.21.15.46:6811/35412412","heartbeat_back_addr":"172.21.15.46:6815/35412412","heartbeat_front_addr":"172.21.15.46:6813/35412412","state":["exists","up"]},{"osd":5,"uuid":"30797882-136d-4d56-8f01-3e02850e5c9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6809","nonce":2080514850}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6811","nonce":2080514850}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6815","nonce":2080514850}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2080514850},{"type":"v1","addr":"172.21.15.203:6813","nonce":2080514850}]},"public_addr":"172.21.15.203:6809/2080514850","cluster_addr":"172.21.15.203:6811/2080514850","heartbeat_back_addr":"172.21.15.203:6815/2080514850","heartbeat_front_addr":"172.21.15.203:6813/2080514850","state":["exists","up"]},{"osd":6,"uuid":"646180e1-c71d-4288-88f0-20ded1848288","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6819","nonce":3071783622}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6821","nonce":3071783622}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6825","nonce":3071783622}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3071783622},{"type":"v1","addr":"172.21.15.7:6823","nonce":3071783622}]},"public_addr":"172.21.15.7:6819/3071783622","cluster_addr":"172.21.15.7:6821/3071783622","heartbeat_back_addr":"172.21.15.7:6825/3071783622","heartbeat_front_addr":"172.21.15.7:6823/3071783622","state":["exists","up"]},{"osd":7,"uuid":"aa2571c2-11ec-4e31-b7a8-5ef0cf2899ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6817","nonce":3445048558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6819","nonce":3445048558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6823","nonce":3445048558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":3445048558},{"type":"v1","addr":"172.21.15.46:6821","nonce":3445048558}]},"public_addr":"172.21.15.46:6817/3445048558","cluster_addr":"172.21.15.46:6819/3445048558","heartbeat_back_addr":"172.21.15.46:6823/3445048558","heartbeat_front_addr":"172.21.15.46:6821/3445048558","state":["exists","up"]},{"osd":8,"uuid":"db012442-b453-4810-8aba-51d2e9506797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6817","nonce":1251446815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6819","nonce":1251446815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6823","nonce":1251446815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":1251446815},{"type":"v1","addr":"172.21.15.203:6821","nonce":1251446815}]},"public_addr":"172.21.15.203:6817/1251446815","cluster_addr":"172.21.15.203:6819/1251446815","heartbeat_back_addr":"172.21.15.203:6823/1251446815","heartbeat_front_addr":"172.21.15.203:6821/1251446815","state":["exists","up"]},{"osd":9,"uuid":"352a39d7-32c4-461f-8c31-61badd6ba568","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6827","nonce":4123652106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6829","nonce":4123652106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6833","nonce":4123652106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":4123652106},{"type":"v1","addr":"172.21.15.7:6831","nonce":4123652106}]},"public_addr":"172.21.15.7:6827/4123652106","cluster_addr":"172.21.15.7:6829/4123652106","heartbeat_back_addr":"172.21.15.7:6833/4123652106","heartbeat_front_addr":"172.21.15.7:6831/4123652106","state":["exists","up"]},{"osd":10,"uuid":"d452ca28-0c10-48a8-bd4a-f01b4e4986c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6825","nonce":511858712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6827","nonce":511858712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6831","nonce":511858712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":511858712},{"type":"v1","addr":"172.21.15.46:6829","nonce":511858712}]},"public_addr":"172.21.15.46:6825/511858712","cluster_addr":"172.21.15.46:6827/511858712","heartbeat_back_addr":"172.21.15.46:6831/511858712","heartbeat_front_addr":"172.21.15.46:6829/511858712","state":["exists","up"]},{"osd":11,"uuid":"7b43ff2d-9d94-4fd3-9a3d-9a2da004016b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6825","nonce":2624070508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6827","nonce":2624070508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6831","nonce":2624070508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":2624070508},{"type":"v1","addr":"172.21.15.203:6829","nonce":2624070508}]},"public_addr":"172.21.15.203:6825/2624070508","cluster_addr":"172.21.15.203:6827/2624070508","heartbeat_back_addr":"172.21.15.203:6831/2624070508","heartbeat_front_addr":"172.21.15.203:6829/2624070508","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:28:49.421753+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:15.032983+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:29:38.540256+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:10.397707+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:30:37.127074+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:01.963481+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:31:36.761908+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:03.643932+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:32:28.576355+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:02.806604+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:30.418416+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T01:33:55.935649+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.10","mappings":[{"from":9,"to":10}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6800/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/3833186711":"2024-03-19T01:26:57.023116+0000","172.21.15.7:0/4234190466":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6800/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:6801/1169030810":"2024-03-19T01:26:57.023116+0000","172.21.15.7:6801/1791220024":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3317891911":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2698516506":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/2862047901":"2024-03-19T01:26:21.850377+0000","172.21.15.7:0/3780594555":"2024-03-19T01:26:57.023116+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62180a2ac0 msgr2=0x7f62180a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62180a2ac0 0x7f62180a4f80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f623819af70 tx=0x7f6220009500 comp rx=0 tx=0).stop 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6238101620 msgr2=0x7f6238199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6238101620 0x7f6238199a60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f622802efd0 tx=0x7f622802c660 comp rx=0 tx=0).stop 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 shutdown_connections 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f62180a2ac0 0x7f62180a4f80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f62381031d0 0x7f62381a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6238102810 0x7f6238199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 --2- 172.21.15.7:0/1271209474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6238101620 0x7f6238199a60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:30.939 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.938+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 >> 172.21.15.7:0/1271209474 conn(0x7f62380fcb90 msgr2=0x7f6238073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:30.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.939+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 shutdown_connections 2024-03-18T01:48:30.940 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:30.939+0000 7f623d200640 1 -- 172.21.15.7:0/1271209474 wait complete. 2024-03-18T01:48:30.961 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T01:48:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 -- 172.21.15.7:0/3330726656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01049e0 msgr2=0x7f24f0106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3330726656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01049e0 0x7f24f0106dd0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009860 tx=0x7f24dc02bb70 comp rx=0 tx=0).stop 2024-03-18T01:48:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 -- 172.21.15.7:0/3330726656 shutdown_connections 2024-03-18T01:48:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3330726656 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24f0107310 0x7f24f0109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3330726656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01049e0 0x7f24f0106dd0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3330726656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f01020b0 0x7f24f01044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 -- 172.21.15.7:0/3330726656 >> 172.21.15.7:0/3330726656 conn(0x7f24f00fbac0 msgr2=0x7f24f00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:31.235 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 -- 172.21.15.7:0/3330726656 shutdown_connections 2024-03-18T01:48:31.236 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.234+0000 7f24f6e00640 1 -- 172.21.15.7:0/3330726656 wait complete. 2024-03-18T01:48:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.235+0000 7f24f6e00640 1 Processor -- start 2024-03-18T01:48:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.235+0000 7f24f6e00640 1 -- start start 2024-03-18T01:48:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01020b0 0x7f24f0101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24f01049e0 0x7f24f0100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f0107310 0x7f24f01005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6e00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f24f010d290 con 0x7f24f01049e0 2024-03-18T01:48:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24f010d110 con 0x7f24f01020b0 2024-03-18T01:48:31.238 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6e00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24f010d410 con 0x7f24f0107310 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f0107310 0x7f24f01005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f5a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01020b0 0x7f24f0101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6400640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f0107310 0x7f24f01005a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:50992/0 (socket says 172.21.15.7:50992) 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f6400640 1 -- 172.21.15.7:0/3685910379 learned_addr learned my addr 172.21.15.7:0/3685910379 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.236+0000 7f24f5000640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24f01049e0 0x7f24f0100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f6400640 1 -- 172.21.15.7:0/3685910379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01020b0 msgr2=0x7f24f0101a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f6400640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01020b0 0x7f24f0101a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f6400640 1 -- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24f01049e0 msgr2=0x7f24f0100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f6400640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24f01049e0 0x7f24f0100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f6400640 1 -- 172.21.15.7:0/3685910379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f0100e30 con 0x7f24f0107310 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f5a00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01020b0 0x7f24f0101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f5000640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24f01049e0 0x7f24f0100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f6400640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f0107310 0x7f24f01005a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f24e800eb10 tx=0x7f24e800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:31.239 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24ee000640 1 -- 172.21.15.7:0/3685910379 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8004b20 con 0x7f24f0107310 2024-03-18T01:48:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24ee000640 1 -- 172.21.15.7:0/3685910379 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24e8004cc0 con 0x7f24f0107310 2024-03-18T01:48:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f01a1a50 con 0x7f24f0107310 2024-03-18T01:48:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.237+0000 7f24ee000640 1 -- 172.21.15.7:0/3685910379 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8010760 con 0x7f24f0107310 2024-03-18T01:48:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.238+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f01a1f70 con 0x7f24f0107310 2024-03-18T01:48:31.240 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.239+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f004f540 con 0x7f24f0107310 2024-03-18T01:48:31.241 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.241+0000 7f24ee000640 1 -- 172.21.15.7:0/3685910379 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f24e8005020 con 0x7f24f0107310 2024-03-18T01:48:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.243+0000 7f24ee000640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24d80a2880 0x7f24d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:31.243 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.243+0000 7f24ee000640 1 -- 172.21.15.7:0/3685910379 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f24e80cf3d0 con 0x7f24f0107310 2024-03-18T01:48:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.243+0000 7f24f5a00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24d80a2880 0x7f24d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:31.244 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.244+0000 7f24f5a00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24d80a2880 0x7f24d80a4d40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f24e40096c0 tx=0x7f24e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:31.251 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.251+0000 7f24ee000640 1 -- 172.21.15.7:0/3685910379 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f24e809c110 con 0x7f24f0107310 2024-03-18T01:48:31.507 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.506+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f24f00623e0 con 0x7f24f0107310 2024-03-18T01:48:31.534 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.534+0000 7f24ee000640 1 -- 172.21.15.7:0/3685910379 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v152) ==== 161+0+0 (secure 0 0 0) 0x7f24e809ba10 con 0x7f24f0107310 2024-03-18T01:48:31.535 INFO:teuthology.orchestra.run.smithi007.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T01:48:31.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.536+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24d80a2880 msgr2=0x7f24d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:31.537 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.536+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24d80a2880 0x7f24d80a4d40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f24e40096c0 tx=0x7f24e4009260 comp rx=0 tx=0).stop 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.536+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f0107310 msgr2=0x7f24f01005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.536+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f0107310 0x7f24f01005a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f24e800eb10 tx=0x7f24e800c2d0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.537+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 shutdown_connections 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.537+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24f0107310 0x7f24f01005a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.537+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f24d80a2880 0x7f24d80a4d40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.537+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f24f01049e0 0x7f24f0100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.537+0000 7f24f6e00640 1 --2- 172.21.15.7:0/3685910379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24f01020b0 0x7f24f0101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.537+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 >> 172.21.15.7:0/3685910379 conn(0x7f24f00fbac0 msgr2=0x7f24f0102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.537+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 shutdown_connections 2024-03-18T01:48:31.538 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.537+0000 7f24f6e00640 1 -- 172.21.15.7:0/3685910379 wait complete. 2024-03-18T01:48:31.562 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T01:48:31.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3862206470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:48:31.676 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:48:31.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:48:31.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:31.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/1271209474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:48:31.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a[31992]: 2024-03-18T01:48:31.524+0000 7f752d800700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:48:31.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3862206470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:48:31.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:48:31.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:48:31.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:31.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/1271209474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:48:31.770 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e[66143]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:31.770 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e[66143]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:31.770 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-mds[66147]: -- 172.21.15.46:0/3333410756 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc4039fa40 con 0x55bc40372c00 2024-03-18T01:48:31.770 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e[66143]: starting mds.e at 2024-03-18T01:48:31.771 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:31.771 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:31.771 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: starting mds.b at 2024-03-18T01:48:31.771 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h[66456]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:31.771 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h[66456]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:31.771 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-mds[66460]: -- 172.21.15.46:0/4176085699 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557aa838da40 con 0x557aa8361000 2024-03-18T01:48:31.771 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h[66456]: starting mds.h at 2024-03-18T01:48:31.772 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k[65822]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:31.772 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k[65822]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:31.772 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-mds[65826]: -- 172.21.15.46:0/104230414 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5640113f5a40 con 0x5640113c9000 2024-03-18T01:48:31.772 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:48:31 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k[65822]: starting mds.k at 2024-03-18T01:48:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 -- 172.21.15.7:0/3394852492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 msgr2=0x7f3238106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 --2- 172.21.15.7:0/3394852492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 0x7f3238106fa0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3220009900 tx=0x7f322002bc40 comp rx=0 tx=0).stop 2024-03-18T01:48:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 -- 172.21.15.7:0/3394852492 shutdown_connections 2024-03-18T01:48:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 --2- 172.21.15.7:0/3394852492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32381074e0 0x7f3238109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 --2- 172.21.15.7:0/3394852492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 0x7f3238106fa0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 --2- 172.21.15.7:0/3394852492 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3238102280 0x7f3238104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 -- 172.21.15.7:0/3394852492 >> 172.21.15.7:0/3394852492 conn(0x7f32380fbc70 msgr2=0x7f32380fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:31.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 -- 172.21.15.7:0/3394852492 shutdown_connections 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.771+0000 7f323d000640 1 -- 172.21.15.7:0/3394852492 wait complete. 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.772+0000 7f323d000640 1 Processor -- start 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.772+0000 7f323d000640 1 -- start start 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.772+0000 7f323d000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3238102280 0x7f3238199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.772+0000 7f323d000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 0x7f323819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.772+0000 7f323d000640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32381074e0 0x7f32381a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.772+0000 7f323d000640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f323810d4d0 con 0x7f3238102280 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.772+0000 7f323d000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f323810d350 con 0x7f3238104bb0 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.772+0000 7f323d000640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f323810d650 con 0x7f32381074e0 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f3235e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3238102280 0x7f3238199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f322d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 0x7f323819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f3235e00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3238102280 0x7f3238199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54738/0 (socket says 172.21.15.7:54738) 2024-03-18T01:48:31.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f3235e00640 1 -- 172.21.15.7:0/3984199896 learned_addr learned my addr 172.21.15.7:0/3984199896 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f3236800640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32381074e0 0x7f32381a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f322d400640 1 -- 172.21.15.7:0/3984199896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32381074e0 msgr2=0x7f32381a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f322d400640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32381074e0 0x7f32381a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f322d400640 1 -- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3238102280 msgr2=0x7f3238199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f322d400640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3238102280 0x7f3238199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f322d400640 1 -- 172.21.15.7:0/3984199896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32381a1960 con 0x7f3238104bb0 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.773+0000 7f3235e00640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3238102280 0x7f3238199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.774+0000 7f322d400640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 0x7f323819a280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3220009230 tx=0x7f322002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.774+0000 7f322f400640 1 -- 172.21.15.7:0/3984199896 <== mon.2 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220041070 con 0x7f3238104bb0 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.774+0000 7f322f400640 1 -- 172.21.15.7:0/3984199896 <== mon.2 v2:172.21.15.46:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32200054e0 con 0x7f3238104bb0 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.774+0000 7f322f400640 1 -- 172.21.15.7:0/3984199896 <== mon.2 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220003e40 con 0x7f3238104bb0 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.774+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32381a1bf0 con 0x7f3238104bb0 2024-03-18T01:48:31.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.774+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32381a2060 con 0x7f3238104bb0 2024-03-18T01:48:31.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.775+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3238109030 con 0x7f3238104bb0 2024-03-18T01:48:31.778 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.777+0000 7f322f400640 1 -- 172.21.15.7:0/3984199896 <== mon.2 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f3220044070 con 0x7f3238104bb0 2024-03-18T01:48:31.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.778+0000 7f322f400640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f321c0a2880 0x7f321c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:31.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.779+0000 7f322f400640 1 -- 172.21.15.7:0/3984199896 <== mon.2 v2:172.21.15.46:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9522+0+0 (secure 0 0 0) 0x7f32200f09a0 con 0x7f3238104bb0 2024-03-18T01:48:31.779 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.779+0000 7f3235e00640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f321c0a2880 0x7f321c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:31.780 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.779+0000 7f3235e00640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f321c0a2880 0x7f321c0a4d40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f322800f7d0 tx=0x7f3228009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:31.783 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:31.783+0000 7f322f400640 1 -- 172.21.15.7:0/3984199896 <== mon.2 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f32200bce80 con 0x7f3238104bb0 2024-03-18T01:48:31.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3862206470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:48:31.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:48:31.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:48:31.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' 2024-03-18T01:48:31.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/1271209474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f[67536]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f[67536]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mds[67540]: -- 172.21.15.203:0/3822890791 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d01a71a40 con 0x555d01a44c00 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f[67536]: starting mds.f at 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i[68153]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i[68153]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mds[68157]: -- 172.21.15.203:0/1974263357 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5563ab57c4e0 con 0x5563ab552800 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mds[68157]: -- 172.21.15.203:0/1974263357 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5563ab57fa40 con 0x5563ab552800 2024-03-18T01:48:31.893 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i[68153]: starting mds.i at 2024-03-18T01:48:31.894 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l[67226]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:31.894 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l[67226]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:31.894 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mds[67230]: -- 172.21.15.203:0/3270162123 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d5b62ba40 con 0x555d5b5fe800 2024-03-18T01:48:31.894 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l[67226]: starting mds.l at 2024-03-18T01:48:31.894 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c[67845]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:31.894 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c[67845]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:31.894 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-mds[67849]: -- 172.21.15.203:0/4033515691 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55907d061a40 con 0x55907d034800 2024-03-18T01:48:31.894 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:48:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c[67845]: starting mds.c at 2024-03-18T01:48:32.029 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.029+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3238062480 con 0x7f3238104bb0 2024-03-18T01:48:32.052 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g[73485]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:32.052 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g[73485]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:32.052 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-mds[73494]: -- 172.21.15.7:0/3839952357 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f788ed3a40 con 0x55f788ea6c00 2024-03-18T01:48:32.052 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g[73485]: starting mds.g at 2024-03-18T01:48:32.052 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j[72489]: ignoring --setuser ceph since I am not root 2024-03-18T01:48:32.053 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j[72489]: ignoring --setgroup ceph since I am not root 2024-03-18T01:48:32.053 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-mds[72493]: -- 172.21.15.7:0/184620369 <== mon.2 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5617d18a3a40 con 0x5617d1876c00 2024-03-18T01:48:32.053 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:48:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j[72489]: starting mds.j at 2024-03-18T01:48:32.540 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.540+0000 7f322f400640 1 -- 172.21.15.7:0/3984199896 <== mon.2 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v153) ==== 104+0+0 (secure 0 0 0) 0x7f32200c0f20 con 0x7f3238104bb0 2024-03-18T01:48:32.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.543+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f321c0a2880 msgr2=0x7f321c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:32.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.543+0000 7f323d000640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f321c0a2880 0x7f321c0a4d40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f322800f7d0 tx=0x7f3228009500 comp rx=0 tx=0).stop 2024-03-18T01:48:32.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.543+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 msgr2=0x7f323819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:32.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.543+0000 7f323d000640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 0x7f323819a280 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3220009230 tx=0x7f322002c660 comp rx=0 tx=0).stop 2024-03-18T01:48:32.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.544+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 shutdown_connections 2024-03-18T01:48:32.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.544+0000 7f323d000640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32381074e0 0x7f32381a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.544+0000 7f323d000640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3238104bb0 0x7f323819a280 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.544 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.544+0000 7f323d000640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f321c0a2880 0x7f321c0a4d40 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.544+0000 7f323d000640 1 --2- 172.21.15.7:0/3984199896 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3238102280 0x7f3238199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.544+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 >> 172.21.15.7:0/3984199896 conn(0x7f32380fbc70 msgr2=0x7f32381019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:32.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.544+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 shutdown_connections 2024-03-18T01:48:32.545 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.545+0000 7f323d000640 1 -- 172.21.15.7:0/3984199896 wait complete. 2024-03-18T01:48:32.564 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T01:48:32.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 -- 172.21.15.7:0/338387640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3714103a20 msgr2=0x7f3714103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:32.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 --2- 172.21.15.7:0/338387640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3714103a20 0x7f3714103e40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3704009860 tx=0x7f370402bb70 comp rx=0 tx=0).stop 2024-03-18T01:48:32.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 -- 172.21.15.7:0/338387640 shutdown_connections 2024-03-18T01:48:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 --2- 172.21.15.7:0/338387640 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f37141055d0 0x7f371410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 --2- 172.21.15.7:0/338387640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3714104c10 0x7f3714105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 --2- 172.21.15.7:0/338387640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3714103a20 0x7f3714103e40 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 -- 172.21.15.7:0/338387640 >> 172.21.15.7:0/338387640 conn(0x7f37140fef90 msgr2=0x7f37141013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 -- 172.21.15.7:0/338387640 shutdown_connections 2024-03-18T01:48:32.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.754+0000 7f3719c00640 1 -- 172.21.15.7:0/338387640 wait complete. 2024-03-18T01:48:32.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.755+0000 7f3719c00640 1 Processor -- start 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.755+0000 7f3719c00640 1 -- start start 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3719c00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3714103a20 0x7f3714199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3719c00640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3714104c10 0x7f371419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3719c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37141055d0 0x7f37141a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3719c00640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f371410f690 con 0x7f3714104c10 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3719c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f371410f510 con 0x7f37141055d0 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3719c00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f371410f810 con 0x7f3714103a20 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3714104c10 0x7f371419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712000640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3714104c10 0x7f371419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54750/0 (socket says 172.21.15.7:54750) 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712000640 1 -- 172.21.15.7:0/2301434992 learned_addr learned my addr 172.21.15.7:0/2301434992 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712a00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3714103a20 0x7f3714199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:32.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3713400640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37141055d0 0x7f37141a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712000640 1 -- 172.21.15.7:0/2301434992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3714103a20 msgr2=0x7f3714199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712000640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3714103a20 0x7f3714199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712000640 1 -- 172.21.15.7:0/2301434992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37141055d0 msgr2=0x7f37141a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712000640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37141055d0 0x7f37141a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712000640 1 -- 172.21.15.7:0/2301434992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37141a1980 con 0x7f3714104c10 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.756+0000 7f3712a00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3714103a20 0x7f3714199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.757+0000 7f3712000640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3714104c10 0x7f371419a2a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f370000e9f0 tx=0x7f370000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.757+0000 7f370b400640 1 -- 172.21.15.7:0/2301434992 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370000cd80 con 0x7f3714104c10 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.757+0000 7f370b400640 1 -- 172.21.15.7:0/2301434992 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3700005500 con 0x7f3714104c10 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.757+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37141a1c70 con 0x7f3714104c10 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.757+0000 7f370b400640 1 -- 172.21.15.7:0/2301434992 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700010690 con 0x7f3714104c10 2024-03-18T01:48:32.759 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.757+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37141a2170 con 0x7f3714104c10 2024-03-18T01:48:32.760 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.758+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371410adc0 con 0x7f3714104c10 2024-03-18T01:48:32.760 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3685910379' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: pgmap v729: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 614 B/s wr, 10 op/s 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3984199896' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:48:32.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:32 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:48:32.761 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.761+0000 7f370b400640 1 -- 172.21.15.7:0/2301434992 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f37000046c0 con 0x7f3714104c10 2024-03-18T01:48:32.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.762+0000 7f370b400640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f36f80a2800 0x7f36f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:32.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.763+0000 7f370b400640 1 -- 172.21.15.7:0/2301434992 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9522+0+0 (secure 0 0 0) 0x7f37000cf350 con 0x7f3714104c10 2024-03-18T01:48:32.763 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.763+0000 7f3712a00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f36f80a2800 0x7f36f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:32.765 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.765+0000 7f3712a00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f36f80a2800 0x7f36f80a4cc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f37040349e0 tx=0x7f370402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:32.768 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.767+0000 7f370b400640 1 -- 172.21.15.7:0/2301434992 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f370009bee0 con 0x7f3714104c10 2024-03-18T01:48:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3685910379' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:48:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:48:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T01:48:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:48:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T01:48:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T01:48:32.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T01:48:32.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: pgmap v729: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 614 B/s wr, 10 op/s 2024-03-18T01:48:32.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3984199896' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:48:32.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:32 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:48:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3685910379' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:48:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:48:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T01:48:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:48:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T01:48:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T01:48:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T01:48:32.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: pgmap v729: 129 pgs: 129 active+clean; 27 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 614 B/s wr, 10 op/s 2024-03-18T01:48:32.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3984199896' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:48:32.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:32 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:48:32.966 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:32.965+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f371402d1b0 con 0x7f3714104c10 2024-03-18T01:48:33.541 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.540+0000 7f370b400640 1 -- 172.21.15.7:0/2301434992 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) ==== 181+0+0 (secure 0 0 0) 0x7f370009b7e0 con 0x7f3714104c10 2024-03-18T01:48:33.541 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_metadata' removed 2024-03-18T01:48:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f36f80a2800 msgr2=0x7f36f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f36f80a2800 0x7f36f80a4cc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f37040349e0 tx=0x7f370402e040 comp rx=0 tx=0).stop 2024-03-18T01:48:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3714104c10 msgr2=0x7f371419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:33.546 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3714104c10 0x7f371419a2a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f370000e9f0 tx=0x7f370000edf0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 shutdown_connections 2024-03-18T01:48:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37141055d0 0x7f37141a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f36f80a2800 0x7f36f80a4cc0 secure :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f37040349e0 tx=0x7f370402e040 comp rx=0 tx=0).stop 2024-03-18T01:48:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f3714104c10 0x7f371419a2a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 --2- 172.21.15.7:0/2301434992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3714103a20 0x7f3714199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.547 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.546+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 >> 172.21.15.7:0/2301434992 conn(0x7f37140fef90 msgr2=0x7f3714107f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:33.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.547+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 shutdown_connections 2024-03-18T01:48:33.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.547+0000 7f3719c00640 1 -- 172.21.15.7:0/2301434992 wait complete. 2024-03-18T01:48:33.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T01:48:33.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T01:48:33.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T01:48:33.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: Cluster is now healthy 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6833/2378910926,v1:172.21.15.46:6835/2378910926] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6836/62176792,v1:172.21.15.46:6838/62176792] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6837/89724151,v1:172.21.15.46:6839/89724151] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.46:6832/3903541788,v1:172.21.15.46:6834/3903541788] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6835/3355062267,v1:172.21.15.203:6839/3355062267] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6833/1254040073,v1:172.21.15.203:6836/1254040073] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6834/3119482568,v1:172.21.15.203:6837/3119482568] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.203:6832/57561908,v1:172.21.15.203:6838/57561908] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6834/3439330420,v1:172.21.15.7:6835/3439330420] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: mds.? [v2:172.21.15.7:6838/1889183623,v1:172.21.15.7:6839/1889183623] up:boot 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: fsmap 12 up:standby 2024-03-18T01:48:33.770 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:48:33.771 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:33 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2301434992' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.799+0000 7f479b200640 1 -- 172.21.15.7:0/1688234817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941031d0 msgr2=0x7f4794109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.799+0000 7f479b200640 1 --2- 172.21.15.7:0/1688234817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941031d0 0x7f4794109a60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f4790009ca0 tx=0x7f479002c100 comp rx=0 tx=0).stop 2024-03-18T01:48:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.799+0000 7f479b200640 1 -- 172.21.15.7:0/1688234817 shutdown_connections 2024-03-18T01:48:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.799+0000 7f479b200640 1 --2- 172.21.15.7:0/1688234817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47941031d0 0x7f4794109a60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.799+0000 7f479b200640 1 --2- 172.21.15.7:0/1688234817 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f4794102810 0x7f4794102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.799+0000 7f479b200640 1 --2- 172.21.15.7:0/1688234817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4794101620 0x7f4794101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.799+0000 7f479b200640 1 -- 172.21.15.7:0/1688234817 >> 172.21.15.7:0/1688234817 conn(0x7f47940fcb90 msgr2=0x7f47940fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:33.800 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.799+0000 7f479b200640 1 -- 172.21.15.7:0/1688234817 shutdown_connections 2024-03-18T01:48:33.801 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.800+0000 7f479b200640 1 -- 172.21.15.7:0/1688234817 wait complete. 2024-03-18T01:48:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.800+0000 7f479b200640 1 Processor -- start 2024-03-18T01:48:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f479b200640 1 -- start start 2024-03-18T01:48:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f479b200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4794101620 0x7f4794199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f479b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4794102810 0x7f479419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f479b200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47941031d0 0x7f47941a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f479b200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7f479410d2a0 con 0x7f47941031d0 2024-03-18T01:48:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f479b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f479410d120 con 0x7f4794102810 2024-03-18T01:48:33.802 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f479b200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f479410d420 con 0x7f4794101620 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f4799e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4794101620 0x7f4794199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f4799e00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4794101620 0x7f4794199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.7:51098/0 (socket says 172.21.15.7:51098) 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f4799e00640 1 -- 172.21.15.7:0/2757593929 learned_addr learned my addr 172.21.15.7:0/2757593929 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f479a800640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47941031d0 0x7f47941a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.801+0000 7f4799400640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4794102810 0x7f479419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f4799e00640 1 -- 172.21.15.7:0/2757593929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4794102810 msgr2=0x7f479419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f4799e00640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4794102810 0x7f479419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f4799e00640 1 -- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47941031d0 msgr2=0x7f47941a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:33.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f4799e00640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47941031d0 0x7f47941a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: Cluster is now healthy 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6833/2378910926,v1:172.21.15.46:6835/2378910926] up:boot 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6836/62176792,v1:172.21.15.46:6838/62176792] up:boot 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6837/89724151,v1:172.21.15.46:6839/89724151] up:boot 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.46:6832/3903541788,v1:172.21.15.46:6834/3903541788] up:boot 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6835/3355062267,v1:172.21.15.203:6839/3355062267] up:boot 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6833/1254040073,v1:172.21.15.203:6836/1254040073] up:boot 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6834/3119482568,v1:172.21.15.203:6837/3119482568] up:boot 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.203:6832/57561908,v1:172.21.15.203:6838/57561908] up:boot 2024-03-18T01:48:33.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6834/3439330420,v1:172.21.15.7:6835/3439330420] up:boot 2024-03-18T01:48:33.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: mds.? [v2:172.21.15.7:6838/1889183623,v1:172.21.15.7:6839/1889183623] up:boot 2024-03-18T01:48:33.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: fsmap 12 up:standby 2024-03-18T01:48:33.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:48:33.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:48:33.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:48:33.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:48:33.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:48:33.805 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T01:48:33.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:48:33.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:48:33.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:48:33.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:48:33.806 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:33 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2301434992' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f4799e00640 1 -- 172.21.15.7:0/2757593929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47941a1720 con 0x7f4794101620 2024-03-18T01:48:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f479a800640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47941031d0 0x7f47941a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:48:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f4799e00640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4794101620 0x7f4794199b00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f478800e9c0 tx=0x7f478800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f478ea00640 1 -- 172.21.15.7:0/2757593929 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478800cd80 con 0x7f4794101620 2024-03-18T01:48:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f478ea00640 1 -- 172.21.15.7:0/2757593929 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47880035f0 con 0x7f4794101620 2024-03-18T01:48:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47941a1a10 con 0x7f4794101620 2024-03-18T01:48:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.802+0000 7f478ea00640 1 -- 172.21.15.7:0/2757593929 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788010690 con 0x7f4794101620 2024-03-18T01:48:33.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.803+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47941a1f10 con 0x7f4794101620 2024-03-18T01:48:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.804+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47941a21a0 con 0x7f4794101620 2024-03-18T01:48:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.806+0000 7f478ea00640 1 -- 172.21.15.7:0/2757593929 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7f47880037a0 con 0x7f4794101620 2024-03-18T01:48:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.807+0000 7f478ea00640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f477c0a2800 0x7f477c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.807+0000 7f478ea00640 1 -- 172.21.15.7:0/2757593929 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9026+0+0 (secure 0 0 0) 0x7f47880cf170 con 0x7f4794101620 2024-03-18T01:48:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.807+0000 7f4799400640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f477c0a2800 0x7f477c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:33.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.808+0000 7f4799400640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f477c0a2800 0x7f477c0a4cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f479419b010 tx=0x7f4784005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:33.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:33.811+0000 7f478ea00640 1 -- 172.21.15.7:0/2757593929 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f478809be90 con 0x7f4794101620 2024-03-18T01:48:33.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: Cluster is now healthy 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6833/2378910926,v1:172.21.15.46:6835/2378910926] up:boot 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6836/62176792,v1:172.21.15.46:6838/62176792] up:boot 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6837/89724151,v1:172.21.15.46:6839/89724151] up:boot 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.46:6832/3903541788,v1:172.21.15.46:6834/3903541788] up:boot 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6835/3355062267,v1:172.21.15.203:6839/3355062267] up:boot 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6833/1254040073,v1:172.21.15.203:6836/1254040073] up:boot 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6834/3119482568,v1:172.21.15.203:6837/3119482568] up:boot 2024-03-18T01:48:33.893 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.203:6832/57561908,v1:172.21.15.203:6838/57561908] up:boot 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6834/3439330420,v1:172.21.15.7:6835/3439330420] up:boot 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: mds.? [v2:172.21.15.7:6838/1889183623,v1:172.21.15.7:6839/1889183623] up:boot 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: fsmap 12 up:standby 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='mgr.14152 172.21.15.7:0/4030290999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:48:33.894 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:33 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2301434992' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:34.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.014+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f47941a2910 con 0x7f4794101620 2024-03-18T01:48:34.548 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.548+0000 7f478ea00640 1 -- 172.21.15.7:0/2757593929 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) ==== 169+0+0 (secure 0 0 0) 0x7f478809b790 con 0x7f4794101620 2024-03-18T01:48:34.549 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data' removed 2024-03-18T01:48:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.551+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f477c0a2800 msgr2=0x7f477c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.551+0000 7f479b200640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f477c0a2800 0x7f477c0a4cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f479419b010 tx=0x7f4784005f90 comp rx=0 tx=0).stop 2024-03-18T01:48:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.551+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4794101620 msgr2=0x7f4794199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:34.552 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.551+0000 7f479b200640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4794101620 0x7f4794199b00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f478800e9c0 tx=0x7f478800ee80 comp rx=0 tx=0).stop 2024-03-18T01:48:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.552+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 shutdown_connections 2024-03-18T01:48:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.552+0000 7f479b200640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7f477c0a2800 0x7f477c0a4cc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.552+0000 7f479b200640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7f47941031d0 0x7f47941a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.552+0000 7f479b200640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4794102810 0x7f479419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.552+0000 7f479b200640 1 --2- 172.21.15.7:0/2757593929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4794101620 0x7f4794199b00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.552+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 >> 172.21.15.7:0/2757593929 conn(0x7f47940fcb90 msgr2=0x7f4794073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.552+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 shutdown_connections 2024-03-18T01:48:34.553 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.552+0000 7f479b200640 1 -- 172.21.15.7:0/2757593929 wait complete. 2024-03-18T01:48:34.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-18T01:48:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 -- 172.21.15.7:0/4169817995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc107c40 msgr2=0x7fdccc108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 --2- 172.21.15.7:0/4169817995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc107c40 0x7fdccc108060 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc009be0 tx=0x7fdcbc02bef0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 -- 172.21.15.7:0/4169817995 shutdown_connections 2024-03-18T01:48:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 --2- 172.21.15.7:0/4169817995 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdccc1097f0 0x7fdccc110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 --2- 172.21.15.7:0/4169817995 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdccc108e30 0x7fdccc1092b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 --2- 172.21.15.7:0/4169817995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc107c40 0x7fdccc108060 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.803 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 -- 172.21.15.7:0/4169817995 >> 172.21.15.7:0/4169817995 conn(0x7fdccc076bd0 msgr2=0x7fdccc079010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 -- 172.21.15.7:0/4169817995 shutdown_connections 2024-03-18T01:48:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.802+0000 7fdcd1200640 1 -- 172.21.15.7:0/4169817995 wait complete. 2024-03-18T01:48:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.804+0000 7fdcd1200640 1 Processor -- start 2024-03-18T01:48:34.804 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.804+0000 7fdcd1200640 1 -- start start 2024-03-18T01:48:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.804+0000 7fdcd1200640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdccc107c40 0x7fdccc19df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.804+0000 7fdcd1200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdccc108e30 0x7fdccc19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.804+0000 7fdcd1200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc1097f0 0x7fdccc1a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.804+0000 7fdcd1200640 1 -- --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc113910 con 0x7fdccc107c40 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.804+0000 7fdcd1200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc113790 con 0x7fdccc108e30 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.804+0000 7fdcd1200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc113a90 con 0x7fdccc1097f0 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdccc107c40 0x7fdccc19df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 --2- >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdccc107c40 0x7fdccc19df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.7:3300/0 says I am v2:172.21.15.7:54788/0 (socket says 172.21.15.7:54788) 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 -- 172.21.15.7:0/3402046099 learned_addr learned my addr 172.21.15.7:0/3402046099 (peer_addr_for_me v2:172.21.15.7:0/0) 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdcc2a00640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdccc108e30 0x7fdccc19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccbe00640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc1097f0 0x7fdccc1a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 -- 172.21.15.7:0/3402046099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc1097f0 msgr2=0x7fdccc1a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:34.806 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc1097f0 0x7fdccc1a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 -- 172.21.15.7:0/3402046099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdccc108e30 msgr2=0x7fdccc19e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdccc108e30 0x7fdccc19e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 -- 172.21.15.7:0/3402046099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdccc1a5ba0 con 0x7fdccc107c40 2024-03-18T01:48:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccbe00640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc1097f0 0x7fdccc1a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:48:34.807 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.805+0000 7fdccb400640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdccc107c40 0x7fdccc19df80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc02c500 tx=0x7fdcbc033580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.806+0000 7fdcc8c00640 1 -- 172.21.15.7:0/3402046099 <== mon.0 v2:172.21.15.7:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc043020 con 0x7fdccc107c40 2024-03-18T01:48:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.806+0000 7fdcc8c00640 1 -- 172.21.15.7:0/3402046099 <== mon.0 v2:172.21.15.7:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcbc003e50 con 0x7fdccc107c40 2024-03-18T01:48:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.806+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdccc1a5e30 con 0x7fdccc107c40 2024-03-18T01:48:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.806+0000 7fdcc8c00640 1 -- 172.21.15.7:0/3402046099 <== mon.0 v2:172.21.15.7:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc034690 con 0x7fdccc107c40 2024-03-18T01:48:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.806+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdccc1a6290 con 0x7fdccc107c40 2024-03-18T01:48:34.808 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.807+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdccc10ef60 con 0x7fdccc107c40 2024-03-18T01:48:34.810 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.810+0000 7fdcc8c00640 1 -- 172.21.15.7:0/3402046099 <== mon.0 v2:172.21.15.7:3300/0 4 ==== mgrmap(e 18) ==== 137289+0+0 (secure 0 0 0) 0x7fdcbc046020 con 0x7fdccc107c40 2024-03-18T01:48:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.811+0000 7fdcc8c00640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdcb00a2800 0x7fdcb00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:48:34.811 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.811+0000 7fdcc8c00640 1 -- 172.21.15.7:0/3402046099 <== mon.0 v2:172.21.15.7:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8496+0+0 (secure 0 0 0) 0x7fdcbc0f13a0 con 0x7fdccc107c40 2024-03-18T01:48:34.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.811+0000 7fdcc2a00640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdcb00a2800 0x7fdcb00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:48:34.812 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.812+0000 7fdcc2a00640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdcb00a2800 0x7fdcb00a4cc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdccc19f490 tx=0x7fdcb4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:48:34.816 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:34.816+0000 7fdcc8c00640 1 -- 172.21.15.7:0/3402046099 <== mon.0 v2:172.21.15.7:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdcbc04d020 con 0x7fdccc107c40 2024-03-18T01:48:34.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:34 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2301434992' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:34.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:34 smithi203 ceph-mon[36345]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T01:48:34.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:34 smithi203 ceph-mon[36345]: pgmap v731: 97 pgs: 97 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:48:34.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:34 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/2757593929' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:34.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:34 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:34 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2301434992' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:34 smithi046 ceph-mon[36108]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T01:48:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:34 smithi046 ceph-mon[36108]: pgmap v731: 97 pgs: 97 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:48:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:34 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/2757593929' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:35.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:34 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:34 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2301434992' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:34 smithi007 ceph-mon[31996]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T01:48:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:34 smithi007 ceph-mon[31996]: pgmap v731: 97 pgs: 97 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:48:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:34 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/2757593929' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:34 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:35.096 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.096+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 --> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdccc02d1b0 con 0x7fdccc107c40 2024-03-18T01:48:35.555 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.555+0000 7fdcc8c00640 1 -- 172.21.15.7:0/3402046099 <== mon.0 v2:172.21.15.7:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v95) ==== 178+0+0 (secure 0 0 0) 0x7fdcbc0be300 con 0x7fdccc107c40 2024-03-18T01:48:35.556 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'cephfs_data_ec' removed 2024-03-18T01:48:35.559 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.558+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdcb00a2800 msgr2=0x7fdcb00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdcb00a2800 0x7fdcb00a4cc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdccc19f490 tx=0x7fdcb4009260 comp rx=0 tx=0).stop 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdccc107c40 msgr2=0x7fdccc19df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdccc107c40 0x7fdccc19df80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc02c500 tx=0x7fdcbc033580 comp rx=0 tx=0).stop 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 shutdown_connections 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdccc1097f0 0x7fdccc1a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdccc108e30 0x7fdccc19e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:6800/2743163723,v1:172.21.15.7:6801/2743163723] conn(0x7fdcb00a2800 0x7fdcb00a4cc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 --2- 172.21.15.7:0/3402046099 >> [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] conn(0x7fdccc107c40 0x7fdccc19df80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 >> 172.21.15.7:0/3402046099 conn(0x7fdccc076bd0 msgr2=0x7fdccc10c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 shutdown_connections 2024-03-18T01:48:35.560 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-18T01:48:35.559+0000 7fdcd1200640 1 -- 172.21.15.7:0/3402046099 wait complete. 2024-03-18T01:48:35.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:48:35.706 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:48:35.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:48:35.828 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:35 smithi203 ceph-mon[36345]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:35.828 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:35 smithi203 ceph-mon[36345]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T01:48:35.828 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:35 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3402046099' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:35.841 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T01:48:35.841 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:48:35.896 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:48:35.923 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:48:35.946 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T01:48:35.947 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T01:48:35.947 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a 2024-03-18T01:48:35.954 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:35 smithi007 ceph-mon[31996]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:35.954 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:35 smithi007 ceph-mon[31996]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T01:48:35.954 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:35 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3402046099' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:36.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:35 smithi046 ceph-mon[36108]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:36.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:35 smithi046 ceph-mon[36108]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T01:48:36.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:35 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3402046099' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:48:36.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:36 smithi007 systemd[1]: Stopping Ceph mon.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:48:36.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:36 smithi007 ceph-mon[31996]: from='client.? 172.21.15.7:0/3402046099' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:36.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:36 smithi007 ceph-mon[31996]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T01:48:36.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:36 smithi007 ceph-mon[31996]: pgmap v734: 1 pgs: 1 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:48:36.828 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:36 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a[31992]: 2024-03-18T01:48:36.742+0000 7f7534600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:48:36.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:36 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a[31992]: 2024-03-18T01:48:36.742+0000 7f7534600700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T01:48:36.891 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:36 smithi203 ceph-mon[36345]: from='client.? 172.21.15.7:0/3402046099' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:36.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:36 smithi203 ceph-mon[36345]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T01:48:36.892 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:36 smithi203 ceph-mon[36345]: pgmap v734: 1 pgs: 1 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:48:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:36 smithi046 ceph-mon[36108]: from='client.? 172.21.15.7:0/3402046099' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:48:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:36 smithi046 ceph-mon[36108]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T01:48:37.019 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:36 smithi046 ceph-mon[36108]: pgmap v734: 1 pgs: 1 active+clean; 577 KiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T01:48:37.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:36 smithi007 podman[98165]: 2024-03-18 01:48:36.827953781 +0000 UTC m=+0.273303946 container died f5ed85e015f25a9911abb223ea01342bc4eac987d8cdb5e31bbedb7b4e2d6399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T01:48:37.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:37 smithi007 podman[98165]: 2024-03-18 01:48:37.591900967 +0000 UTC m=+1.037251160 container cleanup f5ed85e015f25a9911abb223ea01342bc4eac987d8cdb5e31bbedb7b4e2d6399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:48:37.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:37 smithi007 bash[98165]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a 2024-03-18T01:48:38.265 INFO:journalctl@ceph.mon.a.smithi007.stdout:Mar 18 01:48:37 smithi007 podman[98175]: 2024-03-18 01:48:37.826795345 +0000 UTC m=+0.980247131 container remove f5ed85e015f25a9911abb223ea01342bc4eac987d8cdb5e31bbedb7b4e2d6399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:48:38.292 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T01:48:38.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:48:38.436 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T01:48:38.436 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T01:48:38.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.b 2024-03-18T01:48:38.769 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:38 smithi046 systemd[1]: Stopping Ceph mon.b for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:48:39.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:38 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-b[36104]: 2024-03-18T01:48:38.979+0000 7fb2e99a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:48:39.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:38 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-b[36104]: 2024-03-18T01:48:38.979+0000 7fb2e99a5700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T01:48:39.270 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:39 smithi046 podman[71310]: 2024-03-18 01:48:39.031302984 +0000 UTC m=+0.357014367 container died f6e73844f7920847edf806c89a9b7d470a56a39d53926d87c3aca977318ab1b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:48:39.751 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:39 smithi046 podman[71310]: 2024-03-18 01:48:39.438847623 +0000 UTC m=+0.764558995 container cleanup f6e73844f7920847edf806c89a9b7d470a56a39d53926d87c3aca977318ab1b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-b, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:48:39.751 INFO:journalctl@ceph.mon.b.smithi046.stdout:Mar 18 01:48:39 smithi046 bash[71310]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-b 2024-03-18T01:48:39.926 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T01:48:39.991 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:48:39.991 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T01:48:39.991 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T01:48:39.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.c 2024-03-18T01:48:40.382 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:40 smithi203 systemd[1]: Stopping Ceph mon.c for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:48:40.641 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:40 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-c[36319]: 2024-03-18T01:48:40.381+0000 7f6745171700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:48:40.642 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:40 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-c[36319]: 2024-03-18T01:48:40.381+0000 7f6745171700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T01:48:41.059 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:40 smithi203 podman[72650]: 2024-03-18 01:48:40.657825707 +0000 UTC m=+0.405596595 container died 459e01704c44f6672e945efba134ee3847269d0c96721f8194c861dc0c8d6ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T01:48:41.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:41 smithi203 podman[72650]: 2024-03-18 01:48:41.183348207 +0000 UTC m=+0.931119072 container cleanup 459e01704c44f6672e945efba134ee3847269d0c96721f8194c861dc0c8d6ee9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T01:48:41.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Mar 18 01:48:41 smithi203 bash[72650]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mon-c 2024-03-18T01:48:41.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T01:48:41.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:48:41.667 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T01:48:41.668 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T01:48:41.668 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.x 2024-03-18T01:48:42.052 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:48:41 smithi007 systemd[1]: Stopping Ceph mgr.x for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:48:42.497 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:48:42 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:48:42.395+0000 7faa48200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:48:42.497 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:48:42 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x[32237]: 2024-03-18T01:48:42.395+0000 7faa48200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-18T01:48:42.802 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:48:42 smithi007 podman[98279]: 2024-03-18 01:48:42.501639258 +0000 UTC m=+0.356280656 container died 7d71c73ad82fbb3a7186dae6d09c5ba04e885c4554ef15603d73bbf7fd395afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T01:48:43.250 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:48:43 smithi007 podman[98279]: 2024-03-18 01:48:43.038962358 +0000 UTC m=+0.893603773 container cleanup 7d71c73ad82fbb3a7186dae6d09c5ba04e885c4554ef15603d73bbf7fd395afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T01:48:43.250 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:48:43 smithi007 bash[98279]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x 2024-03-18T01:48:43.552 INFO:journalctl@ceph.mgr.x.smithi007.stdout:Mar 18 01:48:43 smithi007 podman[98293]: 2024-03-18 01:48:43.248731542 +0000 UTC m=+0.734753183 container remove 7d71c73ad82fbb3a7186dae6d09c5ba04e885c4554ef15603d73bbf7fd395afe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-18T01:48:43.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T01:48:43.905 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:48:43.906 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T01:48:43.906 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T01:48:43.906 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.y 2024-03-18T01:48:44.247 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:48:43 smithi046 systemd[1]: Stopping Ceph mgr.y for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:48:44.519 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:48:44 smithi046 podman[71433]: 2024-03-18 01:48:44.247393026 +0000 UTC m=+0.134481822 container died 532e3a82d13e75bc91874cc0cee488c04c6e431e7b1862e3b9a49bb35fa77815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:48:44.954 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:48:44 smithi046 podman[71433]: 2024-03-18 01:48:44.652090062 +0000 UTC m=+0.539178849 container cleanup 532e3a82d13e75bc91874cc0cee488c04c6e431e7b1862e3b9a49bb35fa77815 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-18T01:48:44.954 INFO:journalctl@ceph.mgr.y.smithi046.stdout:Mar 18 01:48:44 smithi046 bash[71433]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-y 2024-03-18T01:48:45.131 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T01:48:45.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:48:45.172 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T01:48:45.173 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T01:48:45.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.z 2024-03-18T01:48:45.509 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:48:45 smithi203 systemd[1]: Stopping Ceph mgr.z for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:48:45.799 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:48:45 smithi203 podman[72773]: 2024-03-18 01:48:45.509488951 +0000 UTC m=+0.149755551 container died 94ad95bb8a37c6f6356fa41f052828780b4b61637ca2cd8cc4791eec03128b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-18T01:48:46.142 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:48:45 smithi203 podman[72773]: 2024-03-18 01:48:45.906742144 +0000 UTC m=+0.547008730 container cleanup 94ad95bb8a37c6f6356fa41f052828780b4b61637ca2cd8cc4791eec03128b88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:48:46.142 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Mar 18 01:48:45 smithi203 bash[72773]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mgr-z 2024-03-18T01:48:46.336 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T01:48:46.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:48:46.377 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T01:48:46.377 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T01:48:46.377 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.0 2024-03-18T01:48:46.802 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:46 smithi007 systemd[1]: Stopping Ceph osd.0 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:48:47.302 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:46 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0[46887]: 2024-03-18T01:48:46.990+0000 7f4a32e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:48:47.303 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:46 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0[46887]: 2024-03-18T01:48:46.990+0000 7f4a32e00700 -1 osd.0 95 *** Got signal Terminated *** 2024-03-18T01:48:47.303 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:46 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0[46887]: 2024-03-18T01:48:46.990+0000 7f4a32e00700 -1 osd.0 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:48:52.802 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:52 smithi007 podman[98399]: 2024-03-18 01:48:52.644334615 +0000 UTC m=+5.794076619 container died 2cd58b43030517348df5ac6f6f029f65aac02ceee9ca997be7165802360d09ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T01:48:53.479 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:53 smithi007 podman[98399]: 2024-03-18 01:48:53.201297685 +0000 UTC m=+6.351039729 container cleanup 2cd58b43030517348df5ac6f6f029f65aac02ceee9ca997be7165802360d09ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:48:53.479 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:53 smithi007 bash[98399]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0 2024-03-18T01:48:53.802 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:53 smithi007 podman[98418]: 2024-03-18 01:48:53.477800665 +0000 UTC m=+0.812039174 container remove 2cd58b43030517348df5ac6f6f029f65aac02ceee9ca997be7165802360d09ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:48:54.661 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:54 smithi007 podman[98507]: 2024-03-18 01:48:54.252214557 +0000 UTC m=+0.044077470 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:48:54.661 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:54 smithi007 podman[98507]: 2024-03-18 01:48:54.378759352 +0000 UTC m=+0.170622258 container create 9821e6ac1d8fce744cc7d664c0de061c37a5e1f6e267e173d0d59e184eca928c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:48:55.052 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:54 smithi007 podman[98507]: 2024-03-18 01:48:54.660118275 +0000 UTC m=+0.451981188 container init 9821e6ac1d8fce744cc7d664c0de061c37a5e1f6e267e173d0d59e184eca928c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-18T01:48:55.052 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:54 smithi007 podman[98507]: 2024-03-18 01:48:54.671426535 +0000 UTC m=+0.463289447 container start 9821e6ac1d8fce744cc7d664c0de061c37a5e1f6e267e173d0d59e184eca928c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-18T01:48:55.052 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:54 smithi007 podman[98507]: 2024-03-18 01:48:54.781242468 +0000 UTC m=+0.573105381 container attach 9821e6ac1d8fce744cc7d664c0de061c37a5e1f6e267e173d0d59e184eca928c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T01:48:55.803 INFO:journalctl@ceph.osd.0.smithi007.stdout:Mar 18 01:48:55 smithi007 podman[98507]: 2024-03-18 01:48:55.541224931 +0000 UTC m=+1.333087815 container died 9821e6ac1d8fce744cc7d664c0de061c37a5e1f6e267e173d0d59e184eca928c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-18T01:48:56.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T01:48:56.428 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:48:56.429 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T01:48:56.429 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T01:48:56.429 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.1 2024-03-18T01:48:56.751 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:48:56 smithi046 systemd[1]: Stopping Ceph osd.1 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:48:57.033 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:48:56 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1[40708]: 2024-03-18T01:48:56.748+0000 7faa9e112700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:48:57.033 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:48:56 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1[40708]: 2024-03-18T01:48:56.748+0000 7faa9e112700 -1 osd.1 95 *** Got signal Terminated *** 2024-03-18T01:48:57.033 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:48:56 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1[40708]: 2024-03-18T01:48:56.748+0000 7faa9e112700 -1 osd.1 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:49:02.519 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:02 smithi046 podman[71558]: 2024-03-18 01:49:02.243138831 +0000 UTC m=+5.612151264 container died 4b2ce3d6211950349e3087ca4e98c697e4dc9a14c07fc34a21f354b7b2c3dcf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:02.908 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:02 smithi046 podman[71558]: 2024-03-18 01:49:02.615372494 +0000 UTC m=+5.984384924 container cleanup 4b2ce3d6211950349e3087ca4e98c697e4dc9a14c07fc34a21f354b7b2c3dcf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T01:49:02.909 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:02 smithi046 bash[71558]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1 2024-03-18T01:49:03.269 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:02 smithi046 podman[71576]: 2024-03-18 01:49:02.925276827 +0000 UTC m=+0.681326238 container remove 4b2ce3d6211950349e3087ca4e98c697e4dc9a14c07fc34a21f354b7b2c3dcf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:03.532 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:03 smithi046 podman[71673]: 2024-03-18 01:49:03.18152612 +0000 UTC m=+0.020847823 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:49:03.533 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:03 smithi046 podman[71673]: 2024-03-18 01:49:03.284742944 +0000 UTC m=+0.124064630 container create 697b179b675ea14384c8a4d7f6d53624fa0709f75c6978ee0870951ac64bda60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-18T01:49:04.019 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:03 smithi046 podman[71673]: 2024-03-18 01:49:03.663025096 +0000 UTC m=+0.502346815 container init 697b179b675ea14384c8a4d7f6d53624fa0709f75c6978ee0870951ac64bda60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T01:49:04.019 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:03 smithi046 podman[71673]: 2024-03-18 01:49:03.666863991 +0000 UTC m=+0.506185681 container start 697b179b675ea14384c8a4d7f6d53624fa0709f75c6978ee0870951ac64bda60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:04.019 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:03 smithi046 podman[71673]: 2024-03-18 01:49:03.762448441 +0000 UTC m=+0.601770127 container attach 697b179b675ea14384c8a4d7f6d53624fa0709f75c6978ee0870951ac64bda60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T01:49:04.508 INFO:journalctl@ceph.osd.1.smithi046.stdout:Mar 18 01:49:04 smithi046 podman[71673]: 2024-03-18 01:49:04.206784988 +0000 UTC m=+1.046106682 container died 697b179b675ea14384c8a4d7f6d53624fa0709f75c6978ee0870951ac64bda60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:49:04.847 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T01:49:04.899 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:49:04.900 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T01:49:04.900 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T01:49:04.900 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.2 2024-03-18T01:49:05.229 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:04 smithi203 systemd[1]: Stopping Ceph osd.2 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:49:05.644 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:05 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2[41472]: 2024-03-18T01:49:05.228+0000 7ff8616f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:49:05.644 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:05 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2[41472]: 2024-03-18T01:49:05.228+0000 7ff8616f1700 -1 osd.2 95 *** Got signal Terminated *** 2024-03-18T01:49:05.644 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:05 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2[41472]: 2024-03-18T01:49:05.228+0000 7ff8616f1700 -1 osd.2 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:49:11.128 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:10 smithi203 podman[72910]: 2024-03-18 01:49:10.807754092 +0000 UTC m=+5.709227089 container died 0ce68efba2a1a1021a0f145754c04f0ec603b332a5b29991f418aabd1390549a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T01:49:11.392 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:11 smithi203 podman[72910]: 2024-03-18 01:49:11.128504857 +0000 UTC m=+6.029977837 container cleanup 0ce68efba2a1a1021a0f145754c04f0ec603b332a5b29991f418aabd1390549a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T01:49:11.392 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:11 smithi203 bash[72910]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2 2024-03-18T01:49:11.392 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:11 smithi203 podman[72927]: 2024-03-18 01:49:11.371870923 +0000 UTC m=+0.562188383 container remove 0ce68efba2a1a1021a0f145754c04f0ec603b332a5b29991f418aabd1390549a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T01:49:12.093 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:11 smithi203 podman[73017]: 2024-03-18 01:49:11.691173344 +0000 UTC m=+0.018028896 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:49:12.093 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:11 smithi203 podman[73017]: 2024-03-18 01:49:11.822957304 +0000 UTC m=+0.149812850 container create 4025b1d8724f7465c4b37a7830a1afea54ae08dccb5114974ac8527266856e11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:49:12.392 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:12 smithi203 podman[73017]: 2024-03-18 01:49:12.092788007 +0000 UTC m=+0.419643561 container init 4025b1d8724f7465c4b37a7830a1afea54ae08dccb5114974ac8527266856e11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-18T01:49:12.392 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:12 smithi203 podman[73017]: 2024-03-18 01:49:12.096207527 +0000 UTC m=+0.423063059 container start 4025b1d8724f7465c4b37a7830a1afea54ae08dccb5114974ac8527266856e11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:49:12.392 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:12 smithi203 podman[73017]: 2024-03-18 01:49:12.209880135 +0000 UTC m=+0.536735687 container attach 4025b1d8724f7465c4b37a7830a1afea54ae08dccb5114974ac8527266856e11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T01:49:12.892 INFO:journalctl@ceph.osd.2.smithi203.stdout:Mar 18 01:49:12 smithi203 podman[73017]: 2024-03-18 01:49:12.616014287 +0000 UTC m=+0.942869831 container died 4025b1d8724f7465c4b37a7830a1afea54ae08dccb5114974ac8527266856e11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T01:49:13.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T01:49:13.284 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:49:13.284 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T01:49:13.284 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T01:49:13.284 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.3 2024-03-18T01:49:13.674 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:13 smithi007 systemd[1]: Stopping Ceph osd.3 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:49:14.052 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:13 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3[53436]: 2024-03-18T01:49:13.893+0000 7fd271000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:49:14.052 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:13 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3[53436]: 2024-03-18T01:49:13.893+0000 7fd271000700 -1 osd.3 95 *** Got signal Terminated *** 2024-03-18T01:49:14.052 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:13 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3[53436]: 2024-03-18T01:49:13.893+0000 7fd271000700 -1 osd.3 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:49:20.052 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:19 smithi007 podman[98611]: 2024-03-18 01:49:19.778403856 +0000 UTC m=+6.031042831 container died 10f50ec8fd5ecb07a15bd17a4bdf213a2ae47cfa3e98b809fe245e6de78ae179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-18T01:49:20.472 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:20 smithi007 podman[98611]: 2024-03-18 01:49:20.253256513 +0000 UTC m=+6.505895505 container cleanup 10f50ec8fd5ecb07a15bd17a4bdf213a2ae47cfa3e98b809fe245e6de78ae179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:49:20.472 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:20 smithi007 bash[98611]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3 2024-03-18T01:49:20.802 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:20 smithi007 podman[98644]: 2024-03-18 01:49:20.471006489 +0000 UTC m=+0.670853175 container remove 10f50ec8fd5ecb07a15bd17a4bdf213a2ae47cfa3e98b809fe245e6de78ae179 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:49:21.552 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:21 smithi007 podman[98732]: 2024-03-18 01:49:21.167907271 +0000 UTC m=+0.037118209 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:49:21.553 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:21 smithi007 podman[98732]: 2024-03-18 01:49:21.305365964 +0000 UTC m=+0.174576939 container create 391884454dd91691b2d0c3d9ea578a6f1687053842eb06962554b36c89a36a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-18T01:49:22.052 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:21 smithi007 podman[98732]: 2024-03-18 01:49:21.609510015 +0000 UTC m=+0.478720997 container init 391884454dd91691b2d0c3d9ea578a6f1687053842eb06962554b36c89a36a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:49:22.053 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:21 smithi007 podman[98732]: 2024-03-18 01:49:21.620225878 +0000 UTC m=+0.489436860 container start 391884454dd91691b2d0c3d9ea578a6f1687053842eb06962554b36c89a36a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T01:49:22.053 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:21 smithi007 podman[98732]: 2024-03-18 01:49:21.708397147 +0000 UTC m=+0.577608162 container attach 391884454dd91691b2d0c3d9ea578a6f1687053842eb06962554b36c89a36a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:49:22.458 INFO:journalctl@ceph.osd.3.smithi007.stdout:Mar 18 01:49:22 smithi007 podman[98732]: 2024-03-18 01:49:22.069335144 +0000 UTC m=+0.938546169 container died 391884454dd91691b2d0c3d9ea578a6f1687053842eb06962554b36c89a36a4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:22.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T01:49:22.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:49:22.974 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T01:49:22.974 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T01:49:22.974 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.4 2024-03-18T01:49:23.269 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:23 smithi046 systemd[1]: Stopping Ceph osd.4 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:49:23.769 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:23 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4[47240]: 2024-03-18T01:49:23.284+0000 7f4ace403700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:49:23.769 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:23 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4[47240]: 2024-03-18T01:49:23.284+0000 7f4ace403700 -1 osd.4 95 *** Got signal Terminated *** 2024-03-18T01:49:23.769 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:23 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4[47240]: 2024-03-18T01:49:23.284+0000 7f4ace403700 -1 osd.4 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:49:29.012 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:28 smithi046 podman[71783]: 2024-03-18 01:49:28.747106935 +0000 UTC m=+5.567000985 container died fae8736942cff8fdec43a422f0f1df2e2cf019f49287ef787d88b427c2726e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T01:49:29.269 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:29 smithi046 podman[71783]: 2024-03-18 01:49:29.094177931 +0000 UTC m=+5.914071965 container cleanup fae8736942cff8fdec43a422f0f1df2e2cf019f49287ef787d88b427c2726e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-18T01:49:29.269 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:29 smithi046 bash[71783]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4 2024-03-18T01:49:29.721 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:29 smithi046 podman[71798]: 2024-03-18 01:49:29.312030819 +0000 UTC m=+0.560096189 container remove fae8736942cff8fdec43a422f0f1df2e2cf019f49287ef787d88b427c2726e06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-18T01:49:30.019 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:29 smithi046 podman[71893]: 2024-03-18 01:49:29.623809506 +0000 UTC m=+0.017913758 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:49:30.019 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:29 smithi046 podman[71893]: 2024-03-18 01:49:29.74613152 +0000 UTC m=+0.140235754 container create 0b4cb381c11af8e4ae59af05bf9f9aeb7959a35c6731181025e310d37ee3cef4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T01:49:30.366 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:30 smithi046 podman[71893]: 2024-03-18 01:49:30.066233406 +0000 UTC m=+0.460337658 container init 0b4cb381c11af8e4ae59af05bf9f9aeb7959a35c6731181025e310d37ee3cef4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-18T01:49:30.367 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:30 smithi046 podman[71893]: 2024-03-18 01:49:30.069792446 +0000 UTC m=+0.463896685 container start 0b4cb381c11af8e4ae59af05bf9f9aeb7959a35c6731181025e310d37ee3cef4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-18T01:49:30.367 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:30 smithi046 podman[71893]: 2024-03-18 01:49:30.148800963 +0000 UTC m=+0.542905204 container attach 0b4cb381c11af8e4ae59af05bf9f9aeb7959a35c6731181025e310d37ee3cef4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T01:49:30.735 INFO:journalctl@ceph.osd.4.smithi046.stdout:Mar 18 01:49:30 smithi046 podman[71893]: 2024-03-18 01:49:30.366246613 +0000 UTC m=+0.760350852 container died 0b4cb381c11af8e4ae59af05bf9f9aeb7959a35c6731181025e310d37ee3cef4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T01:49:31.059 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T01:49:31.107 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:49:31.107 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T01:49:31.107 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T01:49:31.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.5 2024-03-18T01:49:31.392 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:31 smithi203 systemd[1]: Stopping Ceph osd.5 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:49:31.892 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5[48455]: 2024-03-18T01:49:31.433+0000 7f1878f1b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:49:31.892 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5[48455]: 2024-03-18T01:49:31.433+0000 7f1878f1b700 -1 osd.5 95 *** Got signal Terminated *** 2024-03-18T01:49:31.892 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:31 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5[48455]: 2024-03-18T01:49:31.433+0000 7f1878f1b700 -1 osd.5 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:49:37.235 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:36 smithi203 podman[73139]: 2024-03-18 01:49:36.959776493 +0000 UTC m=+5.651500930 container died 483d7e25c69ae5ce1fed8b1d04f3f4e46abb99bdae1efc9ea63ffc728eb8a330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T01:49:37.640 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:37 smithi203 podman[73139]: 2024-03-18 01:49:37.325811281 +0000 UTC m=+6.017535719 container cleanup 483d7e25c69ae5ce1fed8b1d04f3f4e46abb99bdae1efc9ea63ffc728eb8a330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True) 2024-03-18T01:49:37.640 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:37 smithi203 bash[73139]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5 2024-03-18T01:49:37.892 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:37 smithi203 podman[73156]: 2024-03-18 01:49:37.639823744 +0000 UTC m=+0.684249337 container remove 483d7e25c69ae5ce1fed8b1d04f3f4e46abb99bdae1efc9ea63ffc728eb8a330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T01:49:38.257 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:37 smithi203 podman[73251]: 2024-03-18 01:49:37.895082559 +0000 UTC m=+0.017843724 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:49:38.257 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:38 smithi203 podman[73251]: 2024-03-18 01:49:38.037661993 +0000 UTC m=+0.160423187 container create 01774f417bee53fc25e43bcb390f7a94ef93f9c1153d5ed82f2161c1c4fed040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:38.257 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:38 smithi203 podman[73251]: 2024-03-18 01:49:38.25708899 +0000 UTC m=+0.379850140 container init 01774f417bee53fc25e43bcb390f7a94ef93f9c1153d5ed82f2161c1c4fed040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:49:38.551 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:38 smithi203 podman[73251]: 2024-03-18 01:49:38.260357976 +0000 UTC m=+0.383119118 container start 01774f417bee53fc25e43bcb390f7a94ef93f9c1153d5ed82f2161c1c4fed040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:38.551 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:38 smithi203 podman[73251]: 2024-03-18 01:49:38.332261903 +0000 UTC m=+0.455023064 container attach 01774f417bee53fc25e43bcb390f7a94ef93f9c1153d5ed82f2161c1c4fed040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:38.852 INFO:journalctl@ceph.osd.5.smithi203.stdout:Mar 18 01:49:38 smithi203 podman[73251]: 2024-03-18 01:49:38.550708534 +0000 UTC m=+0.673469697 container died 01774f417bee53fc25e43bcb390f7a94ef93f9c1153d5ed82f2161c1c4fed040 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:49:39.126 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T01:49:39.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:49:39.167 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T01:49:39.168 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T01:49:39.168 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.6 2024-03-18T01:49:39.552 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:39 smithi007 systemd[1]: Stopping Ceph osd.6 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:49:40.053 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:39 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-18T01:49:39.779+0000 7f16fc000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:49:40.053 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:39 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-18T01:49:39.779+0000 7f16fc000700 -1 osd.6 95 *** Got signal Terminated *** 2024-03-18T01:49:40.053 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:39 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6[60156]: 2024-03-18T01:49:39.779+0000 7f16fc000700 -1 osd.6 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:49:45.932 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:45 smithi007 podman[98837]: 2024-03-18 01:49:45.640374581 +0000 UTC m=+5.998277073 container died 9076a90d57fc34c953679fc19090c47b7bb34f551860e7ced1aaa3c7353f6e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:49:46.296 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:46 smithi007 podman[98837]: 2024-03-18 01:49:46.05664957 +0000 UTC m=+6.414552082 container cleanup 9076a90d57fc34c953679fc19090c47b7bb34f551860e7ced1aaa3c7353f6e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:46.296 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:46 smithi007 bash[98837]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6 2024-03-18T01:49:46.553 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:46 smithi007 podman[98856]: 2024-03-18 01:49:46.295359341 +0000 UTC m=+0.636432214 container remove 9076a90d57fc34c953679fc19090c47b7bb34f551860e7ced1aaa3c7353f6e09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T01:49:47.463 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:47 smithi007 podman[98945]: 2024-03-18 01:49:47.076003185 +0000 UTC m=+0.038555131 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:49:47.463 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:47 smithi007 podman[98945]: 2024-03-18 01:49:47.175168527 +0000 UTC m=+0.137720518 container create f2f2b00ff26f1499b7986096f042d05988779145d89859ba1adb6417a502cbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:47.463 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:47 smithi007 podman[98945]: 2024-03-18 01:49:47.462545578 +0000 UTC m=+0.425097562 container init f2f2b00ff26f1499b7986096f042d05988779145d89859ba1adb6417a502cbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T01:49:47.723 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:47 smithi007 podman[98945]: 2024-03-18 01:49:47.474157548 +0000 UTC m=+0.436709505 container start f2f2b00ff26f1499b7986096f042d05988779145d89859ba1adb6417a502cbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T01:49:47.724 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:47 smithi007 podman[98945]: 2024-03-18 01:49:47.569633318 +0000 UTC m=+0.532185360 container attach f2f2b00ff26f1499b7986096f042d05988779145d89859ba1adb6417a502cbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-18T01:49:48.293 INFO:journalctl@ceph.osd.6.smithi007.stdout:Mar 18 01:49:47 smithi007 podman[98975]: 2024-03-18 01:49:47.990338057 +0000 UTC m=+0.037059317 container died f2f2b00ff26f1499b7986096f042d05988779145d89859ba1adb6417a502cbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:49:48.722 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T01:49:48.839 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:49:48.840 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T01:49:48.840 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T01:49:48.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.7 2024-03-18T01:49:49.155 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:48 smithi046 systemd[1]: Stopping Ceph osd.7 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:49:49.427 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:49 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7[54267]: 2024-03-18T01:49:49.153+0000 7fa76c978700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:49:49.428 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:49 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7[54267]: 2024-03-18T01:49:49.154+0000 7fa76c978700 -1 osd.7 95 *** Got signal Terminated *** 2024-03-18T01:49:49.428 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:49 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7[54267]: 2024-03-18T01:49:49.154+0000 7fa76c978700 -1 osd.7 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:49:53.553 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:49:53 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:49:53.166+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:49:54.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:49:53 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:49:53.870+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:49:54.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:49:54 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:49:54.169+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:49:55.165 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:54 smithi046 podman[71997]: 2024-03-18 01:49:54.78402966 +0000 UTC m=+5.739319944 container died ed03de2b7e76dee9750a6b501b4a342aea62d8dce2299ea59496d4ba56dd6e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-18T01:49:55.165 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:49:54 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:49:54.857+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:49:55.519 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:55 smithi046 podman[71997]: 2024-03-18 01:49:55.262811346 +0000 UTC m=+6.218101612 container cleanup ed03de2b7e76dee9750a6b501b4a342aea62d8dce2299ea59496d4ba56dd6e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T01:49:55.519 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:55 smithi046 bash[71997]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7 2024-03-18T01:49:55.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:49:55 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:49:55.181+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:49:55.833 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:55 smithi046 podman[72028]: 2024-03-18 01:49:55.556150964 +0000 UTC m=+0.774451244 container remove ed03de2b7e76dee9750a6b501b4a342aea62d8dce2299ea59496d4ba56dd6e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:49:56.142 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:55 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:49:55.715+0000 7f93aa829700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.315138+0000 front 2024-03-18T01:48:50.315072+0000 (oldest deadline 2024-03-18T01:49:55.614998+0000) 2024-03-18T01:49:56.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:49:55 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:49:55.832+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:49:56.269 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:55 smithi046 podman[72119]: 2024-03-18 01:49:55.792181186 +0000 UTC m=+0.018081720 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:49:56.269 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:55 smithi046 podman[72119]: 2024-03-18 01:49:55.940362682 +0000 UTC m=+0.166263194 container create b3b153b657e82135647034baa921fefd7e5a1a0f34c6f0dbde10a619774385e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-18T01:49:56.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:49:56 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:49:56.158+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:49:56.613 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:56 smithi046 podman[72119]: 2024-03-18 01:49:56.304564693 +0000 UTC m=+0.530465222 container init b3b153b657e82135647034baa921fefd7e5a1a0f34c6f0dbde10a619774385e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-18T01:49:56.614 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:56 smithi046 podman[72119]: 2024-03-18 01:49:56.307932894 +0000 UTC m=+0.533833412 container start b3b153b657e82135647034baa921fefd7e5a1a0f34c6f0dbde10a619774385e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:49:56.614 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:56 smithi046 podman[72119]: 2024-03-18 01:49:56.384194714 +0000 UTC m=+0.610095243 container attach b3b153b657e82135647034baa921fefd7e5a1a0f34c6f0dbde10a619774385e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:49:56.701 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:56 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:49:56.699+0000 7f93aa829700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.315138+0000 front 2024-03-18T01:48:50.315072+0000 (oldest deadline 2024-03-18T01:49:55.614998+0000) 2024-03-18T01:49:56.702 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:49:56 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:49:56.568+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:49:56.929 INFO:journalctl@ceph.osd.7.smithi046.stdout:Mar 18 01:49:56 smithi046 podman[72119]: 2024-03-18 01:49:56.613627039 +0000 UTC m=+0.839527563 container died b3b153b657e82135647034baa921fefd7e5a1a0f34c6f0dbde10a619774385e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:49:56.929 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:49:56 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:49:56.843+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:49:57.280 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T01:49:57.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:49:57.334 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T01:49:57.334 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T01:49:57.334 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.8 2024-03-18T01:49:57.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:49:57 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:49:57.144+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:49:57.639 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:57 smithi203 systemd[1]: Stopping Ceph osd.8 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:49:57.639 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:49:57 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:49:57.560+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:49:57.892 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:57 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:49:57.638+0000 7f93b223d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:49:57.892 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:57 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:49:57.638+0000 7f93b223d700 -1 osd.8 95 *** Got signal Terminated *** 2024-03-18T01:49:57.892 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:57 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:49:57.638+0000 7f93b223d700 -1 osd.8 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:49:57.892 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:57 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:49:57.711+0000 7f93aa829700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.315138+0000 front 2024-03-18T01:48:50.315072+0000 (oldest deadline 2024-03-18T01:49:55.614998+0000) 2024-03-18T01:49:58.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:49:57 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:49:57.829+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:49:58.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:49:58 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:49:58.171+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:49:58.892 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:58 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:49:58.732+0000 7f93aa829700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.315138+0000 front 2024-03-18T01:48:50.315072+0000 (oldest deadline 2024-03-18T01:49:55.614998+0000) 2024-03-18T01:49:58.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:49:58 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:49:58.536+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:49:59.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:49:58 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:49:58.863+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:49:59.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:49:59 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:49:59.201+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:49:59.892 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:49:59 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:49:59.701+0000 7f93aa829700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.315138+0000 front 2024-03-18T01:48:50.315072+0000 (oldest deadline 2024-03-18T01:49:55.614998+0000) 2024-03-18T01:49:59.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:49:59 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:49:59.569+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:00.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:49:59 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:49:59.890+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:00.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:00 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:00.244+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:00.680 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:00 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:00.546+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:01.142 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:00 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:50:00.678+0000 7f93aa829700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.315138+0000 front 2024-03-18T01:48:50.315072+0000 (oldest deadline 2024-03-18T01:49:55.614998+0000) 2024-03-18T01:50:01.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:00 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:00.845+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:01.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:01 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:01.208+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:01.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:01 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:01.528+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:01.892 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:01 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8[55330]: 2024-03-18T01:50:01.651+0000 7f93aa829700 -1 osd.8 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.315138+0000 front 2024-03-18T01:48:50.315072+0000 (oldest deadline 2024-03-18T01:49:55.614998+0000) 2024-03-18T01:50:02.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:01 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:01.833+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:02.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:02 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:02.192+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:02.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:02 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:02.573+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:03.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:02 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:02.843+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:03.316 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:03 smithi203 podman[73368]: 2024-03-18 01:50:03.040010375 +0000 UTC m=+5.522429380 container died ce4fc9dce4d9507fa9cc203a6a021951af9a9a00f886c96815f5259afb4d34d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-18T01:50:03.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:03 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:03.202+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:03.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:03 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:03.202+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:48:58.959525+0000 front 2024-03-18T01:48:58.959461+0000 (oldest deadline 2024-03-18T01:50:02.459742+0000) 2024-03-18T01:50:03.642 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:03 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:03.561+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:03.642 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:03 smithi203 podman[73368]: 2024-03-18 01:50:03.4140866 +0000 UTC m=+5.896505637 container cleanup ce4fc9dce4d9507fa9cc203a6a021951af9a9a00f886c96815f5259afb4d34d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T01:50:03.642 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:03 smithi203 bash[73368]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8 2024-03-18T01:50:03.992 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:03 smithi203 podman[73389]: 2024-03-18 01:50:03.69052805 +0000 UTC m=+0.646763656 container remove ce4fc9dce4d9507fa9cc203a6a021951af9a9a00f886c96815f5259afb4d34d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-03-18T01:50:04.241 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:03 smithi203 podman[73481]: 2024-03-18 01:50:03.891641039 +0000 UTC m=+0.017836515 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:50:04.241 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:04 smithi203 podman[73481]: 2024-03-18 01:50:04.000175576 +0000 UTC m=+0.126371108 container create dc6fde242781a668d8bf08a3e29cbc4a48c8bf29db128afcdb67afa9ddac4c7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:50:04.242 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:04 smithi203 podman[73481]: 2024-03-18 01:50:04.236207865 +0000 UTC m=+0.362403328 container init dc6fde242781a668d8bf08a3e29cbc4a48c8bf29db128afcdb67afa9ddac4c7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T01:50:04.242 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:04 smithi203 podman[73481]: 2024-03-18 01:50:04.239254962 +0000 UTC m=+0.365450430 container start dc6fde242781a668d8bf08a3e29cbc4a48c8bf29db128afcdb67afa9ddac4c7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T01:50:04.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:03 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:03.819+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:04.533 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:04 smithi203 podman[73481]: 2024-03-18 01:50:04.319824946 +0000 UTC m=+0.446020424 container attach dc6fde242781a668d8bf08a3e29cbc4a48c8bf29db128afcdb67afa9ddac4c7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:50:04.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:04 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:04.222+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:04.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:04 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:04.222+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:48:58.959525+0000 front 2024-03-18T01:48:58.959461+0000 (oldest deadline 2024-03-18T01:50:02.459742+0000) 2024-03-18T01:50:04.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:04 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:04.548+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:04.831 INFO:journalctl@ceph.osd.8.smithi203.stdout:Mar 18 01:50:04 smithi203 podman[73481]: 2024-03-18 01:50:04.531465603 +0000 UTC m=+0.657661081 container died dc6fde242781a668d8bf08a3e29cbc4a48c8bf29db128afcdb67afa9ddac4c7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:50:05.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T01:50:05.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:05.222 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T01:50:05.222 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T01:50:05.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.9 2024-03-18T01:50:05.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:04 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:04.824+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:05.553 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:05 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:05.186+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:05.553 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:05 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:05.186+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:48:58.959525+0000 front 2024-03-18T01:48:58.959461+0000 (oldest deadline 2024-03-18T01:50:02.459742+0000) 2024-03-18T01:50:05.553 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:05 smithi007 systemd[1]: Stopping Ceph osd.9 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:05.891 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:05 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:05.519+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:06.239 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:05 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:05.882+0000 7ff08a600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:06.239 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:05 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:05.882+0000 7ff08a600700 -1 osd.9 95 *** Got signal Terminated *** 2024-03-18T01:50:06.239 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:05 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:05.882+0000 7ff08a600700 -1 osd.9 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:50:06.239 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:06 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:06.237+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:06.239 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:06 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:06.237+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:48:58.959525+0000 front 2024-03-18T01:48:58.959461+0000 (oldest deadline 2024-03-18T01:50:02.459742+0000) 2024-03-18T01:50:06.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:05 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:05.816+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:06.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:06 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:06.533+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:07.020 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:06 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:06.772+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:07.020 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:06 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:06.772+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:07.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:07 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:07.268+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:07.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:07 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:07.268+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:48:58.959525+0000 front 2024-03-18T01:48:58.959461+0000 (oldest deadline 2024-03-18T01:50:02.459742+0000) 2024-03-18T01:50:07.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:07 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:07.491+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:08.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:07 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:07.757+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:08.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:07 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:07.757+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:08.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:08 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:08.235+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:08.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:08 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:08.235+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:48:58.959525+0000 front 2024-03-18T01:48:58.959461+0000 (oldest deadline 2024-03-18T01:50:02.459742+0000) 2024-03-18T01:50:08.891 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:08 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:08.523+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:09.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:08 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:08.733+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:09.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:08 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:08.733+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:09.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:09 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:09.216+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:09.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:09 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:09.216+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:48:58.959525+0000 front 2024-03-18T01:48:58.959461+0000 (oldest deadline 2024-03-18T01:50:02.459742+0000) 2024-03-18T01:50:09.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:09 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:09.541+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:10.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:09 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:09.713+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:10.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:09 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:09.713+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:10.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:10 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:10.236+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.255434+0000 front 2024-03-18T01:48:50.255859+0000 (oldest deadline 2024-03-18T01:49:53.155375+0000) 2024-03-18T01:50:10.552 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:10 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9[66907]: 2024-03-18T01:50:10.236+0000 7ff085e00700 -1 osd.9 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:48:58.959525+0000 front 2024-03-18T01:48:58.959461+0000 (oldest deadline 2024-03-18T01:50:02.459742+0000) 2024-03-18T01:50:10.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:10 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:10.540+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:11.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:10 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:10.720+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:11.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:10 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:10.720+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:11.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:11 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:11.553+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:12.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:11 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:11.703+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:12.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:11 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:11.703+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:12.167 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:11 smithi007 podman[99063]: 2024-03-18 01:50:11.808531047 +0000 UTC m=+6.114693679 container died cc5f2dce9df77ed18d74cd1a85047decae238014dc81f19fb540ec21c7b8ce6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-03-18T01:50:12.553 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:12 smithi007 podman[99063]: 2024-03-18 01:50:12.166767014 +0000 UTC m=+6.472929640 container cleanup cc5f2dce9df77ed18d74cd1a85047decae238014dc81f19fb540ec21c7b8ce6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T01:50:12.553 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:12 smithi007 bash[99063]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9 2024-03-18T01:50:12.553 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:12 smithi007 podman[99083]: 2024-03-18 01:50:12.359662005 +0000 UTC m=+0.544851129 container remove cc5f2dce9df77ed18d74cd1a85047decae238014dc81f19fb540ec21c7b8ce6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-18T01:50:12.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:12 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:12.596+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:12.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:12 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:12.596+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:13.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:12 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:12.718+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:13.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:12 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:12.718+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:13.472 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:13 smithi007 podman[99171]: 2024-03-18 01:50:13.099082228 +0000 UTC m=+0.038194032 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:50:13.472 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:13 smithi007 podman[99171]: 2024-03-18 01:50:13.210473698 +0000 UTC m=+0.149585539 container create 8c97edd8c8fddac20749b6d698f87a460b6f9201923244cb62ecd9527fd4c0ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:50:13.802 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:13 smithi007 podman[99171]: 2024-03-18 01:50:13.471780176 +0000 UTC m=+0.410892004 container init 8c97edd8c8fddac20749b6d698f87a460b6f9201923244cb62ecd9527fd4c0ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T01:50:13.802 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:13 smithi007 podman[99171]: 2024-03-18 01:50:13.482671124 +0000 UTC m=+0.421782972 container start 8c97edd8c8fddac20749b6d698f87a460b6f9201923244cb62ecd9527fd4c0ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T01:50:13.803 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:13 smithi007 podman[99171]: 2024-03-18 01:50:13.554427264 +0000 UTC m=+0.493539084 container attach 8c97edd8c8fddac20749b6d698f87a460b6f9201923244cb62ecd9527fd4c0ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T01:50:13.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:13 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:13.574+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:13.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:13 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:13.574+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:14.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:13 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:13.710+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:14.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:13 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:13.710+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:14.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:13 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:13.710+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:08.944517+0000 front 2024-03-18T01:49:08.944558+0000 (oldest deadline 2024-03-18T01:50:13.044181+0000) 2024-03-18T01:50:14.243 INFO:journalctl@ceph.osd.9.smithi007.stdout:Mar 18 01:50:13 smithi007 podman[99171]: 2024-03-18 01:50:13.947991224 +0000 UTC m=+0.887103037 container died 8c97edd8c8fddac20749b6d698f87a460b6f9201923244cb62ecd9527fd4c0ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-18T01:50:14.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T01:50:14.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:14.724 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T01:50:14.724 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T01:50:14.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.10 2024-03-18T01:50:14.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:14 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:14.614+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:14.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:14 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:14.614+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:15.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:14 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:14.688+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:15.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:14 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:14.688+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:15.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:14 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:14.688+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:08.944517+0000 front 2024-03-18T01:49:08.944558+0000 (oldest deadline 2024-03-18T01:50:13.044181+0000) 2024-03-18T01:50:15.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:14 smithi046 systemd[1]: Stopping Ceph osd.10 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:15.519 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:15 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:15.026+0000 7f71e4b45700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:15.519 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:15 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:15.026+0000 7f71e4b45700 -1 osd.10 95 *** Got signal Terminated *** 2024-03-18T01:50:15.519 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:15 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:15.026+0000 7f71e4b45700 -1 osd.10 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:50:15.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:15 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:15.639+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:15.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:15 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:15.639+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:16.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:15 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:15.660+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:16.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:15 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:15.660+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:16.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:15 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:15.660+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:08.944517+0000 front 2024-03-18T01:49:08.944558+0000 (oldest deadline 2024-03-18T01:50:13.044181+0000) 2024-03-18T01:50:16.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:16 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:16.636+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:16.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:16 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:16.636+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:17.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:16 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:16.685+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:17.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:16 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:16.685+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:17.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:16 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:16.685+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:08.944517+0000 front 2024-03-18T01:49:08.944558+0000 (oldest deadline 2024-03-18T01:50:13.044181+0000) 2024-03-18T01:50:18.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:17 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:17.724+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:18.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:17 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:17.724+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:18.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:17 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:17.724+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:08.944517+0000 front 2024-03-18T01:49:08.944558+0000 (oldest deadline 2024-03-18T01:50:13.044181+0000) 2024-03-18T01:50:18.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:17 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:17.656+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:18.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:17 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:17.656+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:19.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:18 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:18.730+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:19.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:18 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:18.730+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:19.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:18 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:18.730+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:08.944517+0000 front 2024-03-18T01:49:08.944558+0000 (oldest deadline 2024-03-18T01:50:13.044181+0000) 2024-03-18T01:50:19.141 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:18 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:18.674+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:19.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:18 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:18.674+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:20.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:19 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:19.699+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:50.940681+0000 front 2024-03-18T01:48:50.940853+0000 (oldest deadline 2024-03-18T01:49:53.240614+0000) 2024-03-18T01:50:20.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:19 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:19.699+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.46:6804 osd.1 since back 2024-03-18T01:49:00.743152+0000 front 2024-03-18T01:49:00.743270+0000 (oldest deadline 2024-03-18T01:50:06.643310+0000) 2024-03-18T01:50:20.019 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:19 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10[61066]: 2024-03-18T01:50:19.699+0000 7f71e1139700 -1 osd.10 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:08.944517+0000 front 2024-03-18T01:49:08.944558+0000 (oldest deadline 2024-03-18T01:50:13.044181+0000) 2024-03-18T01:50:20.141 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:19.683+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:20.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:19 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:19.683+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:20.930 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:20 smithi046 podman[72226]: 2024-03-18 01:50:20.520380956 +0000 UTC m=+5.593648709 container died 3964f92208be0d2245a3870af7ad8f19e0d19aafc9910a38d1ba0436a7f19bb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-18T01:50:21.141 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:20 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:20.713+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:21.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:20 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:20.713+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:21.214 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:21 smithi046 podman[72226]: 2024-03-18 01:50:21.026087668 +0000 UTC m=+6.099355417 container cleanup 3964f92208be0d2245a3870af7ad8f19e0d19aafc9910a38d1ba0436a7f19bb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-18T01:50:21.214 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:21 smithi046 bash[72226]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10 2024-03-18T01:50:21.519 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:21 smithi046 podman[72252]: 2024-03-18 01:50:21.302390578 +0000 UTC m=+0.780811849 container remove 3964f92208be0d2245a3870af7ad8f19e0d19aafc9910a38d1ba0436a7f19bb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-18T01:50:21.914 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:21 smithi046 podman[72347]: 2024-03-18 01:50:21.506146362 +0000 UTC m=+0.018157786 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:50:21.914 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:21 smithi046 podman[72347]: 2024-03-18 01:50:21.628158584 +0000 UTC m=+0.140170017 container create cebc877f9fd02507ab439b5f3311fd03fb67d3a709a029fcc2ee9ae0ec4025a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-18T01:50:21.914 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:21 smithi046 podman[72347]: 2024-03-18 01:50:21.822841557 +0000 UTC m=+0.334852980 container init cebc877f9fd02507ab439b5f3311fd03fb67d3a709a029fcc2ee9ae0ec4025a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T01:50:21.914 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:21 smithi046 podman[72347]: 2024-03-18 01:50:21.826147004 +0000 UTC m=+0.338158413 container start cebc877f9fd02507ab439b5f3311fd03fb67d3a709a029fcc2ee9ae0ec4025a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:50:21.914 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:21 smithi046 podman[72347]: 2024-03-18 01:50:21.913360817 +0000 UTC m=+0.425372245 container attach cebc877f9fd02507ab439b5f3311fd03fb67d3a709a029fcc2ee9ae0ec4025a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T01:50:22.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:21 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:21.699+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:22.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:21 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:21.699+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:22.269 INFO:journalctl@ceph.osd.10.smithi046.stdout:Mar 18 01:50:22 smithi046 podman[72347]: 2024-03-18 01:50:22.130492657 +0000 UTC m=+0.642504074 container died cebc877f9fd02507ab439b5f3311fd03fb67d3a709a029fcc2ee9ae0ec4025a9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T01:50:22.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T01:50:22.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:22.879 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T01:50:22.879 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T01:50:22.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.11 2024-03-18T01:50:23.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:22 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:22.729+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:23.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:22 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:22.729+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:23.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:22 smithi203 systemd[1]: Stopping Ceph osd.11 for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:23.723 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:23 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:23.451+0000 7fbb0e94e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:23.724 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:23 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:23.451+0000 7fbb0e94e700 -1 osd.11 95 *** Got signal Terminated *** 2024-03-18T01:50:23.724 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:23 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:23.451+0000 7fbb0e94e700 -1 osd.11 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:50:23.724 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:23 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:23.723+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:23.724 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:23 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:23.723+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:25.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:24 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:24.763+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:25.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:24 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:24.763+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:25.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:24 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:24.763+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-03-18T01:49:14.298859+0000 front 2024-03-18T01:49:14.298567+0000 (oldest deadline 2024-03-18T01:50:24.298758+0000) 2024-03-18T01:50:26.043 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:25 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:25.716+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:26.043 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:25 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:25.716+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:26.043 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:25 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:25.716+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-03-18T01:49:14.298859+0000 front 2024-03-18T01:49:14.298567+0000 (oldest deadline 2024-03-18T01:50:24.298758+0000) 2024-03-18T01:50:27.141 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:26 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:26.716+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:27.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:26 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:26.716+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:27.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:26 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:26.716+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-03-18T01:49:14.298859+0000 front 2024-03-18T01:49:14.298567+0000 (oldest deadline 2024-03-18T01:50:24.298758+0000) 2024-03-18T01:50:28.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:27 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:27.705+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-03-18T01:48:51.594288+0000 front 2024-03-18T01:48:51.594462+0000 (oldest deadline 2024-03-18T01:49:56.294265+0000) 2024-03-18T01:50:28.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:27 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:27.705+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-03-18T01:49:06.796800+0000 front 2024-03-18T01:49:06.796678+0000 (oldest deadline 2024-03-18T01:50:12.096789+0000) 2024-03-18T01:50:28.142 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:27 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11[62414]: 2024-03-18T01:50:27.705+0000 7fbb0af42700 -1 osd.11 95 heartbeat_check: no reply from 172.21.15.7:6814 osd.3 since back 2024-03-18T01:49:14.298859+0000 front 2024-03-18T01:49:14.298567+0000 (oldest deadline 2024-03-18T01:50:24.298758+0000) 2024-03-18T01:50:29.317 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:29 smithi203 podman[73586]: 2024-03-18 01:50:29.005702391 +0000 UTC m=+5.933808843 container died be1945d11dda51a17886cc716a85863f2ff7526b216df3db23434cc8e82d62c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T01:50:29.317 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:29 smithi203 podman[73586]: 2024-03-18 01:50:29.317251034 +0000 UTC m=+6.245357468 container cleanup be1945d11dda51a17886cc716a85863f2ff7526b216df3db23434cc8e82d62c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:50:29.642 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:29 smithi203 bash[73586]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11 2024-03-18T01:50:29.642 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:29 smithi203 podman[73603]: 2024-03-18 01:50:29.502026363 +0000 UTC m=+0.496477455 container remove be1945d11dda51a17886cc716a85863f2ff7526b216df3db23434cc8e82d62c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:50:30.182 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:29 smithi203 podman[73694]: 2024-03-18 01:50:29.797444598 +0000 UTC m=+0.018010297 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:50:30.182 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:29 smithi203 podman[73694]: 2024-03-18 01:50:29.911321525 +0000 UTC m=+0.131887234 container create e449612e70025c57b2813e235ec6884b98d42465b53dfe9a5b4b680e604eb091 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:50:30.182 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:30 smithi203 podman[73694]: 2024-03-18 01:50:30.105693848 +0000 UTC m=+0.326259557 container init e449612e70025c57b2813e235ec6884b98d42465b53dfe9a5b4b680e604eb091 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True) 2024-03-18T01:50:30.182 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:30 smithi203 podman[73694]: 2024-03-18 01:50:30.108766007 +0000 UTC m=+0.329331723 container start e449612e70025c57b2813e235ec6884b98d42465b53dfe9a5b4b680e604eb091 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:50:30.182 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:30 smithi203 podman[73694]: 2024-03-18 01:50:30.180712936 +0000 UTC m=+0.401278664 container attach e449612e70025c57b2813e235ec6884b98d42465b53dfe9a5b4b680e604eb091 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T01:50:30.642 INFO:journalctl@ceph.osd.11.smithi203.stdout:Mar 18 01:50:30 smithi203 podman[73694]: 2024-03-18 01:50:30.393922967 +0000 UTC m=+0.614488686 container died e449612e70025c57b2813e235ec6884b98d42465b53dfe9a5b4b680e604eb091 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T01:50:30.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T01:50:31.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:31.015 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T01:50:31.016 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T01:50:31.016 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.a 2024-03-18T01:50:31.552 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:50:31 smithi007 systemd[1]: Stopping Ceph mds.a for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:32.052 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:50:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a[73874]: 2024-03-18T01:50:31.583+0000 7f5767400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:32.052 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:50:31 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a[73874]: 2024-03-18T01:50:31.583+0000 7f5767400700 -1 mds.a *** got signal Terminated *** 2024-03-18T01:50:35.552 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:50:35 smithi007 podman[99287]: 2024-03-18 01:50:35.288219264 +0000 UTC m=+3.823150946 container died a8aabe0db2a02e37f24642a9de6bfcec43b490ae48d3c943346bffcb0402d9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T01:50:36.021 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:50:35 smithi007 podman[99287]: 2024-03-18 01:50:35.69229592 +0000 UTC m=+4.227227595 container cleanup a8aabe0db2a02e37f24642a9de6bfcec43b490ae48d3c943346bffcb0402d9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:50:36.022 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:50:35 smithi007 bash[99287]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a 2024-03-18T01:50:36.298 INFO:journalctl@ceph.mds.a.smithi007.stdout:Mar 18 01:50:36 smithi007 podman[99302]: 2024-03-18 01:50:36.020850063 +0000 UTC m=+0.720297145 container remove a8aabe0db2a02e37f24642a9de6bfcec43b490ae48d3c943346bffcb0402d9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-18T01:50:36.331 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T01:50:36.456 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:36.456 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T01:50:36.456 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T01:50:36.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.d 2024-03-18T01:50:37.052 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:50:36 smithi007 systemd[1]: Stopping Ceph mds.d for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:37.552 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:50:37 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d[72857]: 2024-03-18T01:50:37.123+0000 7fa84c400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:37.552 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:50:37 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d[72857]: 2024-03-18T01:50:37.123+0000 7fa84c400700 -1 mds.d *** got signal Terminated *** 2024-03-18T01:50:40.242 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:50:39 smithi007 podman[99407]: 2024-03-18 01:50:39.98069959 +0000 UTC m=+3.017595006 container died 0387ab145024b00de49efe003416e571d97b4b76265cdc599975d0dec8165f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:50:40.552 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:50:40 smithi007 podman[99407]: 2024-03-18 01:50:40.354126118 +0000 UTC m=+3.391021560 container cleanup 0387ab145024b00de49efe003416e571d97b4b76265cdc599975d0dec8165f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-18T01:50:40.552 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:50:40 smithi007 bash[99407]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d 2024-03-18T01:50:41.052 INFO:journalctl@ceph.mds.d.smithi007.stdout:Mar 18 01:50:40 smithi007 podman[99421]: 2024-03-18 01:50:40.597010255 +0000 UTC m=+0.604964720 container remove 0387ab145024b00de49efe003416e571d97b4b76265cdc599975d0dec8165f02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-18T01:50:41.111 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T01:50:41.238 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:41.238 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T01:50:41.238 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T01:50:41.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.g 2024-03-18T01:50:41.802 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:50:41 smithi007 systemd[1]: Stopping Ceph mds.g for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:42.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:50:41 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g[73485]: 2024-03-18T01:50:41.880+0000 7fa111000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:42.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:50:41 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g[73485]: 2024-03-18T01:50:41.880+0000 7fa111000700 -1 mds.g *** got signal Terminated *** 2024-03-18T01:50:43.991 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:50:43 smithi007 podman[99527]: 2024-03-18 01:50:43.73454628 +0000 UTC m=+1.993433859 container died 62fd0322f37c82259e6953f61ad164b0aa4e3013d8086d2d9d1ce4cc98aabc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T01:50:44.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:50:44 smithi007 podman[99527]: 2024-03-18 01:50:44.0830536 +0000 UTC m=+2.341941160 container cleanup 62fd0322f37c82259e6953f61ad164b0aa4e3013d8086d2d9d1ce4cc98aabc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-18T01:50:44.302 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:50:44 smithi007 bash[99527]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g 2024-03-18T01:50:44.802 INFO:journalctl@ceph.mds.g.smithi007.stdout:Mar 18 01:50:44 smithi007 podman[99541]: 2024-03-18 01:50:44.329803435 +0000 UTC m=+0.580205437 container remove 62fd0322f37c82259e6953f61ad164b0aa4e3013d8086d2d9d1ce4cc98aabc89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:50:44.855 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T01:50:44.974 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:44.975 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T01:50:44.975 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T01:50:44.975 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.j 2024-03-18T01:50:45.420 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:50:45 smithi007 systemd[1]: Stopping Ceph mds.j for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:45.726 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:50:45 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j[72489]: 2024-03-18T01:50:45.573+0000 7f2442200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:45.726 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:50:45 smithi007 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j[72489]: 2024-03-18T01:50:45.573+0000 7f2442200700 -1 mds.j *** got signal Terminated *** 2024-03-18T01:50:48.037 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:50:47 smithi007 podman[99648]: 2024-03-18 01:50:47.72484431 +0000 UTC m=+2.283180636 container died 6f3aae8dc00727f1a5157192dc6b879d0f247a6467adc3b3daa1ec35428b9156 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-03-18T01:50:48.302 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:50:48 smithi007 podman[99648]: 2024-03-18 01:50:48.145226406 +0000 UTC m=+2.703562749 container cleanup 6f3aae8dc00727f1a5157192dc6b879d0f247a6467adc3b3daa1ec35428b9156 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T01:50:48.302 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:50:48 smithi007 bash[99648]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j 2024-03-18T01:50:48.802 INFO:journalctl@ceph.mds.j.smithi007.stdout:Mar 18 01:50:48 smithi007 podman[99673]: 2024-03-18 01:50:48.427031625 +0000 UTC m=+0.684943302 container remove 6f3aae8dc00727f1a5157192dc6b879d0f247a6467adc3b3daa1ec35428b9156 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-j, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:50:48.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T01:50:49.059 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:49.059 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T01:50:49.060 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T01:50:49.060 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.b 2024-03-18T01:50:49.485 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:50:49 smithi046 systemd[1]: Stopping Ceph mds.b for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:49.763 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:50:49 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: 2024-03-18T01:50:49.484+0000 7f768d43b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:49.764 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:50:49 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b[66763]: 2024-03-18T01:50:49.484+0000 7f768d43b700 -1 mds.b *** got signal Terminated *** 2024-03-18T01:50:51.903 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:50:51 smithi046 podman[72454]: 2024-03-18 01:50:51.627947743 +0000 UTC m=+2.365784410 container died efeff0e0d0eb1cc1bc175bc69973aabab95d78394acf7f4a634783ae4fd47911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-18T01:50:52.269 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:50:51 smithi046 podman[72454]: 2024-03-18 01:50:51.985470292 +0000 UTC m=+2.723306952 container cleanup efeff0e0d0eb1cc1bc175bc69973aabab95d78394acf7f4a634783ae4fd47911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:50:52.269 INFO:journalctl@ceph.mds.b.smithi046.stdout:Mar 18 01:50:51 smithi046 bash[72454]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-b 2024-03-18T01:50:52.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T01:50:52.442 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:52.443 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T01:50:52.443 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T01:50:52.443 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.e 2024-03-18T01:50:52.928 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:50:52 smithi046 systemd[1]: Stopping Ceph mds.e for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:53.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:50:52 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e[66143]: 2024-03-18T01:50:52.926+0000 7f1b29dd2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:53.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:50:52 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e[66143]: 2024-03-18T01:50:52.926+0000 7f1b29dd2700 -1 mds.e *** got signal Terminated *** 2024-03-18T01:50:55.932 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:50:55 smithi046 podman[72587]: 2024-03-18 01:50:55.627376355 +0000 UTC m=+2.956681951 container died f2113119d359b53f57681741274160f3c89783b0bce2cc8d086da67c92d8b238 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T01:50:56.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:50:56 smithi046 podman[72587]: 2024-03-18 01:50:56.022866397 +0000 UTC m=+3.352171984 container cleanup f2113119d359b53f57681741274160f3c89783b0bce2cc8d086da67c92d8b238 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:50:56.269 INFO:journalctl@ceph.mds.e.smithi046.stdout:Mar 18 01:50:56 smithi046 bash[72587]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-e 2024-03-18T01:50:56.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T01:50:56.580 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:50:56.580 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T01:50:56.580 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T01:50:56.581 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.h 2024-03-18T01:50:56.901 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:50:56 smithi046 systemd[1]: Stopping Ceph mds.h for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:50:57.269 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:50:56 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h[66456]: 2024-03-18T01:50:56.900+0000 7f832a48f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:50:57.269 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:50:56 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h[66456]: 2024-03-18T01:50:56.900+0000 7f832a48f700 -1 mds.h *** got signal Terminated *** 2024-03-18T01:50:59.893 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:50:59 smithi046 podman[72709]: 2024-03-18 01:50:59.621472105 +0000 UTC m=+2.820954279 container died 43371fca873109d18a91d9d1e2f368de963242a7121e10f0a914b7e1082b7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-18T01:51:00.144 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:50:59 smithi046 podman[72709]: 2024-03-18 01:50:59.893296701 +0000 UTC m=+3.092778852 container cleanup 43371fca873109d18a91d9d1e2f368de963242a7121e10f0a914b7e1082b7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:51:00.144 INFO:journalctl@ceph.mds.h.smithi046.stdout:Mar 18 01:50:59 smithi046 bash[72709]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-h 2024-03-18T01:51:00.289 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T01:51:00.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:51:00.327 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T01:51:00.327 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T01:51:00.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.k 2024-03-18T01:51:00.769 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:00 smithi046 systemd[1]: Stopping Ceph mds.k for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:51:00.769 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:00 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k[65822]: 2024-03-18T01:51:00.638+0000 7f60b6bc4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:51:00.769 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:00 smithi046 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k[65822]: 2024-03-18T01:51:00.638+0000 7f60b6bc4700 -1 mds.k *** got signal Terminated *** 2024-03-18T01:51:03.981 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:03 smithi046 podman[72830]: 2024-03-18 01:51:03.628553535 +0000 UTC m=+3.076553258 container died e12538bc638e1defbc0a33966eaa58c1de6bce182d0f9cdeb6eafaf47e0d51de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T01:51:04.256 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:04 smithi046 podman[72830]: 2024-03-18 01:51:04.055260121 +0000 UTC m=+3.503259837 container cleanup e12538bc638e1defbc0a33966eaa58c1de6bce182d0f9cdeb6eafaf47e0d51de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:51:04.257 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:04 smithi046 bash[72830]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k 2024-03-18T01:51:04.498 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T01:51:04.519 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:04 smithi046 podman[72843]: 2024-03-18 01:51:04.256168954 +0000 UTC m=+0.615654424 container remove e12538bc638e1defbc0a33966eaa58c1de6bce182d0f9cdeb6eafaf47e0d51de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T01:51:04.519 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:04 smithi046 systemd[1]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.k.service: Deactivated successfully. 2024-03-18T01:51:04.519 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:04 smithi046 systemd[1]: Stopped Ceph mds.k for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:51:04.519 INFO:journalctl@ceph.mds.k.smithi046.stdout:Mar 18 01:51:04 smithi046 systemd[1]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.k.service: Consumed 2.285s CPU time. 2024-03-18T01:51:04.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:51:04.537 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T01:51:04.537 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T01:51:04.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.c 2024-03-18T01:51:04.892 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:51:04 smithi203 systemd[1]: Stopping Ceph mds.c for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:51:05.391 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:51:04 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c[67845]: 2024-03-18T01:51:04.920+0000 7f526ef33700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:51:05.392 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:51:04 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c[67845]: 2024-03-18T01:51:04.920+0000 7f526ef33700 -1 mds.c *** got signal Terminated *** 2024-03-18T01:51:07.961 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:51:07 smithi203 podman[73800]: 2024-03-18 01:51:07.644290327 +0000 UTC m=+2.914614514 container died d0bd257a2babf70a56e47979d228b72ce390ef86d4e8fe2a8176e09f7155a8ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-18T01:51:08.236 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:51:08 smithi203 podman[73800]: 2024-03-18 01:51:08.051017356 +0000 UTC m=+3.321341534 container cleanup d0bd257a2babf70a56e47979d228b72ce390ef86d4e8fe2a8176e09f7155a8ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-18T01:51:08.236 INFO:journalctl@ceph.mds.c.smithi203.stdout:Mar 18 01:51:08 smithi203 bash[73800]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-c 2024-03-18T01:51:08.370 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T01:51:08.408 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:51:08.408 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T01:51:08.409 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T01:51:08.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.f 2024-03-18T01:51:08.892 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:51:08 smithi203 systemd[1]: Stopping Ceph mds.f for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:51:08.892 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:51:08 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f[67536]: 2024-03-18T01:51:08.721+0000 7efe0d2f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:51:08.892 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:51:08 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f[67536]: 2024-03-18T01:51:08.721+0000 7efe0d2f6700 -1 mds.f *** got signal Terminated *** 2024-03-18T01:51:11.895 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:51:11 smithi203 podman[73931]: 2024-03-18 01:51:11.641923916 +0000 UTC m=+3.018999884 container died 66bc46eea89d2ab821becba72b98940068d04c7d6bf2284d791f99ed3aeb3348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T01:51:12.273 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:51:11 smithi203 podman[73931]: 2024-03-18 01:51:11.9886473 +0000 UTC m=+3.365723265 container cleanup 66bc46eea89d2ab821becba72b98940068d04c7d6bf2284d791f99ed3aeb3348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T01:51:12.273 INFO:journalctl@ceph.mds.f.smithi203.stdout:Mar 18 01:51:11 smithi203 bash[73931]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-f 2024-03-18T01:51:12.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T01:51:12.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:51:12.444 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T01:51:12.445 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T01:51:12.445 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.i 2024-03-18T01:51:12.892 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:12 smithi203 systemd[1]: Stopping Ceph mds.i for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:51:12.892 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:12 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i[68153]: 2024-03-18T01:51:12.758+0000 7f964d82f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:51:12.892 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:12 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i[68153]: 2024-03-18T01:51:12.758+0000 7f964d82f700 -1 mds.i *** got signal Terminated *** 2024-03-18T01:51:15.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:15 smithi203 podman[74052]: 2024-03-18 01:51:15.637642015 +0000 UTC m=+2.975664706 container died b94be326eadc665ac4f60567c3632b9779ca1bb1f1ac5252e7f4510682ce532d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:51:16.142 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:15 smithi203 podman[74052]: 2024-03-18 01:51:15.976165143 +0000 UTC m=+3.314187833 container cleanup b94be326eadc665ac4f60567c3632b9779ca1bb1f1ac5252e7f4510682ce532d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-18T01:51:16.142 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:15 smithi203 bash[74052]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i 2024-03-18T01:51:16.396 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T01:51:16.420 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:16 smithi203 podman[74066]: 2024-03-18 01:51:16.169108107 +0000 UTC m=+0.527935826 container remove b94be326eadc665ac4f60567c3632b9779ca1bb1f1ac5252e7f4510682ce532d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-i, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T01:51:16.420 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:16 smithi203 systemd[1]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.i.service: Deactivated successfully. 2024-03-18T01:51:16.420 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:16 smithi203 systemd[1]: Stopped Ceph mds.i for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715. 2024-03-18T01:51:16.421 INFO:journalctl@ceph.mds.i.smithi203.stdout:Mar 18 01:51:16 smithi203 systemd[1]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.i.service: Consumed 4min 11.988s CPU time. 2024-03-18T01:51:16.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:51:16.430 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T01:51:16.430 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T01:51:16.431 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.l 2024-03-18T01:51:16.739 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:51:16 smithi203 systemd[1]: Stopping Ceph mds.l for 3fdfdc7c-e4c6-11ee-95c9-87774f69a715... 2024-03-18T01:51:17.142 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:51:16 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l[67226]: 2024-03-18T01:51:16.738+0000 7fdd3323b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:51:17.142 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:51:16 smithi203 ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l[67226]: 2024-03-18T01:51:16.738+0000 7fdd3323b700 -1 mds.l *** got signal Terminated *** 2024-03-18T01:51:20.041 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:51:19 smithi203 podman[74172]: 2024-03-18 01:51:19.642504665 +0000 UTC m=+2.999734956 container died f292167f2cf18883133742c8b9858825766320ef89e754357bd9213f0a5d71a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:51:20.342 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:51:20 smithi203 podman[74172]: 2024-03-18 01:51:20.122531784 +0000 UTC m=+3.479762067 container cleanup f292167f2cf18883133742c8b9858825766320ef89e754357bd9213f0a5d71a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T01:51:20.343 INFO:journalctl@ceph.mds.l.smithi203.stdout:Mar 18 01:51:20 smithi203 bash[74172]: ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715-mds-l 2024-03-18T01:51:20.560 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3fdfdc7c-e4c6-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T01:51:20.594 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:51:20.594 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T01:51:20.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T01:51:20.931 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:51:26.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T01:51:26.541 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:51:29.178 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T01:51:29.369 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T01:51:31.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:51:32.057 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:51:32.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:51:32.127 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T01:51:32.129 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi007/crash 2024-03-18T01:51:32.130 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash -- . 2024-03-18T01:51:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T01:51:32.254 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2024-03-18T01:51:32.260 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi046/crash 2024-03-18T01:51:32.261 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash -- . 2024-03-18T01:51:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:tar: /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T01:51:32.322 INFO:teuthology.orchestra.run.smithi046.stderr:tar: Error is not recoverable: exiting now 2024-03-18T01:51:32.324 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi203/crash 2024-03-18T01:51:32.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash -- . 2024-03-18T01:51:32.361 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T01:51:32.361 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-03-18T01:51:32.361 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T01:51:32.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T01:51:32.548 INFO:teuthology.orchestra.run.smithi007.stdout:2024-03-18T01:28:04.885193+0000 mon.a (mon.0) 211 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] is down (out of quorum) 2024-03-18T01:51:32.548 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-18T01:51:32.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[SEC\]' /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T01:51:32.657 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]' /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T01:51:32.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[WRN\]' /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T01:51:32.843 INFO:teuthology.orchestra.run.smithi007.stdout:2024-03-18T01:28:04.885193+0000 mon.a (mon.0) 211 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] is down (out of quorum) 2024-03-18T01:51:32.844 INFO:tasks.cephadm:Compressing logs... 2024-03-18T01:51:32.844 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T01:51:32.848 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T01:51:32.851 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T01:51:32.892 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T01:51:32.898 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T01:51:32.898 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T01:51:32.898 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-18T01:51:32.898 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T01:51:32.898 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-18T01:51:32.898 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-18T01:51:32.898 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-18T01:51:32.898 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-18T01:51:32.899 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-18T01:51:32.899 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-18T01:51:32.899 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T01:51:32.899 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log 2024-03-18T01:51:32.899 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T01:51:32.899 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-18T01:51:32.899 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T01:51:32.899 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-18T01:51:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T01:51:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-18T01:51:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T01:51:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-18T01:51:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-18T01:51:32.901 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.1.log: /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.e.log: /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-18T01:51:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log: /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.b.log: 92.7% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-18T01:51:32.934 INFO:teuthology.orchestra.run.smithi046.stderr: 82.0% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T01:51:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T01:51:32.939 INFO:teuthology.orchestra.run.smithi007.stderr:: No such file or directory 2024-03-18T01:51:32.950 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-18T01:51:32.952 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-18T01:51:32.953 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T01:51:32.954 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-18T01:51:32.955 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-18T01:51:32.956 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-18T01:51:32.957 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-18T01:51:32.959 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T01:51:32.959 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T01:51:32.960 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log 2024-03-18T01:51:32.961 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T01:51:32.962 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-18T01:51:32.963 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-18T01:51:32.964 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-18T01:51:32.966 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.9.log: /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-18T01:51:32.966 INFO:teuthology.orchestra.run.smithi007.stderr: 87.8% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T01:51:32.967 INFO:teuthology.orchestra.run.smithi203.stderr: 90.3% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T01:51:32.969 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 94.5% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-18T01:51:32.970 INFO:teuthology.orchestra.run.smithi007.stderr: 94.4% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-18T01:51:32.972 INFO:teuthology.orchestra.run.smithi203.stderr: 82.0% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T01:51:33.010 INFO:teuthology.orchestra.run.smithi046.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T01:51:33.010 INFO:teuthology.orchestra.run.smithi007.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-18T01:51:33.042 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T01:51:33.078 INFO:teuthology.orchestra.run.smithi007.stderr: 82.5% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T01:51:33.083 INFO:teuthology.orchestra.run.smithi203.stderr: 87.8% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T01:51:33.101 INFO:teuthology.orchestra.run.smithi007.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T01:51:33.109 INFO:teuthology.orchestra.run.smithi046.stderr: 90.3% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T01:51:33.110 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T01:51:33.141 INFO:teuthology.orchestra.run.smithi203.stderr: 90.9% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T01:51:33.144 INFO:teuthology.orchestra.run.smithi007.stderr: 90.2% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T01:51:33.288 INFO:teuthology.orchestra.run.smithi046.stderr: 91.0% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T01:51:33.334 INFO:teuthology.orchestra.run.smithi007.stderr: 90.8% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T01:51:34.038 INFO:teuthology.orchestra.run.smithi203.stderr: 94.8% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-18T01:51:34.397 INFO:teuthology.orchestra.run.smithi007.stderr: 94.1% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-18T01:51:34.551 INFO:teuthology.orchestra.run.smithi007.stderr: 89.8% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-18T01:51:35.272 INFO:teuthology.orchestra.run.smithi046.stderr: 95.2% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-18T01:51:35.410 INFO:teuthology.orchestra.run.smithi046.stderr: 95.1% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-18T01:51:35.790 INFO:teuthology.orchestra.run.smithi203.stderr: 91.7% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-18T01:51:36.562 INFO:teuthology.orchestra.run.smithi203.stderr: 96.2% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-18T01:51:36.808 INFO:teuthology.orchestra.run.smithi046.stderr: 91.6% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-18T01:51:37.769 INFO:teuthology.orchestra.run.smithi203.stderr: 94.4% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-18T01:51:42.513 INFO:teuthology.orchestra.run.smithi007.stderr: 90.4% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-18T01:51:43.464 INFO:teuthology.orchestra.run.smithi007.stderr: 95.0% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-18T01:51:46.469 INFO:teuthology.orchestra.run.smithi046.stderr: 95.5% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-18T01:51:55.342 INFO:teuthology.orchestra.run.smithi203.stderr: 95.5% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-18T01:52:15.158 INFO:teuthology.orchestra.run.smithi046.stderr: 93.8% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-18T02:01:46.955 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-18T02:02:18.166 INFO:teuthology.orchestra.run.smithi046.stderr: 92.7% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T02:03:48.609 INFO:teuthology.orchestra.run.smithi203.stderr: 92.7% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-18T02:06:15.588 INFO:teuthology.orchestra.run.smithi203.stderr: 92.3% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-18T02:06:20.095 INFO:teuthology.orchestra.run.smithi046.stderr: 92.6% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-18T02:06:45.502 INFO:teuthology.orchestra.run.smithi046.stderr: 91.9% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-18T02:06:45.569 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-18T02:06:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:real 15m12.678s 2024-03-18T02:06:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:user 7m50.473s 2024-03-18T02:06:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m18.379s 2024-03-18T02:07:48.302 INFO:teuthology.orchestra.run.smithi203.stderr: 92.2% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T02:08:40.773 INFO:teuthology.orchestra.run.smithi203.stderr: 92.2% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-18T02:08:40.786 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-18T02:08:40.786 INFO:teuthology.orchestra.run.smithi203.stderr:real 17m7.909s 2024-03-18T02:08:40.787 INFO:teuthology.orchestra.run.smithi203.stderr:user 7m20.460s 2024-03-18T02:08:40.787 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m18.232s 2024-03-18T02:10:37.006 INFO:teuthology.orchestra.run.smithi007.stderr: 92.2% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T02:11:39.701 INFO:teuthology.orchestra.run.smithi007.stderr: 92.5% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-18T02:11:59.905 INFO:teuthology.orchestra.run.smithi007.stderr: 92.0% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-18T02:13:11.652 INFO:teuthology.orchestra.run.smithi007.stderr: 91.8% -- replaced with /var/log/ceph/3fdfdc7c-e4c6-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-18T02:13:11.657 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-18T02:13:11.657 INFO:teuthology.orchestra.run.smithi007.stderr:real 21m38.776s 2024-03-18T02:13:11.657 INFO:teuthology.orchestra.run.smithi007.stderr:user 25m41.015s 2024-03-18T02:13:11.658 INFO:teuthology.orchestra.run.smithi007.stderr:sys 3m3.752s 2024-03-18T02:13:11.659 INFO:tasks.cephadm:Archiving logs... 2024-03-18T02:13:11.659 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi007/log 2024-03-18T02:13:11.660 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T02:14:40.271 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi046/log 2024-03-18T02:14:40.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T02:15:50.899 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi203/log 2024-03-18T02:15:50.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T02:17:02.661 INFO:tasks.cephadm:Removing cluster... 2024-03-18T02:17:02.661 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 --force 2024-03-18T02:17:03.267 INFO:teuthology.orchestra.run.smithi007.stdout:Deleting cluster with fsid: 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T02:17:05.454 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 --force 2024-03-18T02:17:05.847 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T02:17:06.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 --force 2024-03-18T02:17:06.947 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: 3fdfdc7c-e4c6-11ee-95c9-87774f69a715 2024-03-18T02:17:07.671 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T02:17:07.671 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T02:17:07.747 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T02:17:07.765 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T02:17:07.781 INFO:tasks.cephadm:Teardown complete 2024-03-18T02:17:07.781 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T02:17:07.793 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T02:17:07.793 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T02:17:07.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T02:17:07.808 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T02:17:07.976 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T02:17:07.986 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T02:17:07.987 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T02:17:07.995 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T02:17:07.997 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T02:17:08.011 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-18T02:17:08.012 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-03-18T02:17:08.045 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-18T02:17:08.103 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T02:17:08.103 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-03-18T02:17:08.103 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +4636us[+4636us] +/- 138ms 2024-03-18T02:17:08.104 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -1996us[-1996us] +/- 58ms 2024-03-18T02:17:08.104 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +176us[ +176us] +/- 73ms 2024-03-18T02:17:08.104 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T02:17:08.121 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T02:17:08.121 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-18T02:17:08.121 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +4637us[+4637us] +/- 138ms 2024-03-18T02:17:08.121 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -1981us[-1984us] +/- 58ms 2024-03-18T02:17:08.121 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +180us[ +180us] +/- 73ms 2024-03-18T02:17:08.121 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T02:17:08.145 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T02:17:08.145 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-18T02:17:08.145 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 +4591us[+4591us] +/- 138ms 2024-03-18T02:17:08.145 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -2042us[-2035us] +/- 58ms 2024-03-18T02:17:08.145 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +111us[ +118us] +/- 73ms 2024-03-18T02:17:08.145 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T02:17:08.149 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T02:17:08.159 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T02:17:08.160 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T02:17:08.170 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T02:17:08.496 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T02:17:08.624 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T02:17:08.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T02:17:08.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T02:17:08.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T02:17:08.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T02:17:08.884 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T02:17:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T02:17:08.915 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T02:17:08.926 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T02:17:08.937 INFO:teuthology.task.internal:Duration was 3720.546850 seconds 2024-03-18T02:17:08.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T02:17:08.948 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T02:17:08.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T02:17:08.956 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T02:17:08.959 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T02:17:09.029 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T02:17:09.032 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T02:17:09.177 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T02:17:09.681 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T02:17:09.681 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-03-18T02:17:09.682 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T02:17:09.753 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-03-18T02:17:09.753 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T02:17:09.785 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-03-18T02:17:09.785 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T02:17:09.808 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T02:17:09.808 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T02:17:09.816 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T02:17:09.828 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T02:17:09.915 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T02:17:09.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T02:17:09.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T02:17:09.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T02:17:10.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T02:17:10.486 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T02:17:10.486 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T02:17:10.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T02:17:10.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T02:17:10.663 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T02:17:10.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T02:17:10.682 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T02:17:10.684 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T02:17:10.723 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-03-18T02:17:10.744 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-03-18T02:17:10.835 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-03-18T02:17:10.922 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T02:17:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T02:17:10.979 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T02:17:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T02:17:10.995 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T02:17:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T02:17:11.010 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T02:17:11.022 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T02:17:11.023 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi007 2024-03-18T02:17:11.023 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T02:17:11.145 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi046 2024-03-18T02:17:11.146 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T02:17:11.191 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564/remote/smithi203 2024-03-18T02:17:11.192 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T02:17:11.231 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T02:17:11.231 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T02:17:11.239 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T02:17:11.242 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T02:17:11.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T02:17:11.303 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T02:17:11.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T02:17:11.317 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T02:17:11.317 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T02:17:11.325 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T02:17:11.327 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T02:17:11.341 INFO:teuthology.orchestra.run.smithi046.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 02:17 /home/ubuntu/cephtest 2024-03-18T02:17:11.342 INFO:teuthology.orchestra.run.smithi203.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 02:17 /home/ubuntu/cephtest 2024-03-18T02:17:11.374 INFO:teuthology.orchestra.run.smithi007.stdout: 393696 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 02:17 /home/ubuntu/cephtest 2024-03-18T02:17:11.378 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T02:17:11.389 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T02:17:11.413 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T02:17:11.440 DEBUG:teuthology.nuke:shortname: smithi007 2024-03-18T02:17:11.440 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T02:17:11.462 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 01:05:31.806140', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFGVfZM9nUxiR2h/U9LYTl5msB9B7oCYm/aF/4Wtym0Uh1rFBzt66mbiDiDwdoIJ/SypsKQviP9PmDjSV3ETBdQ='} 2024-03-18T02:17:11.466 DEBUG:teuthology.nuke:shortname: smithi046 2024-03-18T02:17:11.467 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T02:17:11.489 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 01:05:31.807473', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF4mBO/8A11c0wg5qNETN3FIlA//9VpkOejz2ACa8BDhRWPN6/24DTAEK3zqvKdPi1v1wr0axrF0qTQjc/OXvGI='} 2024-03-18T02:17:11.494 DEBUG:teuthology.nuke:shortname: smithi203 2024-03-18T02:17:11.494 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T02:17:11.528 INFO:teuthology.orchestra.console.smithi007:Power off 2024-03-18T02:17:11.528 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T02:17:11.540 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608564', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 01:05:31.804310', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEkLfbf54T28KbGzzmjAKyRgVbMPkJs1/WkxT10hhTRpIIpvXnTVv0iyouhKjXpYOVnG01NaU9XgGSz9cr0SHJw='} 2024-03-18T02:17:11.554 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-03-18T02:17:11.554 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:11.575 INFO:teuthology.orchestra.console.smithi046:Power off 2024-03-18T02:17:11.575 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T02:17:11.585 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-18T02:17:11.597 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-03-18T02:17:11.597 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:11.612 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-18T02:17:11.612 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T02:17:11.622 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-03-18T02:17:11.635 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-18T02:17:11.635 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:11.661 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-18T02:17:15.586 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:15.623 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:15.662 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:15.697 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-18T02:17:15.734 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-03-18T02:17:15.771 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-18T02:17:19.702 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:19.735 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:19.772 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:19.815 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-18T02:17:19.844 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-03-18T02:17:19.882 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-18T02:17:19.946 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-03-18T02:17:19.983 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-18T02:17:23.818 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T02:17:23.929 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-03-18T02:17:24.030 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-03-18T02:17:24.136 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3845b7ac715a3ff198582638d199a90e30e254da description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} duration: 3720.546849966049 failure_reason: '"2024-03-18T01:28:04.885193+0000 mon.a (mon.0) 211 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] is down (out of quorum)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-18T02:17:24.136 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T02:17:24.231 INFO:teuthology.run:FAIL